Evalml: AutoML: use separate CV split for ensembling

Plan: if stacking is enabled, we'll create a separate split which can be passed to stacked ensembling for CV

It could be neat to look into supporting using the out-of-sample predictions (validation split from original CV) as the data passed to stacking. However I suggest we start with the simpler approach of just creating a separate split if stacking is enabled.

RE our discussion, some supporting evidence for why we should withhold a separate split which stacked ensembling can use to perform CV:

• "By using the cross-validated [predictions,] stacking avoids giving unfairly high weight to models with higher complexity." AKA overfitting
• "The most common approach to preparing the training dataset for the meta-model is via k-fold cross-validation of the base models, where the out-of-fold predictions are used as the basis for the training dataset for the meta-model. The training data for the meta-model may also include the inputs to the base models, e.g. input elements of the training data. This can provide an additional context to the meta-model as to how to best combine the predictions from the meta-model. Once the training dataset is prepared for the meta-model, the meta-model can be trained in isolation on this dataset, and the base-models can be trained on the entire original training dataset." -- blog post
• "It is important that the meta-learner is trained on a separate dataset to the examples used to train the level 0 models to avoid overfitting." -- another blog post
• Original paper abstract which discusses how stacked ensembling can be viewed as a generalization of cross-validation
• I also found this to be a good read.

@rpeck FYI, after some spinning we are following your suggestion 😆

@dsherry @rpeck @angela97lin I started looking at this issue, but it seems like sklearn's StackedClassifier and StackedRegressor classes do use internal cross validation during the training of the model in order to prevent overfitting. This seems to be the same problem that we are trying to solve with this issue, so it seems like it should be resolved. I don't think we'll need to do a separate CV fold for training/validating the Stacked Ensembling methods, but what do you all think?

After discussion with @dsherry, here's the idea we want to proceed with

Plan discussed with @bchen1116 : this issue tracks:

• Create separate split for training the metalearning for ensemble pipelines
• Continue to use sklearn impl for stacked ensembling

Separate performance enhancement: better support for small data. Don't create separate split for ensemling. Use out-of-sample pipeline preds from normal CV (from across all the CV folds) to train the ensembler. #1898

Another separate performance enhancement: train the pipelines and the metalearner on different data. #1897