问题
目前,堆叠集成器有自己的 CV 设置。 IterativeAlgorithm
调用_make_stacked_ensembler
util ,但当前未通过 automl 搜索中的数据拆分器进行线程化。
堆叠集成器中默认设置的数据拆分器未设置shuffle=True
,如果输入数据集具有排序,这可能会导致性能不佳。 它也不会对n_folds
等其他参数具有相同的设置,这并不理想。
此外,这种差异使我们无法支持 sklearn 0.24.0 。 解决这个问题应该能让我们支持那个版本。
使固定
让我们让 automl 通过IterativeAlgorithm
将其数据拆分器向下传递到堆叠集成器中。
@angela97lin我的解释是否有意义/我们在设置堆叠时是否有理由选择不这样做? :)
@dsherry我认为你的解释是有道理的! IIRC 当我们设置堆叠并试图使其性能更高/使堆叠运行更快时,我们希望默认为没有太多折叠的东西 - 因此我们采用self._default_cv(n_splits=3, random_state=random_state)
行默认由 scikit-learn 指定,并将n_splits
硬编码为 3。
再深入一点,并尝试将 AutoML 使用的数据拆分方法编织到堆叠集成组件中。 但是,我遇到了这个问题(在解决了让我们的TrainingValidationSplit
类工作所需的 API 更新之后):
estimator = WrappedSKClassifier(pipeline=LogisticRegressionBinaryPipeline(parameters={'Imputer':{'categorical_impute_strategy': 'm...Logistic Regression Classifier':{'penalty': 'l2', 'C': 1.0, 'n_jobs': -1, 'multi_class': 'auto', 'solver': 'lbfgs'},}))
X = 0 1 2 3 4
0 0.965469 0.041236 0.028701 0.659165 0.213375
1 0.043831...978 0.079577
48 0.376344 0.920154 0.314640 0.180086 0.197598
49 0.682661 0.046529 0.400513 0.412513 0.751464
y = array([1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0,
1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0,
1, 0, 1, 0, 1, 0])
<strong i="7">@_deprecate_positional_args</strong>
def cross_val_predict(estimator, X, y=None, *, groups=None, cv=None,
n_jobs=None, verbose=0, fit_params=None,
pre_dispatch='2*n_jobs', method='predict'):
"""Generate cross-validated estimates for each input data point
The data is split according to the cv parameter. Each sample belongs
to exactly one test set, and its prediction is computed with an
estimator fitted on the corresponding training set.
Passing these predictions into an evaluation metric may not be a valid
way to measure generalization performance. Results can differ from
:func:`cross_validate` and :func:`cross_val_score` unless all tests sets
have equal size and the metric decomposes over samples.
Read more in the :ref:`User Guide <cross_validation>`.
Parameters
----------
estimator : estimator object implementing 'fit' and 'predict'
The object to use to fit the data.
X : array-like of shape (n_samples, n_features)
The data to fit. Can be, for example a list, or an array at least 2d.
y : array-like of shape (n_samples,) or (n_samples, n_outputs), \
default=None
The target variable to try to predict in the case of
supervised learning.
groups : array-like of shape (n_samples,), default=None
Group labels for the samples used while splitting the dataset into
train/test set. Only used in conjunction with a "Group" :term:`cv`
instance (e.g., :class:`GroupKFold`).
cv : int, cross-validation generator or an iterable, default=None
Determines the cross-validation splitting strategy.
Possible inputs for cv are:
- None, to use the default 5-fold cross validation,
- int, to specify the number of folds in a `(Stratified)KFold`,
- :term:`CV splitter`,
- An iterable yielding (train, test) splits as arrays of indices.
For int/None inputs, if the estimator is a classifier and ``y`` is
either binary or multiclass, :class:`StratifiedKFold` is used. In all
other cases, :class:`KFold` is used.
Refer :ref:`User Guide <cross_validation>` for the various
cross-validation strategies that can be used here.
.. versionchanged:: 0.22
``cv`` default value if None changed from 3-fold to 5-fold.
n_jobs : int, default=None
Number of jobs to run in parallel. Training the estimator and
predicting are parallelized over the cross-validation splits.
``None`` means 1 unless in a :obj:`joblib.parallel_backend` context.
``-1`` means using all processors. See :term:`Glossary <n_jobs>`
for more details.
verbose : int, default=0
The verbosity level.
fit_params : dict, defualt=None
Parameters to pass to the fit method of the estimator.
pre_dispatch : int or str, default='2*n_jobs'
Controls the number of jobs that get dispatched during parallel
execution. Reducing this number can be useful to avoid an
explosion of memory consumption when more jobs get dispatched
than CPUs can process. This parameter can be:
- None, in which case all the jobs are immediately
created and spawned. Use this for lightweight and
fast-running jobs, to avoid delays due to on-demand
spawning of the jobs
- An int, giving the exact number of total jobs that are
spawned
- A str, giving an expression as a function of n_jobs,
as in '2*n_jobs'
method : {'predict', 'predict_proba', 'predict_log_proba', \
'decision_function'}, default='predict'
The method to be invoked by `estimator`.
Returns
-------
predictions : ndarray
This is the result of calling `method`. Shape:
- When `method` is 'predict' and in special case where `method` is
'decision_function' and the target is binary: (n_samples,)
- When `method` is one of {'predict_proba', 'predict_log_proba',
'decision_function'} (unless special case above):
(n_samples, n_classes)
- If `estimator` is :term:`multioutput`, an extra dimension
'n_outputs' is added to the end of each shape above.
See Also
--------
cross_val_score : Calculate score for each CV split.
cross_validate : Calculate one or more scores and timings for each CV
split.
Notes
-----
In the case that one or more classes are absent in a training portion, a
default score needs to be assigned to all instances for that class if
``method`` produces columns per class, as in {'decision_function',
'predict_proba', 'predict_log_proba'}. For ``predict_proba`` this value is
0. In order to ensure finite output, we approximate negative infinity by
the minimum finite float value for the dtype in other cases.
Examples
--------
>>> from sklearn import datasets, linear_model
>>> from sklearn.model_selection import cross_val_predict
>>> diabetes = datasets.load_diabetes()
>>> X = diabetes.data[:150]
>>> y = diabetes.target[:150]
>>> lasso = linear_model.Lasso()
>>> y_pred = cross_val_predict(lasso, X, y, cv=3)
"""
X, y, groups = indexable(X, y, groups)
cv = check_cv(cv, y, classifier=is_classifier(estimator))
splits = list(cv.split(X, y, groups))
test_indices = np.concatenate([test for _, test in splits])
if not _check_is_permutation(test_indices, _num_samples(X)):
> raise ValueError('cross_val_predict only works for partitions')
E ValueError: cross_val_predict only works for partitions
../venv/lib/python3.7/site-packages/sklearn/model_selection/_validation.py:845: ValueError
这是我们尝试调用时抛出的错误:
clf = StackedEnsembleClassifier(input_pipelines=[logistic_regression_binary_pipeline_class(parameters={})], cv=TrainingValidationSplit())
clf.fit(X, y)
之所以会这样,是因为scikit-learn验证通过的cv确实是交叉验证的方法; 它对诸如TrainingValidationSplit
类的单个拆分不满意,其中某些数据永远不会出现在测试数据中(因为只有一个拆分)。
因此,我认为现在最好的计划是做简单的事情来支持 scikit-learn 0.24 并设置默认 cv 的shuffle=True
。 如果我们认为这是一件有用的事情,我们可以重新审视它。 想法,@dsherry?
相关: https :