使用 sklearn 进行网格搜索的显式（预定义）验证集

Question

我有一个数据集，之前已分为 3 组：训练、验证和测试。为了比较不同算法的性能，必须按照给定的方式使用这些集合。

我现在想使用验证集优化我的 SVM 的参数。但是，我找不到如何将验证集显式输入sklearn.grid_search.GridSearchCV()。下面是我之前用于在训练集上进行 K 折交叉验证的一些代码。但是，对于这个问题，我需要使用给定的验证集。我该怎么做？

from sklearn import svm, cross_validation
from sklearn.grid_search import GridSearchCV

# (some code left out to simplify things)

skf = cross_validation.StratifiedKFold(y_train, n_folds=5, shuffle = True)
clf = GridSearchCV(svm.SVC(tol=0.005, cache_size=6000,
                             class_weight=penalty_weights),
                     param_grid=tuned_parameters,
                     n_jobs=2,
                     pre_dispatch="n_jobs",
                     cv=skf,
                     scoring=scorer)
clf.fit(X_train, y_train)

Answer 1

使用PredefinedSplit

ps = PredefinedSplit(test_fold=your_test_fold)

然后在GridSearchCV中设置cv=ps

test_fold : “类数组，形状 (n_samples,)

test_fold[i] 给出样本 i 的测试集折叠。值 -1 表示对应的样本不属于任何测试集折叠，而是始终放入训练折叠中。

另见here

使用验证集时，将属于验证集的所有样本的 test_fold 设置为 0，将所有其他样本的 test_fold 设置为 -1。

Answer 2

考虑使用我作为作者的 hypopt Python 包 (pip install hypopt)。这是一个专业的包，专为使用验证集进行参数优化而创建。它适用于任何开箱即用的 scikit-learn 模型，也可以与 Tensorflow、PyTorch、Caffe2 等一起使用。

# Code from https://github.com/cgnorthcutt/hypopt
# Assuming you already have train, test, val sets and a model.
from hypopt import GridSearch
param_grid = [
  {'C': [1, 10, 100], 'kernel': ['linear']},
  {'C': [1, 10, 100], 'gamma': [0.001, 0.0001], 'kernel': ['rbf']},
 ]
# Grid-search all parameter combinations using a validation set.
opt = GridSearch(model = SVR(), param_grid = param_grid)
opt.fit(X_train, y_train, X_val, y_val)
print('Test Score for Optimized Parameters:', opt.score(X_test, y_test))

编辑：我（我想我）收到了 -1 的回复，因为我建议我编写一个包。这是不幸的，因为该软件包是专门为解决此类问题而创建的。

Answer 3

# Import Libraries
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.model_selection import PredefinedSplit

# Split Data to Train and Validation
X_train, X_val, y_train, y_val = train_test_split(X, y, train_size = 0.8, stratify = y,random_state = 2020)

# Create a list where train data indices are -1 and validation data indices are 0
split_index = [-1 if x in X_train.index else 0 for x in X.index]

# Use the list to create PredefinedSplit
pds = PredefinedSplit(test_fold = split_index)

# Use PredefinedSplit in GridSearchCV
clf = GridSearchCV(estimator = estimator,
                   cv=pds,
                   param_grid=param_grid)

# Fit with all data
clf.fit(X, y)

Answer 4

添加到原始答案中，当 train-valid-test 拆分未使用 Scikit-learn 的 train_test_split() 函数完成时，即数据帧已经预先手动拆分并缩放/归一化，以防止训练泄漏数据，numpy数组可以连接起来。

import numpy as np
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_val = scaler.transform(X_val)
X_test = scaler.transform(X_test)

from sklearn.model_selection import PredefinedSplit, GridSearchCV
split_index = [-1]*len(X_train) + [0]*len(X_val)
X = np.concatenate((X_train, X_val), axis=0)
y = np.concatenate((y_train, y_val), axis=0)
pds = PredefinedSplit(test_fold = split_index)

clf = GridSearchCV(estimator = estimator,
                   cv=pds,
                   param_grid=param_grid)

# Fit with all data
clf.fit(X, y)

Answer 5

我想提供一些可重现的代码，使用最后 20% 的观察结果创建验证拆分。

from sklearn import datasets
from sklearn.model_selection import PredefinedSplit, GridSearchCV
from sklearn.ensemble import GradientBoostingRegressor

# load data
df_train = datasets.fetch_california_housing(as_frame=True).data
y = datasets.fetch_california_housing().target

param_grid = {"max_depth": [5, 6],
              'learning_rate': [0.03, 0.06],
              'subsample': [.5, .75]
             }

model = GradientBoostingRegressor()

# Create a single validation split
val_prop = .2
n_val_rows = round(len(df_train) * val_prop)
val_starting_index = len(df_train) - n_val_rows
cv = PredefinedSplit([-1 if i < val_starting_index else 0 for i in df_train.index])


# Use PredefinedSplit in GridSearchCV
results = GridSearchCV(estimator = model,
                       cv=cv,
                       param_grid=param_grid,
                       verbose=True,
                       n_jobs=-1)

# Fit with all data
results.fit(df_train, y)

results.best_params_