K-fold 交叉验证的准确度排名与单个模型的准确度排名不一致答案

【问题标题】：Accuracy ranking of K-fold cross validation doesn't agree with accuracy ranking of individual modelK-fold 交叉验证的准确度排名与单个模型的准确度排名不一致
【发布时间】：2018-09-10 02:26:43
【问题描述】：

这是我第一次运行 k 折交叉验证，我对从输出中看到的现象感到困惑。基本上，5 折交叉验证始终为模型 8（Adaboost 分类器）和模型 9（梯度提升分类器）提供最高的准确度分数，如下所示。但是，当我使用 20% 的数据集作为测试数据单独运行这些 ML 模型时，根据混淆矩阵和 AUC，模型 7（随机森林分类器）总是在所有 5 个模型中产生最高的准确度。我最初的期望是，如果我要单独运行该 ML 模型，则具有高 k-fold 交叉验证精度的 ML 模型也应该返回高精度。这似乎不是这里的情况。有人可以向我解释为什么我会看到这种差异吗？

这些是我用来训练数据的 ML 模型：

model6 = DecisionTreeClassifier()
model7 = RandomForestClassifier(n_estimators=300)
model8 = AdaBoostClassifier(n_estimators=300)
model9 = GradientBoostingClassifier(n_estimators=300, learning_rate=1.0, max_depth=1, random_state=0)
model10 = KNeighborsClassifier(n_neighbors=5)

这是我的 5 倍 CV 和个人 ML 模型的完整代码：

X_train, X_test, Y_train, Y_test = train_test_split(whole_data_input, whole_data_output, test_size=0.2)
X_train.reset_index(inplace=True)
#To remove the index column:
X_train.drop(['index'],axis=1,inplace=True)

X_test.reset_index(inplace=True)
#To remove the index column:
X_test.drop(['index'],axis=1,inplace=True)

Y_train.reset_index(inplace=True)
#To remove the index column:
Y_train.drop(['index'],axis=1,inplace=True)

Y_test.reset_index(inplace=True)
#To remove the index column:
Y_test.drop(['index'],axis=1,inplace=True)

warnings.filterwarnings('ignore')

model6 = DecisionTreeClassifier()
model7 = RandomForestClassifier(n_estimators=300)
model8 = AdaBoostClassifier(n_estimators=300)
model9 = GradientBoostingClassifier(n_estimators=300, 
learning_rate=1.0,max_depth=1, random_state=0)
model10 = KNeighborsClassifier(n_neighbors=5)

model6.fit(X_train, Y_train)
model7.fit(X_train, Y_train)
model8.fit(X_train, Y_train)
model9.fit(X_train, Y_train)
model10.fit(X_train, Y_train)

# Perform 5-fold cross validation across different models:

#Here I am calling 'whole_data['label'] instead of the 'whole_data[['label']] I created earlier because cross validation only works with this data shape:
whole_data_output=whole_data['label']    

print('THE FOLLOWING OUTPUT REPRESENT ACCURACIES OF 5-FOLD VALIDATIONS FROM VARIOUS ML MODELS:')
print()
scores = cross_val_score(model6, whole_data_input, whole_data_output, cv=5)
print('Cross-validated scores for model6, Decision Tree Classifier, is:' + str(scores))

print()
scores = cross_val_score(model7, whole_data_input, whole_data_output, cv=5)
print('Cross-validated scores for model7, Random Forest Classifier, is:' + str(scores))

print()
scores = cross_val_score(model8, whole_data_input, whole_data_output, cv=5)
print('Cross-validated scores for model8, Adaboost Classifier, is:' + str(scores))

print()
scores = cross_val_score(model9, whole_data_input, whole_data_output, cv=5)
print('Cross-validated scores for model9, Gradient Boosting Classifier, is:' + str(scores))

print()
scores = cross_val_score(model10, whole_data_input, whole_data_output, cv=5)
print('Cross-validated scores for model10, K Neighbors Classifier, is:' + str(scores))

print('THE FOLLOWING OUTPUT REPRESENT RESULTS FROM VARIOUS ML MODELS:')
print()

result6 = model6.predict(X_test)
result7 = model7.predict(X_test)
result8 = model8.predict(X_test)
result9 = model9.predict(X_test)
result10 = model10.predict(X_test)

from sklearn.metrics import classification_report

print('Classification report for model 6, decision tree classifier, is: ')
print(confusion_matrix(Y_test,result6))
print()
print(classification_report(Y_test,result6))
print()
print("Area under curve (auc) of model6 is: ", metrics.roc_auc_score(Y_test, result6)) 
print()

print('Classification report for model 7, random forest classifier, is: ')
print(confusion_matrix(Y_test,result7))
print()
print(classification_report(Y_test,result7))
print()
print("Area under curve (auc) of model7 is: ", metrics.roc_auc_score(Y_test, result7)) 
print()

print('Classification report for model 8, adaboost classifier, is: ')
print(confusion_matrix(Y_test,result8))
print()
print(classification_report(Y_test,result8))
print()
print("Area under curve (auc) of model8 is: ", metrics.roc_auc_score(Y_test, result8)) 
print()

print('Classification report for model 9, gradient boosting classifier, is: ')
print(confusion_matrix(Y_test,result9))
print()
print(classification_report(Y_test,result9))
print()
print("Area under curve (auc) of model9 is: ", metrics.roc_auc_score(Y_test, result9)) 
print()

print('Classification report for model 10, K neighbors classifier, is: ')
print(confusion_matrix(Y_test,result10))
print()
print(classification_report(Y_test,result10))
print()
print("Area under curve (auc) of model10 is: ", metrics.roc_auc_score(Y_test, result10)) 
print()

以下输出表示来自各种 ML 模型的 5 折交叉验证的准确度：

Cross-validated scores for model6, Decision Tree Classifier, is:[ 0.61364665  0.75754735  0.77046902]

Cross-validated scores for model7, Random Forest Classifier, is:[ 0.62463637  0.79326395  0.8073181 ]

Cross-validated scores for model8, Adaboost Classifier, is:[ 0.64916931  0.81960696  0.84196916]

Cross-validated scores for model9, Gradient Boosting Classifier, is:[ 0.64910466  0.82177258  0.83909235]

Cross-validated scores for model10, K Neighbors Classifier, is:[ 0.61180425  0.75412115  0.73012897]

以下输出代表来自各种 ML 模型的结果：

Classification report for model 6, decision tree classifier, is: 
[[6975 1804]
[1893 7891]]

         precision    recall  f1-score   support

     -1       0.79      0.79      0.79      8779
      1       0.81      0.81      0.81      9784
avg / total       0.80      0.80      0.80     18563

Area under curve (auc) of model6 is:  0.800515237805

Classification report for model 7, random forest classifier, is: 
[[6883 1896]
[1216 8568]]

         precision    recall  f1-score   support

     -1       0.85      0.78      0.82      8779
      1       0.82      0.88      0.85      9784
avg / total       0.83      0.83      0.83     18563

Area under curve (auc) of model7 is:  0.829872762782

Classification report for model 8, adaboost classifier, is: 
[[5851 2928]
[ 891 8893]]

         precision    recall  f1-score   support

     -1       0.87      0.67      0.75      8779
      1       0.75      0.91      0.82      9784
avg / total       0.81      0.79      0.79     18563

Area under curve (auc) of model8 is:  0.787704885721

Classification report for model 9, gradient boosting classifier, is: 
[[5905 2874]
[ 918 8866]]

         precision    recall  f1-score   support

     -1       0.87      0.67      0.76      8779
      1       0.76      0.91      0.82      9784
avg / total       0.81      0.80      0.79     18563

Area under curve (auc) of model9 is:  0.789400603089

Classification report for model 10, K neighbors classifier, is: 
[[6467 2312]
[1666 8118]]

         precision    recall  f1-score   support

     -1       0.80      0.74      0.76      8779
      1       0.78      0.83      0.80      9784

avg / total       0.79      0.79      0.79     18563

Area under curve (auc) of model10 is:  0.783183129908

【问题讨论】：

当您不使用交叉验证时，您是否在拆分训练数据和测试数据之前对数据进行了抽样？
@pythonic833 在拆分训练数据和测试数据之前，您能否解释一下样本数据的含义？你指的是分层抽样吗？从包含所有内容的原始数据集中，我将该数据分成 80% 的训练和 20% 的测试。这是正确的做法吗？
在将数据拆分为训练和测试数据之前，您是否对数据进行了采样/重新排列？有时异常值可能位于数据集的开头或结尾。抽样应防止此类异常值集群
@Stanleyrr 尝试在您的cross_val_score 中设置cv=ms.StratifiedKFold(n_splits=5, shuffle=True)，看看它是否有所作为。我的理解是train_test_split 将在类中随机抽样，但cross_val_score 不会（默认情况下）。
没问题，我认为这可能是问题所在，因为我之前也遇到过同样的问题 :) 我已将我的评论添加为答案，请随时接受。

标签： python cross-validation precision-recall auc

【解决方案1】：

尝试在您的 cross_val_score 中设置cv=StratifiedKFold(n_splits=5, shuffle=True)，看看它是否有所作为。我的理解是train_test_split 将在类中随机抽样，但cross_val_score 不会（默认情况下）。

您可以使用from sklearn.model_selection import StratifiedKFold 导入分层kfold

【讨论】：