sklearn GridSearchCV（评分函数错误）答案

【问题标题】：sklearn GridSearchCV (Scoring Function error)sklearn GridSearchCV（评分函数错误）
【发布时间】：2015-10-15 09:37:57
【问题描述】：

我想知道您是否可以帮助我解决我在运行网格搜索时收到的错误。我认为这可能是由于对网格搜索的实际工作方式存在误解。

我现在正在运行一个应用程序，我需要在其中使用不同的评分函数进行网格搜索来评估最佳参数。我正在使用 RandomForestClassifier 将大型 X 数据集拟合到特征向量 Y，该向量 Y 是 0 和 1 的列表。（完全二进制）。我的评分函数 (MCC) 要求预测输入和实际输入完全是二元的。但是，由于某种原因，我不断收到 ValueError: multiclass is not supported。

我的理解是网格搜索，对数据集进行交叉验证，提出基于交叉验证的预测输入，然后将特征向量和预测插入到函数中。由于我的特征向量是完全二进制的，所以我的预测向量也应该是二进制的，并且在评估分数时不会出现问题。当我使用单个定义的参数（不使用网格搜索）运行随机森林时，将预测数据和特征向量插入 MCC 评分函数运行得非常好。所以我有点不知道运行网格搜索会如何导致任何错误。

数据快照：

        print len(X)
        print X[0]
        print len(Y)
        print Y[2990:3000]
17463699
[38.110903683955435, 38.110903683955435, 38.110903683955435, 9.899495124816895, 294.7808837890625, 292.3835754394531, 293.81494140625, 291.11065673828125, 293.51739501953125, 283.6424865722656, 13.580912590026855, 4.976086616516113, 1.1271398067474365, 0.9465181231498718, 0.5066819190979004, 0.1808401197195053, 0.0]
17463699
[0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0]

代码：

def overall_average_score(actual,prediction):
    precision = precision_recall_fscore_support(actual, prediction, average = 'binary')[0]
    recall = precision_recall_fscore_support(actual, prediction, average = 'binary')[1]
    f1_score = precision_recall_fscore_support(actual, prediction, average = 'binary')[2]
    total_score = matthews_corrcoef(actual, prediction)+accuracy_score(actual, prediction)+precision+recall+f1_score
    return total_score/5

grid_scorer = make_scorer(overall_average_score, greater_is_better=True)
parameters = {'n_estimators': [10,20,30], 'max_features': ['auto','sqrt','log2',0.5,0.3], }
random  = RandomForestClassifier()
clf = grid_search.GridSearchCV(random, parameters, cv = 5, scoring = grid_scorer)
clf.fit(X,Y)

错误：

ValueError                                Traceback (most recent call last)
<ipython-input-39-a8686eb798b2> in <module>()
     18 random  = RandomForestClassifier()
     19 clf = grid_search.GridSearchCV(random, parameters, cv = 5, scoring = grid_scorer)
---> 20 clf.fit(X,Y)
     21 
     22 

/shared/studies/nonregulated/neurostream/neurostream/local/lib/python2.7/site-packages/sklearn/grid_search.pyc in fit(self, X, y)
    730 
    731         """
--> 732         return self._fit(X, y, ParameterGrid(self.param_grid))
    733 
    734 

/shared/studies/nonregulated/neurostream/neurostream/local/lib/python2.7/site-packages/sklearn/grid_search.pyc in _fit(self, X, y, parameter_iterable)
    503                                     self.fit_params, return_parameters=True,
    504                                     error_score=self.error_score)
--> 505                 for parameters in parameter_iterable
    506                 for train, test in cv)
    507 

/shared/studies/nonregulated/neurostream/neurostream/local/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in __call__(self, iterable)
    657             self._iterating = True
    658             for function, args, kwargs in iterable:
--> 659                 self.dispatch(function, args, kwargs)
    660 
    661             if pre_dispatch == "all" or n_jobs == 1:

/shared/studies/nonregulated/neurostream/neurostream/local/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in dispatch(self, func, args, kwargs)
    404         """
    405         if self._pool is None:
--> 406             job = ImmediateApply(func, args, kwargs)
    407             index = len(self._jobs)
    408             if not _verbosity_filter(index, self.verbose):

/shared/studies/nonregulated/neurostream/neurostream/local/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in __init__(self, func, args, kwargs)
    138         # Don't delay the application, to avoid keeping the input
    139         # arguments in memory
--> 140         self.results = func(*args, **kwargs)
    141 
    142     def get(self):

/shared/studies/nonregulated/neurostream/neurostream/local/lib/python2.7/site-packages/sklearn/cross_validation.pyc in _fit_and_score(estimator, X, y, scorer, train, test, verbose, parameters, fit_params, return_train_score, return_parameters, error_score)
   1476 
   1477     else:
-> 1478         test_score = _score(estimator, X_test, y_test, scorer)
   1479         if return_train_score:
   1480             train_score = _score(estimator, X_train, y_train, scorer)

/shared/studies/nonregulated/neurostream/neurostream/local/lib/python2.7/site-packages/sklearn/cross_validation.pyc in _score(estimator, X_test, y_test, scorer)
   1532         score = scorer(estimator, X_test)
   1533     else:
-> 1534         score = scorer(estimator, X_test, y_test)
   1535     if not isinstance(score, numbers.Number):
   1536         raise ValueError("scoring must return a number, got %s (%s) instead."

/shared/studies/nonregulated/neurostream/neurostream/local/lib/python2.7/site-packages/sklearn/metrics/scorer.pyc in __call__(self, estimator, X, y_true, sample_weight)
     87         else:
     88             return self._sign * self._score_func(y_true, y_pred,
---> 89                                                  **self._kwargs)
     90 
     91 

<ipython-input-39-a8686eb798b2> in overall_average_score(actual, prediction)
      3     recall = precision_recall_fscore_support(actual, prediction, average = 'binary')[1]
      4     f1_score = precision_recall_fscore_support(actual, prediction, average = 'binary')[2]
----> 5     total_score = matthews_corrcoef(actual, prediction)+accuracy_score(actual, prediction)+precision+recall+f1_score
      6     return total_score/5
      7 def show_score(actual,prediction):

/shared/studies/nonregulated/neurostream/neurostream/local/lib/python2.7/site-packages/sklearn/metrics/classification.pyc in matthews_corrcoef(y_true, y_pred)
    395 
    396     if y_type != "binary":
--> 397         raise ValueError("%s is not supported" % y_type)
    398 
    399     lb = LabelEncoder()

ValueError: multiclass is not supported

【问题讨论】：

看起来错误可以追溯到您的评分函数调用的东西：matthews_corrcoef(y_true, y_pred)，问题不在于网格搜索本身。我猜如果您使用内置而不是自定义评分功能，则不会出现错误。我将专注于解决评分功能而不是网格搜索。
请至少添加一些数据快照。喜欢print(X)、print(Y) 声明，如果您希望我们能够帮助您。在旁注中，您对精度、召回率和 f1_score 的三行分配可以重写为一行：precision, recall, f1_score, _ = precision_recall_fscore_support(actual, prediction, average='binary')
@user3914041 感谢您的回答！是的，我知道捷径是完全可行的，但由于某种原因，ipython notebook 很挑剔，并没有让我为该特定功能使用元组分配。我不确定它为什么会这样。但为了方便起见，我现在添加了我的数据快照。
@Ryan 我认为这与评分功能有关，但我不明白用 Matthews 重新运行它会有什么不同。
@ShageenthSandrakumar 我的主要观点是在 grid_search 的上下文之外对评分函数本身进行故障排除，例如通过为评分函数提供一些组成的数据（在调用 make_scorer 之前）并制作确保你得到你正在寻找的结果。

标签： python machine-learning scikit-learn grid-search

【解决方案1】：

我复制了您的实验，但没有收到任何错误。该错误表明您的向量之一 actual 或 prediction 包含两个以上的离散值。

你能够在GridSearchCV之外训练随机森林得分确实很奇怪。
你能提供你运行的确切代码吗？

这是我用来尝试重现错误的代码：

from sklearn.datasets import make_classification
from sklearn.grid_search import GridSearchCV
from sklearn.metrics import precision_recall_fscore_support, accuracy_score, \
    matthews_corrcoef, make_scorer
from sklearn.ensemble import RandomForestClassifier
from sklearn.cross_validation import train_test_split

def overall_average_score(actual,prediction):
    precision, recall, f1_score, _ = precision_recall_fscore_support(
        actual, prediction, average='binary')
    total_score = (matthews_corrcoef(actual, prediction) +
        accuracy_score(actual, prediction) + precision + recall + f1_score)
    return total_score / 5

grid_scorer = make_scorer(overall_average_score, greater_is_better=True)

print("Without GridSearchCV")
X, y = make_classification(n_samples=500, n_informative=10, n_classes=2)
X_train, X_test, y_train, y_test = train_test_split(X, y,
    test_size=0.5, random_state=0)
rf = RandomForestClassifier()
rf.fit(X_train, y_train)
y_pred = rf.predict(X_test)
print("Overall average score: ", overall_average_score(y_test, y_pred))

print("-" * 30)
print("With GridSearchCV:")

parameters = {'n_estimators': [10,20,30],
              'max_features': ['auto','sqrt','log2',0.5,0.3], }
gs_rf = GridSearchCV(rf, parameters, cv=5, scoring=grid_scorer)
gs_rf.fit(X_train,y_train)
print("Best score with grid search: ", gs_rf.best_score_)

现在我想对您提供的代码做一些 cmet：

使用诸如random（这通常是一个模块）或f1_score（这与sklearn.metrics.f1_score 方法冲突）之类的变量名并不是一个好习惯。
你可以直接解压precision、recall和f1_score，而不是调用3次precision_recall_fscore_support。
在n_estimators 上进行网格搜索实际上没有意义：树越多越好。如果您担心过度拟合，可以通过使用其他参数（例如 max_depth 或 min_samples_split）来降低单个模型的复杂性。

【讨论】：

【解决方案2】：

Matthews Correlation Coefficient 是介于 -1 和 1 之间的分数。因此，计算 f1_score、precision、recall、accuracy_score 和 MCC 之间的平均值是不正确的。

MCC 值表示： 1 是总正相关 0 没有相关性 -1 是完全负相关

而上述其他评估指标介于 0 和 1 之间（从最差到最佳准确度指数）。范围和意义不一样。

【讨论】：