将 K 折验证应用于文本分类

Question

我正在尝试了解 K 折交叉验证，因为我第一次将它用于我的文本分类。但是我对如何在 python 中实现它感到很困惑

我有一个数据框，其中data 是我要预测的文本，标签是预测值（0 或 1）。我目前使用训练测试拆分方法，并在矢量化数据上使用多项式 NB。

from sklearn import model_selection
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer
# split the data into training and testing datasets
X_train, X_test, y_train, y_test = model_selection.train_test_split(df['data'], df['label'], random_state=1)
vect = CountVectorizer(ngram_range=(1,2), max_features=1000 , stop_words="english")
X_train_dtm = vect.fit_transform(X_train)
X_test_dtm = vect.transform(X_test)
nb = MultinomialNB()
nb.fit(X_train_dtm, y_train)
y_pred_class = nb.predict(X_test_dtm)

我只是想知道如何以类似的方式实现 5 折验证。我查看了很多示例，但对于如何以正确的方式进行操作感到很困惑，因为我是初学者。

Answer 1

只需使用 scikit-learn

https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html

>>> import numpy as np
>>> from sklearn.model_selection import KFold
>>> X = np.array([[1, 2], [3, 4], [1, 2], [3, 4]])
>>> y = np.array([1, 2, 3, 4])
>>> kf = KFold(n_splits=2)
>>> kf.get_n_splits(X)
2
>>> print(kf)
KFold(random_state=None, shuffle=False)
>>> for train_index, test_index in kf.split(X):
...     print("TRAIN:", train_index, "TEST:", test_index)
...     X_train, X_test = X[train_index], X[test_index]
...     y_train, y_test = y[train_index], y[test_index]
TRAIN: [2 3] TEST: [0 1]
TRAIN: [0 1] TEST: [2 3]

这里省略了 n_splits 参数，因为默认值为 5，这是您所要求的！

我想这是最简单的方法。始终查看他们提供的带有代码示例的文档，以及所有参数的解释！

这有帮助吗？

编辑：

完整的代码如下所示！

from sklearn import model_selection
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import KFold

X_train, X_test, y_train, y_test = 
model_selection.train_test_split(df['data'], df['label'], 
random_state=1)
kf = KFold(n_splits=2)
kf.get_n_splits(X_train)
for train_index, test_index in kf.split(X):
    print("TRAIN:", train_index, "TEST:", test_index)
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    vect = CountVectorizer(ngram_range=(1,2), max_features=1000 , 
    stop_words="english")
    X_train_dtm = vect.fit_transform(X_train)
    X_test_dtm = vect.transform(X_test)
    nb = MultinomialNB()
    nb.fit(X_train_dtm, y_train)
    y_pred_class = nb.predict(X_test_dtm)

Answer 2

这是一个如何使用KFold的代码示例：

X, y = df['data'], df['label']
metrics = []

skf = StratifiedKFold(n_splits=5)
for train_index, test_index in skf.split(X, y):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]

    vect = CountVectorizer(ngram_range=(1,2), max_features=1000 , stop_words="English")
    X_train_dtm = vect.fit_transform(X_train)
    X_test_dtm = vect.transform(X_test)
    nb = MultinomialNB()
    nb.fit(X_train_dtm, y_train)
    y_pred_class = nb.predict(X_test_dtm)

    metrics.append(accuracy_score(y_test, y_pred_class))

metrics = numpy.array(metrics)
print('Mean accuracy: ', numpy.mean(metrics, axis=0))
print('Std for accuracy: ', numpy.std(metrics, axis=0))

• 主要思想是您可以通过 5 次实验来衡量模型性能。
• 您不仅可以评估 average accuracy，还可以评估 standard deviation - std 越小模型越好。
• 最好使用StratifiedKFold 而不是KFold。