如何使用 SVM 提高不平衡数据集的性能

Question

我正在尝试使用 scikit-learn 在令牌级别对数据进行分类。我已经有 train 和 test 拆分。 数据采用以下\t分隔格式：

-----------------
token       label
-----------------
way          6
to           6
reduce       6
the          6
amount       6
of           6
traffic      6
   ....
public       2
transport    5
is           5
a            5
key          5
factor       5
to           5 
minimize     5
   ....

数据分布如下：

                              Training Data                    Test Data
# Total:                        119490                          29699
# Class 0:                      52631                           13490
# Class 1:                      35116                           8625
# Class 2:                      17968                           4161
# Class 3:                      8658                            2088
# Class 4:                      3002                            800
# Class 5:                      1201                            302
# Class 6:                      592                             153

我正在尝试SVM，而F1-score 非常糟糕。

代码是：

import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import accuracy_score, f1_score
from sklearn.model_selection import KFold

if __name__ == '__main__':
    # reading Files
    train_df = pd.read_csv(TRAINING_DATA_PATH, names=['token', 'label'], sep='\t')
    test_df = pd.read_csv(TEST_DATA_PATH, names=['token', 'label'], sep='\t')

    # getting training and testing data
    train_X = train_df['token'].astype('U')
    test_X = test_df['token'].astype('U')
    train_y = train_df['label']
    test_y = test_df['label']

    # Linear SVM
    sgd = Pipeline([('vect', CountVectorizer()),        
                    ('tfidf', TfidfTransformer()),      
                    ('clf',   SGDClassifier(loss='hinge', penalty='l2', alpha=1e-3, max_iter=100, tol=None)
                   ])
    f1_list = []
    acc_list = []
    cv = KFold(n_splits=5)
    for train_index, test_index in cv.split(train_X):
        X_train, X_val = train_X[train_index], train_X[test_index]
        y_train, y_val = train_y[train_index], train_y[test_index]
        sgd.fit(X_train, y_train)
        predicted = sgd.predict(X_val)
        f1 = f1_score(y_val, predicted, average='macro')
        acc = accuracy_score(y_val, predicted)
        f1_list.append(f1)
        acc_list.append(acc)
    print(f1_list)
    print(acc_list)
    sgd_pred = sgd.predict(test_X)
    print('SVM accuracy: %s' % accuracy_score(sgd_pred, test_y))
    print('SVM F1-macro: %s' % f1_score(sgd_pred, test_y, average='macro'))
    print('SVM F1-weighted: %s' % f1_score(sgd_pred, test_y, average='weighted'))

线性支持向量机的结果如下：

SVM accuracy: 0.49493248930940437
SVM F1-macro: 0.2677988484198396

如何提高性能？

Answer 1

您可以考虑采取多种措施来提高预测性能：

1. 使用class_weight 参数来惩罚少数类较重的错误

   sgd = Pipeline([('vect', CountVectorizer()),        
                 ('tfidf', TfidfTransformer()),      
                 ('clf',   SGDClassifier(loss='hinge', 
                                         penalty='l2',
                                         class_weight='balanced', # add this
                                         alpha=1e-3, 
                                         max_iter=100, 
                                         tol=None)
              ])
   

2. 在使用 imblearn 库进行训练之前重新采样您的数据，例如

   from imblearn.oversampling import SMOTE
   
   # getting training and testing data
   train_X = train_df['token'].astype('U')
   test_X = test_df['token'].astype('U')
   train_y = train_df['label']
   test_y = test_df['label']
   
   sm = SMOTE(random_state=42)
   train_X_res, train_y_res = sm.fit_resample(train_X, train_y)
   

3. 尝试另一个分类器，例如Logistic Regression

   sgd = Pipeline([('vect', CountVectorizer()),        
                 ('tfidf', TfidfTransformer()),      
                 ('clf',   SGDClassifier(loss='log', # change here
                                         penalty='l2',
                                         alpha=1e-3, 
                                         max_iter=100, 
                                         tol=None)
   

您可以单独或组合尝试这些方法。但尤其是前两个是处理不平衡数据集的经典措施。