如何使用已经训练好的模型对记录进行分类？

Question

我已经成功训练和测试了支持向量分类器模型，通过使用两个用户定义函数 (UDF)，根据标题和摘要对每一行进行分类。第一个 UDF 用于数据预处理，第二个 UDF 用于模型构建。为了创建模型，我使用了之前已经分类的 df1

我被困在如何将这个训练有素的模型实施到新的数据框集上，比如未分类的 df2。欢迎任何建议或帮助。

请参阅下面的用户定义的预处理和模型构建函数

def preprocessing(col,h_pct=1,l_pct=1):
      #Lower case
    lower = col.apply(str.lower)
    
     #Stemming
    from nltk.stem import SnowballStemmer
    stem = SnowballStemmer('english')
    stemmed = lower.apply(lambda x: ' '.join(stem.stem(word) for word in str(x).split()))
    
    #removing punctuation
    import re
    rem_punc = stemmed.apply(lambda x: re.sub(r'[^\w\s]',' ',x))
    
    
#removing stopwords and extra spaces

from nltk.corpus import stopwords
stop_words = set(stopwords.words('english'))
rem_stopwords = rem_punc.apply(lambda x: " ".join(x for x in x.split() if x not in stop_words))
    
    #removing numbers
    rem_num = rem_stopwords.apply(lambda x: " ".join(x for x in x.split() if not x.isdigit()))
    
    #remove words having length=1
    rem_lngth1 = rem_num.apply(lambda x: re.sub(r'[^\w\s]',' ',x))
    
    if h_pct != 0:
        #removing the top $h_pct of the most frequent words 
        high_freq = pd.Series(' '.join(rem_lngth1).split()).value_counts()[:int(pd.Series(' '.join(rem_lngth1).split()).count()*h_pct/100)]
        rem_high = rem_lngth1.apply(lambda x: " ".join(x for x in x.split() if x not in high_freq))
    else:
        rem_high = rem_lngth1
    
    if l_pct != 0:
        #removing the top $l_pct of the least frequent words
        low_freq = pd.Series(' '.join(rem_high).split()).value_counts()[:-int(pd.Series(' '.join(rem_high).split()).count()*l_pct/100):-1]
        rem_low = rem_high.apply(lambda x: " ".join(x for x in x.split() if x not in low_freq))
    else:
        rem_low = rem_high
    
    return rem_low

def prep_fit_pred(df, h_pct, l_pct, model, verbose=False):
    
    df['new_Abstract'] = preprocessing(df['Abstract'],h_pct,l_pct)
    df['concat'] = df['Title'] + '\n' + df['new_Abstract']
    #not removing high and low frequency words from headline
    #this is because the headline carries more significance in determining the classification of the news
    df['concat_processed'] = preprocessing(df['concat'],0,0)

    X = df['concat_processed']
    y = df['Category']
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42, 
                                                    stratify=y) 
    
    bow_xtrain = bow.fit_transform(X_train)
    bow_xtest = bow.transform(X_test)

    model.fit(bow_xtrain,y_train)
    preds = model.predict(bow_xtest)

    acc = accuracy_score(y_test,preds)*100
    
    return preds, acc, model

Answer 1

为了使用经过训练的模型（我假设您使用的是 sklearn）。

您应该像处理训练数据一样预处理未标记的数据。 然后使用bow 和model 以与测试数据相同的方式进行转换和预测。要将弓和模型合并到一个对象中，您可以查看Pipeline。

应该是这样的：

def prep_fit_pred(df, h_pct, l_pct, bow, model, verbose=False):
    df['new_Abstract'] = preprocessing(df['Abstract'],h_pct,l_pct)
    df['concat'] = df['Title'] + '\n' + df['new_Abstract']
    #not removing high and low frequency words from headline
    #this is because the headline carries more significance in determining the classification of the news
    df['concat_processed'] = preprocessing(df['concat'],0,0)

    X = df['concat_processed']
    bow_x = bow.transform(X)
    preds = model.predict(bow_x)
    return preds