使用 FeatureUnion 为 scikit-learn 管道中的词袋添加额外的特征答案

【问题标题】：Adding extra features to bag of words in scikit-learn pipeline with FeatureUnion使用 FeatureUnion 为 scikit-learn 管道中的词袋添加额外的特征
【发布时间】：2017-06-12 06:12:51
【问题描述】：

我苦苦挣扎，但仍然不知道如何在 scikit-learn 管道中使用带有 FeatureUnion 的文本功能和额外功能。
我有一个句子列表及其标签来训练模型和一个句子列表作为测试数据。然后我尝试在包词中添加一个额外的特征（比如每个句子的长度）。为此，我编写了一个自定义 LengthTransformer，它返回一个长度列表，并且与我的火车列表具有相同数量的元素。
然后我使用FeatureUnion 将它与TfidfVectorizer 结合起来，但它不起作用。

到目前为止，我想出的是：

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.svm import LinearSVC
from sklearn.multiclass import OneVsRestClassifier
from sklearn import preprocessing

class LengthTransformer(BaseEstimator, TransformerMixin):

    def fit(self, X, y=None):
        return self

    def transform(self, X):
        return [len(x) for x in X]


X_train = ["new york is a hell of a town",
            "new york was originally dutch",
            "the big apple is great",
            "new york is also called the big apple",
            "nyc is nice",
            "people abbreviate new york city as nyc",
            "the capital of great britain is london",
            "london is in the uk",
            "london is in england",
            "london is in great britain",
            "it rains a lot in london",
            "london hosts the british museum",
            "new york is great and so is london",
            "i like london better than new york"]
y_train_text = [["new york"], ["new york"], ["new york"], ["new york"], ["new york"],
                ["new york"], ["london"], ["london"], ["london"], ["london"],
                ["london"], ["london"], ["london", "new york"], ["new york", "london"]]

X_test = ['nice day in nyc',
            'welcome to london',
            'london is rainy',
            'it is raining in britian',
            'it is raining in britian and the big apple',
            'it is raining in britian and nyc',
            'hello welcome to new york. enjoy it here and london too']

lb = preprocessing.MultiLabelBinarizer()
Y = lb.fit_transform(y_train_text)

classifier = Pipeline([
    ('feats', FeatureUnion([
       ('tfidf', TfidfVectorizer()),
       ('len', LengthTransformer())
    ])),
    ('clf', OneVsRestClassifier(LinearSVC()))
])

classifier.fit(X_train, Y)
predicted = classifier.predict(X_test)
all_labels = lb.inverse_transform(predicted)

for item, labels in zip(X_test, all_labels):
    print('{} => {}'.format(item, ', '.join(labels)))

【问题讨论】：

标签： python machine-learning scikit-learn

【解决方案1】：

LengthTransformer.transform 返回形状错误 - 它返回每个输入文档的标量，而转换器应该返回每个文档的特征向量。您可以通过将 LengthTransformer.transform 中的 [len(x) for x in X] 更改为 [[len(x)] for x in X] 来使其工作。

【讨论】：