CountVectorizer 不适用于 Python 中的训练数据

Question

我正在使用 scikit learn 对文本进行分类。我用过CountVectorizer。我认为CountVectorizer 应该只用于训练数据，而不是所有数据（特征）。

我已在所有数据（特征）上使用它并且代码有效，但是当我仅在训练中使用它时，它显示此错误：

TypeError：传递了稀疏矩阵，但需要密集数据。利用 X.toarray() 转换为密集的 numpy 数组。

这是我的代码（代码非常简单，仅举例）：

import pandas as pd

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer

from sklearn import tree
from sklearn.metrics import accuracy_score


df = pd.DataFrame({"second":["yes ofc", "not a chance", " hell no", "yes yes yes", "yes",'yes maybe', 'yes ofc', 'no not'],
                  "third":["true","false", "false", "true", "false", "true","false", "false"]})

##CHANGE HERE
results = df['third']
features = df['second']

cv = CountVectorizer()  
#features = cv.fit_transform(features) #it worked

features_train, features_test, result_train, result_test = train_test_split(features, results, test_size = 0.3, random_state = 42)

#features_train = cv.fit_transform(features_train).toarray() #it does not work
#result_train = cv.fit_transform(result_train).toarray() #it does not work

cls = tree.DecisionTreeClassifier()
model = cls.fit(features_train, result_train)

acc_prediction  = model.predict(features_test)
accuracy_test = accuracy_score(result_test, acc_prediction)

print(accuracy_test)

Answer 1

您应该只在训练数据上训练（或fit）计数器矢量化器，但同时在训练和测试数据上运行。

创建CountVectorizer后：

cv = CountVectorizer()

并将数据分成训练集和测试集：

features_train, features_test, result_train, result_test = train_test_split(features, results, test_size = 0.3, random_state = 42)

fit_transformfeatures_train 在继续之前，因为您想使用计数矢量化器转换的数据来训练您的实际分类器：

features_train = cv.fit_transform(features_train)

现在，在此之后，cv 将拟合训练数据，并且也转换了训练数据。现在，使用这些转换后的数据训练实际的分类器：

cls = tree.DecisionTreeClassifier()
model = cls.fit(features_train, result_train)

现在，您的分类器使用 count vectorized 训练数据进行训练。在测试数据上测试准确性时，首先使用相同的计数向量器对测试数据进行转换：

features_test = cv.transform(features_test)

请注意，您没有再次拟合它，我们只是使用已经训练过的计数矢量化器来转换此处的测试数据。现在，使用经过训练的决策树分类器进行预测：

acc_prediction = model.predict(features_test)
accuracy_test = accuracy_score(result_test, acc_prediction)
print(accuracy_test)

Answer 2

要将拟合模型应用于测试数据，请使用.transform()。下面的代码是我的建议！

另外，.toarray() 是将稀疏矩阵转换为密集矩阵的代价高昂的操作。因此，在绝对需要之前不要使用它。决策树可以使用稀疏矩阵本身。

import pandas as pd

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer

from sklearn import tree
from sklearn.metrics import accuracy_score


df = pd.DataFrame({"second":["yes ofc", "not a chance", " hell no", "yes yes yes", "yes",'yes maybe', 'yes ofc', 'no not'],
                  "third":["true","false", "false", "true", "false", "true","false", "false"]})

##CHANGE HERE
results = df['third']
features = df['second']

cv = CountVectorizer()  

features_train, features_test, result_train, result_test = train_test_split(features, results, test_size = 0.3, random_state = 42)

features_train = cv.fit_transform(features_train) 
features_test = cv.transform(features_test) 

cls = tree.DecisionTreeClassifier()
model = cls.fit(features_train, result_train)

acc_prediction  = model.predict(features_test)
accuracy_test = accuracy_score(result_test, acc_prediction)

print(accuracy_test)

Answer 3

试试这个：

features = cv.fit_transform(features)
X_train , X_test = cross_validation.train_test_split(features , test_size=0.3 , random_state=0)
Y_train , Y_test = cross_validation.train_test_split(results, test_size=0.3 , random_state=0) 

Answer 4

以下代码有效。我猜你在分配结果和特征方面做错了。

import pandas as pd

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer

from sklearn import tree
from sklearn.metrics import accuracy_score


df = pd.DataFrame({"second":["yes ofc", "not a chance", " hell no", "yes yes yes", "yes",'yes maybe', 'yes ofc', 'no not'],
                  "third":["true","false", "false", "true", "false", "true","false", "false"]})

##CHANGE HERE
results = df['third']
features = df['second']

cv = CountVectorizer()  
features = cv.fit_transform(features) #it worked

features_train, features_test, result_train, result_test = train_test_split(features, results, test_size = 0.3, random_state = 42)

#features_train = cv.fit_transform(features_train) #it does not work
#result_train = cv.fit_transform(result_train) #it does not work

cls = tree.DecisionTreeClassifier()
model = cls.fit(features_train, result_train)

acc_prediction  = model.predict(features_test)
accuracy_test = accuracy_score(result_test, acc_prediction)

print(accuracy_test)

如果您想在训练集和测试集上分别运行 CountVectorizer，那么下面是执行此操作的方法：

{SAME AS ABOVE TILL HERE}

results = df['third']
features = df['second']

cv = CountVectorizer()  

features_train, features_test, result_train, result_test = train_test_split(features, results, test_size = 0.3, random_state = 42)

features_train = cv.fit_transform(features_train) #it does not work

cls = tree.DecisionTreeClassifier()
model = cls.fit(features_train, result_train)

acc_prediction  = model.predict(cv.transform(features_test))
accuracy_test = accuracy_score(result_test, acc_prediction)

print(accuracy_test)