tf-idf结果分析用python

Question

我正在尝试在大约 200k 令牌的普通语料库上生成 tf-idf。我首先制作了那个词频的向量计数器。然后我生成了 tf-idf 矩阵并得到了以下结果。我的代码是

from sklearn.feature_extraction.text import TfidfVectorizer
with open("D:\history.txt", encoding='utf8') as infile:
    contents = infile.readlines()
#define vectorizer parameters
tfidf_vectorizer = TfidfVectorizer(max_df=1.0, max_features=200000,
                                 min_df=0.0,
                                 use_idf=True, ngram_range=(1,3))
tfidf_matrix = tfidf_vectorizer.fit_transform(contents) #fit the vectorizer to contents

print(tfidf_matrix)

结果

  (0, 8371)     0.0296607326158
  (0, 27755)    0.159032195629
  (0, 59369)    0.0871403881289
   :    :
  (551, 64746)  0.0324104689629
  (551, 10118)  0.0324104689629
  (551, 9308)   0.0324104689629

虽然我想得到以下形式的结果

   (551, good ) 0.0324104689629

Answer 1

您可以使用稀疏输出 tfidf_matrix 和 TfidfVectorizer.get_feature_names() 中的索引来生成所需的输出：

features = tfidf_vectorizer.get_feature_names()
indices = zip(*tfidf_matrix.nonzero())
for row,column in indices:
    print('(%d, %s) %f' %(row, features[column], X[row, column])