在pyspark中计算姓名/姓氏的tf-idf答案

【问题标题】：Calculating tf-idf for name/surname in pyspark在pyspark中计算姓名/姓氏的tf-idf
【发布时间】：2016-02-09 18:39:02
【问题描述】：

我有以下 RDD（示例）：

names_rdd.take(3)
[u'Daryll Dickenson', u'Dat Naijaboi', u'Duc Dung Lam']

我正在尝试计算 tf_idf：

from pyspark.mllib.feature import HashingTF,IDF
hashingTF = HashingTF()
tf_names = hashingTF.transform(names_rdd)
tf_names.cache()
idf_names =IDF().fit(tf_names)
tfidf_names = idf_names.transform(tf_names)

我不明白为什么tf_names.take(3) 会给出这些结果：

[SparseVector(1048576, {60275: 1.0, 134386: 1.0, 145380: 1.0, 274465: 1.0, 441832: 1.0, 579064: 1.0, 590058: 1.0, 664173: 2.0, 812399: 2.0, 845381: 2.0, 886510: 1.0, 897504: 1.0, 1045730: 1.0}),
 SparseVector(1048576, {208501: 1.0, 274465: 1.0, 441832: 2.0, 515947: 1.0, 537935: 1.0, 845381: 1.0, 886510: 1.0, 897504: 3.0, 971619: 1.0}),
 SparseVector(1048576, {274465: 2.0, 282612: 2.0, 293606: 1.0, 389709: 1.0, 738284: 1.0, 812399: 1.0, 845381: 2.0, 897504: 1.0, 1045730: 1.0})]

不应该是每行都有 2 个值，例如：

[SparseVector(1048576, {60275: 1.0, 134386: 1.0}),
 SparseVector(1048576, {208501: 1.0, 274465: 1.0}),
 SparseVector(1048576, {274365: 2.0, 282612: 2.0})]

【问题讨论】：

标签： apache-spark pyspark tf-idf

【解决方案1】：

我做错了，我让每一行都分割单词并列出它。像这样的：

 def split_name(name):
     list_name = name.split(' ')
     list_name = [word.strip() for word in list_name]
     return list_name

 names = names_rdd.map(lambda name:split_name(name))

 hashingTF = HashingTF()
 tf_names = hashingTF.transform(names_rdd)
                   .
                   .
                   .

【讨论】：