列出不同行中频繁出现的单词

Question

我有以下具有 id、文本的数据。我想从文本列中跨行查找相同“id”的频繁出现的单词。我不想考虑同一行中经常出现的单词（文本列中的句子）。我尝试使用 TF-IDF 算法来实现它。

id,text
1,Interface Down GigabitEthernet0/1/2 null .  
1,Interface Gi0/1/2 Down on node BMAC69RT01  
1,Interface Down MEth0/0/1 null .  
1,Interface MEth0/0/1 Down on node  
2,Interface Up FastEthernet0/0/0 null 
2,Interface Fa0/0/0 Down on node

首先我从文本列创建标记

val tokenizer = new Tokenizer().setInputCol("text").setOutputCol("words")

然后我尝试使用countvectorizer和IDF来获取常用词。我认为这里不需要countvectorizer，因为我不需要考虑同一句话中的词频。

  val countVectors = new CountVectorizer()
    .setInputCol("words")
      .setOutputCol("vectorText")
 val idf = new IDF().setInputCol("vectorText").setOutputCol("features")

这给了我如下输出

|1  |(11,[0,1,2,6],[0.0,0.15415067982725836,0.3364722366212129,1.252762968495368]) 
|1  |(11,[0,1,2,3,4,5,8],[0.0,0.3083013596545167,0.3364722366212129,1.1192315758708453,0.5596157879354227,0.5596157879354227,1.252762968495368]) 
|1  |(11,[0,1,2,3],[0.0,0.15415067982725836,0.3364722366212129,0.5596157879354227]) 
|1  |(11,[0,1,3,4,5],[0.0,0.15415067982725836,0.5596157879354227,0.5596157879354227,0.5596157879354227]) 
|2  |(11,[0,2,7,9],[0.0,0.3364722366212129,1.252762968495368,1.252762968495368]) 
|2  |(11,[0,1,4,5,10],[0.0,0.15415067982725836,0.5596157879354227,0.5596157879354227,1.252762968495368])

我知道上面的输出为我提供了每个单词的特征和频率。但是从上面的输出我怎么能得到真实的单词和它的频率。我想要一个类似于以下输出的输出。 spark中可用的任何其他算法来实现以下输出

Label | (Word, Frequency)
1, | (Interface, 4) (Down, 4) (null, 2) (on, 2)
2, | (Interface, 2) 

Answer 1

认为这篇文章可能对您有所帮助，这是一种使用 fold 操作获得所需输出的方法

import scala.io.Source
Source.fromFile("fileName").getLines()
 .toList.tail //remove headers
 .foldLeft(Map.empty[Int,Map[String,Int]]){ //use fold with seed
    (map, line) => { 
       val words = line.split("\\W+") //split each line into words
       val lineNumber = words.apply(0).toInt //get line number this can throw error
       var existingCount = map.getOrElse(lineNumber, Map.empty[ String, Int]) //check for existing count
       words.tail.map(word => { 
          val result: Int = existingCount.getOrElse(word,0) 
          existingCount = existingCount + (word -> (result + 1))
       })
       map + (lineNumber -> existingCount)
   }
}.foreach(e => println(e._1+ " | "+e._2.map(x => "("+x._1+", "+x._2+")")))

// 1 | List((Interface, 3), (MEth0, 2), (BMAC69RT01, 1), (null, 1), (1, 3), (on, 2), (Down, 3), (0, 2), (Gi0, 1), (2, 1), (node, 2))
// 2 | List((Interface, 2), (null, 1), (Fa0, 1), (on, 1), (Down, 1), (0, 4), (FastEthernet0, 1), (Up, 1), (node, 1))