Scala RDD 匹配类似的措辞答案

【问题标题】：Scala RDD matching with similar wordingScala RDD 匹配类似的措辞
【发布时间】：2021-01-22 07:00:07
【问题描述】：

所以我有一个动词列表

假设：

动词.txt

have, have, having, had
give, give, gave, given
take, take, took, taken

将它们拆分为 rdds

val verbs = sc.textFile("verbs.txt").map(x => x.split("\n").collect()

因此，

verbs: Array[Array[String]] = Array(Array(have, have, having, had), Array(give, give, gave, given), Array(take, take, took, taken))

假设：

val wordcount = sc.textFile("data.txt")

数据.txt

have have have having having had had had had had give give give give give give give give give give gave gave given given given given take take took took took took took took taken taken

我已经计算了字数，因此字数=

(have, 3)
(having, 2)
(had, 5)
(give, 10)
(gave, 2)
(given, 4)
(take, 2)
(took, 6)
(taken, 2)

我希望能够将数据与相同的动词合并在一起示例：(have,3),(having,2),(had,5) => (have, 10)

使用数组的第一个值返回动词的基本形式。我怎么能做到这一点？

【问题讨论】：

如果能按每个词的词干分组就好了...
你是怎么做到的？和 groupBy 哪个词？
你的动词的格式是什么。 txt ?

标签： scala rdd

【解决方案1】：

由于您将问题标记为 RDD，我假设您的字数数据是 RDD。

  // Read text file
  val sc = spark.sparkContext
  val textFile: RDD[String] = sc.textFile("data.txt")

  // So you have this as you said
  val verbs = Array(Array("have", "have", "having", "had"), Array("give", "give", "gave", "given"), Array("take", "take", "took", "taken"))

  val data= textFile
    .flatMap(_.split(" ")) // Split each line to words/tokens its called tokenization (I used backspace as seperator if you have tabs as seperator use that)
    .map(t => (t, 1)) // Generate count per token (i.e. (have, 1))
    .reduceByKey(_ + _) // Count appearance of each token (i.e. (have, 5)


  val t = data.map(d => (verbs.find(v => v.contains(d._1)).map(_.head).getOrElse(d._1), d._2)) // Generates RDD of (optional base verb, count for that verb) e.g (having, 5) => (have, 5), unknown verbs left as it is
    .reduceByKey(_ + _) // Sum all values that having same base verb (have, 5), (have, 3) => (have, 8)

  t.take(10).foreach(println)

其他选项（不收集动词）

  // You dont have to collect this If you want
  val verbs2 = sc.parallelize(Array(Array("have", "have", "having", "had"), Array("give", "give", "gave", "given"), Array("take", "take", "took", "taken"))) // This is the state before collect
    .flatMap(v => v.map(v2 => (v2, v.head))) // This generates tuples of verb -> base verb (e.g had -> have)
    .reduceByKey((k1, k2) => if (k1 == k2) k1 else k2) // Current verbs array generates (have -> have twice, this eliminates duplicate records)

  val data2 = textFile
    .flatMap(_.split(" ")) // Split each line to words/tokens its called tokenization (I used backspace as seperator if you have tabs as seperator use that)
    .map(t => (t, 1)) // Generate count per token (i.e. (have, 1))
    .reduceByKey(_ + _) // Count appearance of each token (i.e. (have, 5)

  val t2 = verbs2.join(data2) // This will join two RDD by their keys (verbs -> (base verb, verb count))
    .map(d => d._2) // This is what we need key is base verb, value is count of that verb
    .reduceByKey(_ + _) // Sum all values that having same base verb (have, 5), (have, 3) => (have, 8)

  t2.take(10).foreach(println)

当然，此答案假定您将始终拥有动词数组，并且第一个元素是基本形式。如果您想要在没有动词数组的情况下工作并将任何动词转换为实际上是 NLP（自然语言处理）任务的基本格式，并且您需要使用某种单词规范化技术，例如 this（如 EmiCareOfCell44 所示）。您还可以在 spark ML 库中找到此类过程的实现。

【讨论】：

我试过用你的方法，但是我有 (none, 36) 代替。
您没有得到任何信息或未知信息吗？这很奇怪，尽管您的结构可能与我的示例代码不同（这是我唯一的猜测）。然而，souma-kole 的回答更高效、更有活力。
如果我通过 sc.textFile("data.txt") 获取数据，你能告诉我该怎么做吗？
data.txt文件的内容是什么？
@JoshSoh 好的，我编辑了我的回复，让它更有效，更像火花。这两个选项都应该适用于您的场景。

【解决方案2】：

最好是广播动词的形式，然后在上面查找。它将使查找变得简单和高效，使执行器中的值在一个步骤中可用。

    val conf = new SparkConf()
      .setAppName("Demo")
      .setMaster("local[2]")
    val sc = new SparkContext(conf)

    val verbs = Array(Array("have", "have", "having", "had"), Array("give", "give", "gave", "given"), Array("take", "take", "took", "taken"))
    //brodcast it as a map
    val verbMap = verbs.flatMap(e => {
      e.map(i => i -> e(0))
    }).toMap
    val bdVerbMap = sc.broadcast(verbMap)

    val data = sc.parallelize(List(("have", 3),
      ("having", 2),
      ("had", 5),
      ("give", 10),
      ("gave", 2),
      ("given", 4),
      ("take", 2),
      ("took", 6),
      ("taken", 2)))

    //Lokkup the broadcast values to map every forms of verb then reduce by key
    val unifiedVerbCnt = data.map(t => (bdVerbMap.value.getOrElse(t._1, t._1), t._2))
      .reduceByKey((x, y) => x+y)

    unifiedVerbCnt.collect.foreach(println)

【讨论】：

所以对于动词和数据，我实际上是从一个动词中得到的。 txt 和 data.txt，所以我使用 sc.textFile("data.txt") 而不是 sc.parallelize() 但是当我运行你的方法时，我没有得到我想要的结果，因为它没有字符串括号（有，2）而不是（“有”，2）。我该怎么办？
需要从两个文件中查看几行。您需要在原始问题中添加它（而不是在 cmets 中，在这里您无法保持正确的格式）。

【解决方案3】：

你可以这样做：

verbs
.flatten // will give single array combining all array
.map(p => (p, 1)) // Array( (have, 1), (given, 1))
.groupBy(_._1) // Map(have -> Array((have,1), (have,1)), given -> Array((given,1)))
.map(l => (l._1, l._2.map(_._2).sum)) //Map(have -> 2, given -> 1, had -> 1, took -> 1)

最好在 scala shell 或 IDE 工具中尝试该代码以理解。

【讨论】：