Spark中的Scala / Java不兼容的布尔类型答案

【问题标题】：Scala/Java incompatible Boolean type in sparkSpark中的Scala / Java不兼容的布尔类型
【发布时间】：2017-08-23 05:02:43
【问题描述】：

从https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-sql-joins.html#joinWith尝试这个例子时

    case class Person(id: Long, name: String, cityId: Long)
    case class City(id: Long, name: String)

    val people = Seq(Person(0, "Agata", 0), Person(1, "Iweta", 0)).toDS
    val cities = Seq(City(0, "Warsaw"), City(1, "Washington")).toDS

    val joined = people.joinWith(cities, people("cityId") === cities("id"))
    joined.show()

我收到此错误

Caused by: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 21, Column 35: Incompatible expression types "boolean" and "java.lang.Boolean"

帮我解决这个问题。提前致谢。

【问题讨论】：

我猜错误不在您在问题中的代码中。它在别的地方。
能否将导入语句（尤其是隐含语句）和初始化SparkSession 的方式添加到列表中？
val conf = new SparkConf().setAppName("test").setMaster("local") val sc = new SparkContext(conf) val sq = SparkSession.builder().getOrCreate() 导入sq.implicits._
我将以上内容粘贴到 Spark 2.2.0 spark-shell 中，效果很好。

标签： java scala apache-spark boolean

【解决方案1】：

我使用 Spark 1.6.0 版、Scala 2.10.5 版（Java HotSpot(TM) 64 位服务器 VM、Java 1.7.0_67）尝试了您的代码并得到了

scala> val joined = people.joinWith(cities, people("cityId") === cities("id"))
<console>:33: error: org.apache.spark.sql.Dataset[Person] does not take parameters
         val joined = people.joinWith(cities, people("cityId") === cities("id"))
                                                    ^

你也可以试试

people.as("p").joinWith(cities.as("c"), $"p.cityId" === $"c.id").show

或

people.joinWith(cities, people.toDF()("cityId") === cities.toDF()("id")).show

或

val peopleDF = Seq(Person(0, "Agata", 0), Person(1, "Iweta", 0)).toDF
val citiesDF = Seq(City(0, "Warsaw"), City(1, "Washington")).toDF
peopleDF.join(citiesDF, peopleDF("cityId") === citiesDF("id")).show

【讨论】：