Spark NoSuchMethodError

Question

我正在使用以下 Docker 命令运行 spark：

docker run -it \
  -p 8088:8088 -p 8042:8042 -p 50070:50070 \
  -v "$(PWD)"/log4j.properties:/usr/local/spark/conf/log4j.properties \
  -v "$(PWD)":/app -h sandbox sequenceiq/spark:1.6.0 bash

运行spark-submit --version 报告版本1.6.0

我的spark-submit 命令如下：

spark-submit --class io.jobi.GithubDay \
  --master local[*] \
  --name "Daily Github Push Counter" \
  /app/min-spark_2.11-1.0.jar \
  "file:///app/data/github-archive/*.json" \
  "/app/data/ghEmployees.txt" \
  "file:///app/data/emp-gh-push-output" "json"

build.sbt

name := """min-spark"""

version := "1.0"

scalaVersion := "2.11.7"

lazy val sparkVersion = "1.6.0"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-sql" % sparkVersion % "provided"
)

// Change this to another test framework if you prefer
libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.4" % "test"

GithubDay.scala

package io.jobi

import org.apache.spark.sql.SQLContext
import org.apache.spark.{SparkConf, SparkContext}

import scala.io.Source.fromFile

/**
  * Created by hammer on 7/15/16.
  */
object GithubDay {

  def main(args: Array[String]): Unit = {
    println("Application arguments: ")
    args.foreach(println)

    val conf = new SparkConf()
    val sc = new SparkContext(conf)
    val sqlContext = new SQLContext(sc)


    try {
      println("args(0): " + args(0))

      val ghLog = sqlContext.read.json(args(0))
      val pushes = ghLog.filter("type = 'PushEvent'")
      val grouped = pushes.groupBy("actor.login").count()
      val ordered = grouped.orderBy(grouped("count").desc)

      val employees = Set() ++ (
        for {
          line <- fromFile(args(1)).getLines()
        } yield line.trim
      )

      val bcEmployees = sc.broadcast(employees)

      import sqlContext.implicits._
      println("register function")
      val isEmployee = sqlContext.udf.register("SetContainsUdf", (u: String) => bcEmployees.value.contains(u))
      println("registered udf")
      val filtered = ordered.filter(isEmployee($"login"))
      println("applied filter")

      filtered.write.format(args(3)).save(args(2))
    } finally {
      sc.stop()
    }
  }
}

我使用sbt clean package 构建，但运行时的输出是：

Application arguments:
file:///app/data/github-archive/*.json
/app/data/ghEmployees.txt
file:///app/data/emp-gh-push-output
json
args(0): file:///app/data/github-archive/*.json
imported implicits
defined isEmp
register function
Exception in thread "main" java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaUniverse$JavaMirror;
    at io.jobi.GithubDay$.main(GithubDay.scala:53)
    at io.jobi.GithubDay.main(GithubDay.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

从我读到的NoSuchMethodError 是版本不兼容的结果，但我正在使用1.6.0 构建并部署到1.6.0，所以我不明白发生了什么。

Answer 1

除非您自己编译 Spark，否则开箱即用的 1.6.0 版本是使用 Scala 2.10.x 编译的。 This is stated in the docs（表示 1.6.2，但也与 1.6.0 相关）：

Spark 可在 Java 7+、Python 2.6+ 和 R 3.1+ 上运行。对于 Scala API， Spark 1.6.2 使用 Scala 2.10。您将需要使用兼容的 Scala 版本 (2.10.x)。

你想要：

scalaVersion := "2.10.6"

一个提示是错误出现在 Scala 类中：scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)