为什么 Spark with Play 失败并显示“NoClassDefFoundError：无法初始化类 org.apache.spark.SparkConf$”？答案

【问题标题】：Why does Spark with Play fail with "NoClassDefFoundError: Could not initialize class org.apache.spark.SparkConf$"?为什么 Spark with Play 失败并显示“NoClassDefFoundError：无法初始化类 org.apache.spark.SparkConf$”？
【发布时间】：2018-01-12 20:08:44
【问题描述】：

我正在尝试使用这个项目 (https://github.com/alexmasselot/spark-play-activator) 作为 Play 和 Spark 的集成示例，以便在我的项目中做同样的事情。因此，我创建了一个启动 Spark 的对象和一个使用 RDD 读取 Json 文件的控制器。下面是我启动 Spark 的对象：

package bootstrap    
import org.apache.spark.sql.SparkSession    
object SparkCommons {    
  val sparkSession = SparkSession
    .builder
    .master("local")
    .appName("ApplicationController")
    .getOrCreate()
}

而我的 build.sbt 是这样的：

import play.sbt.PlayImport._

name := """crypto-miners-demo"""    
version := "1.0-SNAPSHOT"    
lazy val root = (project in file(".")).enablePlugins(PlayScala)    
scalaVersion := "2.12.4"

libraryDependencies += guice
libraryDependencies += evolutions
libraryDependencies += jdbc
libraryDependencies += filters
libraryDependencies += ws

libraryDependencies += "com.h2database" % "h2" % "1.4.194"
libraryDependencies += "com.typesafe.play" %% "anorm" % "2.5.3"
libraryDependencies += "org.scalatestplus.play" %% "scalatestplus-play" % "3.1.0" % Test

libraryDependencies += "com.typesafe.play" %% "play-slick" % "3.0.0"
libraryDependencies += "com.typesafe.play" %% "play-slick-evolutions" % "3.0.0"
libraryDependencies += "org.xerial" % "sqlite-jdbc" % "3.19.3"

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.0"
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.2.0"

dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7"

但是当我尝试调用使用 RDD 的控制器时，我在 Play 框架上收到此错误：

java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.SparkConf$

我正在使用这样的 RDD：val rdd = SparkCommons.sparkSession.read.json("downloads/tweet-json")。我试图复制配置的应用程序运行良好。我只能将jackson-databind lib 导入到我的 build.sbt 中。将libraryDependencies ++= Dependencies.sparkAkkaHadoop 和ivyScala := ivyScala.value map { _.copy(overrideScalaVersion = true) } 复制到我的build.sbt 时出现错误。

【问题讨论】：

标签： scala apache-spark playframework

【解决方案1】：

我会在黑板上写100000次，永远不会忘记。 Spark 2.2.0 仍然不能与 Scala 2.12 一起使用。我还编辑了 Jackson lib 版本。下面是我的 build.sbt。

import play.sbt.PlayImport._

name := """crypto-miners-demo"""

version := "1.0-SNAPSHOT"

lazy val root = (project in file(".")).enablePlugins(PlayScala)

scalaVersion := "2.11.8"

libraryDependencies += guice
libraryDependencies += evolutions
libraryDependencies += jdbc
libraryDependencies += filters
libraryDependencies += ws

libraryDependencies += "com.h2database" % "h2" % "1.4.194"
libraryDependencies += "com.typesafe.play" %% "anorm" % "2.5.3"
libraryDependencies += "org.scalatestplus.play" %% "scalatestplus-play" % "3.1.0" % Test

libraryDependencies += "com.typesafe.play" %% "play-slick" % "3.0.0"
libraryDependencies += "com.typesafe.play" %% "play-slick-evolutions" % "3.0.0"
libraryDependencies += "org.xerial" % "sqlite-jdbc" % "3.19.3"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.2.0"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.2.0"

dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.6.5"

【讨论】：