冲突的跨版本后缀：Spark 作业

Question

下面是我的 build.sbt 文件。我已经搜索过其他类似的问题，但没有一个能帮助我找到答案。我尝试了多种方法来明确使用 2.11 scala，但由于某种原因，我不断收到此错误。

[error] (*:ssExtractDependencies) 跨版本后缀冲突：

org.json4s:json4s-ast, org.apache.spark:spark-network-shuffle, com.twitter:chill, org.json4s:json4s-jackson, com.fasterxml.jackson.module:jackson-module-scala , org.json4s:json4s-core, org.apache.spark:spark-core, org.apache.spark:spark-network-common

[错误] (*:update) 冲突的跨版本后缀：org.json4s:json4s-ast, org.apache.spark:spark-network-shuffle, com.twitter:chill, org.json4s:json4s- jackson, com.fasterxml.jackson.module:jackson-module-scala, org.json4s:json4s-core, org.apache.spark:spark-core, org.apache.spark:spark-network-common

name := "ubi-journeyData-validation"

version := "2.0"

scalaVersion := "2.11.11"
dependencyOverrides += "org.scala-lang" % "scala-compiler" % scalaVersion.value

//updateOptions := updateOptions.value.withCachedResolution(false)

libraryDependencies ++= {
  val sparkVersion = "2.3.0"
  //val sparkVersion = "1.6.3"
     Seq("org.apache.spark" %% "spark-core" % sparkVersion,
    "org.apache.spark" %% "spark-sql" % sparkVersion,
    "org.apache.spark" %% "spark-hive" % sparkVersion,
    "org.elasticsearch" %% "elasticsearch-spark-20" % "5.6.9",
    //"org.elasticsearch" %% "elasticsearch-spark-13" % "5.6.9",
    //"org.elasticsearch" %% "elasticsearch-spark-13" % "5.2.0",
    "org.spark-project.hive" % "hive-cli" % "1.2.1.spark2",
    "org.spark-project.hive" % "hive-metastore" % "1.2.1.spark2",
    "org.spark-project.hive" % "hive-exec" % "1.2.1.spark2"
    //"org.json4s" %% "json4s-jackson" % "3.2.11",
    //"org.apache.calcite" % "calcite-core" % "1.2.0-incubating",
    //"org.pentaho" % "pentaho-aggdesigner" % "5.1.5-jhyde" pomOnly(),
    //"org.pentaho" % "pentaho-aggdesigner-algorithm" % "5.1.5-jhyde" % Test
   )
}

resolvers += Resolver.mavenLocal
resolvers += "Cascading repo" at "http://conjars.org/repo"


assemblyMergeStrategy in assembly := {
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case x => MergeStrategy.first
}

Answer 1

这是因为您的依赖项之一 fe 的依赖项：

"org.spark-project.hive" % "hive-exec" % "1.2.1.spark2"

如果您检查，for example here，这取决于：

org.apache.spark » spark-core_2.10 » 1.3.1 » (optional)

所以恐怕你不能将你的 spark 更新到 2.3.0 和 Scala 到 2.11，除非你转储那些 org.spark-project.hive 模块。