java.lang.NoSuchMethodError: org.json4s.JsonDSL$.seq2jvalue(Lscala/collection/Traversable;Lscala/Function1;)Lorg/json4s/JsonAST$JArray;

Question

我尝试使用 spark-submit 将数据从 Spark DataFrame 写入 Parquet 文件，但使用 JsonDSL 时出现 java.lang.NoSuchMethodError 错误。

从Spark-Shell 运行时，相同的写入块工作正常。使用 sqlContext 写入 parquet 文件时是否需要添加任何其他依赖项？

我正在使用 Spark sqlContext 从 Cassandra 表中读取数据。

inputDF.drop("col1")
.repartition(1)
.write
.option("header", flase)
.mode("overwrite")
.parquet("local_path")

Exception in thread "main" java.lang.NoSuchMethodError: org.json4s.JsonDSL$.seq2jvalue(Lscala/collection/Traversable;Lscala/Function1;)Lorg/json4s/JsonAST$JArray;
    at org.apache.spark.sql.types.StructType$$anonfun$jsonValue$3.apply(StructType.scala:333)
    at org.apache.spark.sql.types.StructType$$anonfun$jsonValue$3.apply(StructType.scala:333)
    at org.json4s.JsonDSL$JsonAssoc.$tilde(JsonDSL.scala:90)
    at org.apache.spark.sql.types.StructType.jsonValue(StructType.scala:333)
    at org.apache.spark.sql.types.StructType.jsonValue(StructType.scala:99)
    at org.apache.spark.sql.types.DataType.json(DataType.scala:67)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport$.setSchema(ParquetWriteSupport.scala:445)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat.prepareWrite(ParquetFileFormat.scala:111)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:103)
    at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
    at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
    at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:83)
    at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:81)
    at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
    at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
    at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)
    at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)
    at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)
    at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:566)

我正在使用以下版本。

scala: 2.11.12
spark: 2.4.6
java: 11

任何帮助将不胜感激。

Answer 1

查看以下帖子后，我将 JSON 版本从 3.7.0-M2 降级到 3.5.3 问题得到解决。

https://github.com/json4s/json4s/issues/623

在 build.gradle 中添加以下依赖项

configurations.all {
    resolutionStrategy {
       // https://mvnrepository.com/artifact/org.json4s/json4s-ast
       force "org.json4s:json4s-ast_2.11:3.5.3"

      // https://mvnrepository.com/artifact/org.json4s/json4s-native
      force "org.json4s:json4s-native_2.11:3.5.3"
    }
}