java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class答案

【问题标题】：java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$classjava.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class
【发布时间】：2017-03-06 02:31:57
【问题描述】：

我正在尝试实现 Apache kafka 和 spark 流集成这是我的python代码：

from __future__ import print_function
import sys
from pyspark.streaming import StreamingContext
from pyspark import SparkContext,SparkConf
from pyspark.streaming.kafka import KafkaUtils

if __name__ == "__main__":
#conf = SparkConf().setAppName("Kafka-Spark").setMaster("spark://127.0.0.1:7077")
conf = SparkConf().setAppName("Kafka-Spark")
#sc = SparkContext(appName="KafkaSpark")
sc = SparkContext(conf=conf)
stream=StreamingContext(sc,1)
map1={'demo':1}
kafkaStream = KafkaUtils.createStream(stream, 'localhost:2181', "test-consumer-group", map1)

# kafkaStream = KafkaUtils.createStream(stream, 'localhost:2181', "name", map1) #tried with localhost:2181 too
lines = kafkaStream.map(lambda x: x[1])
counts = lines.flatMap(lambda line: line.split(" ")) 
     .map(lambda word: (word, 1)) \
     .reduceByKey(lambda a, b: a+b)
counts.pprint()

stream.start()
stream.awaitTermination()

当我运行上面的程序时，它会在终端上显示输出：

16/10/24 15:27:20 错误执行程序：阶段 0.0 (TID 0) 中任务 0.0 中的异常 java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class 在 kafka.utils.Pool.(Pool.scala:28) 在 kafka.consumer.ZookeeperConsumerConnector.(ZookeeperConsumerConnector.scala:91) 在 kafka.consumer.ZookeeperConsumerConnector.(ZookeeperConsumerConnector.scala:143) 在 kafka.consumer.Consumer$.create(ConsumerConnector.scala:94) 在 org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:100) 在 org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:149) 在 org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:131) 在 org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:597) 在 org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:587) 在 org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1993) 在 org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1993) 在 org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) 在 org.apache.spark.scheduler.Task.run(Task.scala:86) 在 org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) 在 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 在 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 在 java.lang.Thread.run(Thread.java:745) 引起：java.lang.ClassNotFoundException：scala.collection.GenTraversableOnce$class 在 java.net.URLClassLoader.findClass(URLClassLoader.java:381) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:424) 在 sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 17 更多 24 年 16 月 10 日 15:27:20 错误 SparkUncaughtExceptionHandler：线程线程中未捕获的异常 [执行器任务启动 worker-0,5，main] java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class 在 kafka.utils.Pool.(Pool.scala:28) 在 kafka.consumer.ZookeeperConsumerConnector.(ZookeeperConsumerConnector.scala:91) 在 kafka.consumer.ZookeeperConsumerConnector.(ZookeeperConsumerConnector.scala:143) 在 kafka.consumer.Consumer$.create(ConsumerConnector.scala:94) 在 org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:100) 在 org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:149) 在 org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:131) 在 org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:597) 在 org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:587) 在 org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1993) 在 org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1993) 在 org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) 在 org.apache.spark.scheduler.Task.run(Task.scala:86) 在 org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) 在 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 在 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 在 java.lang.Thread.run(Thread.java:745) 引起：java.lang.ClassNotFoundException：scala.collection.GenTraversableOnce$class 在 java.net.URLClassLoader.findClass(URLClassLoader.java:381) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:424) 在 sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 17 更多 16/10/24 15:27:20 INFO StreamingContext：从关闭挂钩调用停止（stopGracefully=false） 16/10/24 15:27:20 WARN TaskSetManager: 在阶段 0.0 (TID 0, localhost) 丢失任务 0.0: java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class 在 kafka.utils.Pool.(Pool.scala:28) 在 kafka.consumer.ZookeeperConsumerConnector.(ZookeeperConsumerConnector.scala:91) 在 kafka.consumer.ZookeeperConsumerConnector.(ZookeeperConsumerConnector.scala:143) 在 kafka.consumer.Consumer$.create(ConsumerConnector.scala:94) 在 org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:100) 在 org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:149) 在 org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:131) 在 org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:597) 在 org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:587) 在 org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1993) 在 org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1993) 在 org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) 在 org.apache.spark.scheduler.Task.run(Task.scala:86) 在 org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) 在 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 在 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 在 java.lang.Thread.run(Thread.java:745) 引起：java.lang.ClassNotFoundException：scala.collection.GenTraversableOnce$class 在 java.net.URLClassLoader.findClass(URLClassLoader.java:381) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:424) 在 sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 17 更多

16/10/24 15:27:20 ERROR TaskSetManager: 阶段 0.0 中的任务 0 失败 1 次；

【问题讨论】：

标签： python-2.7 apache-spark apache-kafka pyspark

【解决方案1】：

Scala 2.10 和 2.11 的集合 API 不同

<dependency>
                     <groupId>org.scala-lang</groupId>
                     <artifactId>scala-library</artifactId>
                     <version>2.10.6</version>
  </dependency>

【讨论】：

【解决方案2】：

我可以确认修复是验证依赖项。

我的问题与 SpringBoot/Java 相关。对于 Python 项目，请查看使用 pip install pipdeptree 命令：https://www.geeksforgeeks.org/dependency-tree-of-a-python-module

对我来说，我升级了 SpringBoot 版本，它改变了我的 spring-kafka-test 版本，其中实习生包括 kafka 2.13.x，最终包括 scala 库。

我在项目的构建文件中使用了mvn dependency:tree，并搜索“2.12”以查找旧依赖项的来源。示例依赖树（删除了不相关的依赖）。注意 org.apache.kafka:kafka_2.11:jar:0.10.0.0 包含在 my.company.riptide.api:ness-logger:jar:1.0.0 中，但 org.springframework.kafka:spring-kafka-test:jar:2.7.9 包含更新版本 org.apache.kafka:kafka_2.13:jar:2.7.2

我的解决方案是从我的 ness-logger 依赖项中排除 kafaka_2.11，如下所示：

<dependency>
  <groupId>my.company.riptide.api</groupId>
  <artifactId>ness-logger</artifactId>
  <version>1.0.0</version>
  <exclusions>
    <exclusion>
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka_2.11</artifactId>
    </exclusion>
  </exclusions>
</dependency>

mvn dependency:tree 的截断输出：

[INFO] +- my.company.riptide.api:ness-logger:jar:1.0.0:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-validation:jar:2.5.7:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-el:jar:9.0.55:compile
[INFO] |  |  \- org.hibernate.validator:hibernate-validator:jar:6.2.0.Final:compile
[INFO] |  |     \- jakarta.validation:jakarta.validation-api:jar:2.0.2:compile
[INFO] |  +- commons-io:commons-io:jar:2.7:compile
[INFO] |  +- my.company.eis:ness-logging-package:jar:4.0.1:compile
[INFO] |  |  +- org.apache.avro:avro:jar:1.8.2:compile
[INFO] |  |  |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  |  |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  |  |  +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
[INFO] |  |  |  \- org.tukaani:xz:jar:1.5:compile
[INFO] |  |  +- org.apache.avro:avro-compiler:jar:1.8.2:compile
[INFO] |  |  |  +- org.apache.velocity:velocity:jar:1.7:compile
[INFO] |  |  |  \- joda-time:joda-time:jar:2.7:compile
[INFO] |  |  +- org.apache.kafka:kafka_2.11:jar:0.10.0.0:compile
[INFO] |  |  |  +- com.101tec:zkclient:jar:0.8:compile
[INFO] |  |  |  \- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:compile
[INFO] |  |  \- com.netflix.hystrix:hystrix-core:jar:1.5.18:compile
[INFO] |  |     +- com.netflix.archaius:archaius-core:jar:0.4.1:compile
[INFO] |  |     \- io.reactivex:rxjava:jar:1.3.8:compile
[INFO] |  +- org.springframework.boot:spring-boot-loader-tools:jar:2.5.7:compile
[INFO] |  |  \- org.apache.commons:commons-compress:jar:1.21:compile
[INFO] |  \- my.company.riptide.springboot:graceful-shutdown:jar:1.0.2:compile

[INFO] +- org.springframework.kafka:spring-kafka:jar:2.7.9:compile
[INFO] |  +- org.springframework:spring-messaging:jar:5.3.13:compile
[INFO] |  +- org.springframework:spring-tx:jar:5.3.13:compile
[INFO] |  +- org.springframework.retry:spring-retry:jar:1.3.1:compile
[INFO] |  |  \- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  +- org.apache.kafka:kafka-clients:jar:2.7.2:compile
[INFO] |  |  +- com.github.luben:zstd-jni:jar:1.4.5-6:compile
[INFO] |  |  +- org.lz4:lz4-java:jar:1.7.1:compile
[INFO] |  |  \- org.xerial.snappy:snappy-java:jar:1.1.7.7:compile
[INFO] |  \- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] +- junit:junit:jar:4.13.2:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:2.2:compile
[INFO] +- io.cucumber:cucumber-spring:jar:7.0.0:test
[INFO] |  \- org.apiguardian:apiguardian-api:jar:1.1.2:test
[INFO] +- io.cucumber:cucumber-core:jar:7.0.0:test
[INFO] |  +- io.cucumber:cucumber-gherkin:jar:7.0.0:test
[INFO] |  +- io.cucumber:cucumber-gherkin-messages:jar:7.0.0:test
[INFO] |  +- io.cucumber:messages:jar:17.1.1:test
[INFO] |  +- io.cucumber:tag-expressions:jar:4.0.2:test
[INFO] |  +- io.cucumber:cucumber-expressions:jar:13.0.1:test
[INFO] |  +- io.cucumber:datatable:jar:7.0.0:test
[INFO] |  +- io.cucumber:cucumber-plugin:jar:7.0.0:test
[INFO] |  +- io.cucumber:docstring:jar:7.0.0:test
[INFO] |  +- io.cucumber:html-formatter:jar:17.0.0:test
[INFO] |  \- io.cucumber:create-meta:jar:6.0.1:test

[INFO] +- org.springframework.kafka:spring-kafka-test:jar:2.7.9:test
[INFO] |  +- org.apache.kafka:kafka-clients:jar:test:2.7.2:test
[INFO] |  +- org.apache.kafka:kafka-streams:jar:2.7.2:test
[INFO] |  |  +- org.apache.kafka:connect-json:jar:2.7.2:test
[INFO] |  |  |  \- org.apache.kafka:connect-api:jar:2.7.2:test
[INFO] |  |  \- org.rocksdb:rocksdbjni:jar:5.18.4:test
[INFO] |  +- org.apache.kafka:kafka-streams-test-utils:jar:2.7.2:test
[INFO] |  +- org.apache.kafka:kafka_2.13:jar:2.7.2:test
[INFO] |  |  +- org.apache.kafka:kafka-raft:jar:2.7.2:test
[INFO] |  |  +- com.fasterxml.jackson.module:jackson-module-scala_2.13:jar:2.12.5:test
[INFO] |  |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-csv:jar:2.12.5:test
[INFO] |  |  +- net.sf.jopt-simple:jopt-simple:jar:5.0.4:compile
[INFO] |  |  +- com.yammer.metrics:metrics-core:jar:2.2.0:compile
[INFO] |  |  +- org.scala-lang.modules:scala-collection-compat_2.13:jar:2.2.0:test
[INFO] |  |  +- org.scala-lang.modules:scala-java8-compat_2.13:jar:0.9.1:test
[INFO] |  |  +- org.scala-lang:scala-library:jar:2.13.3:compile
[INFO] |  |  +- org.scala-lang:scala-reflect:jar:2.13.3:test
[INFO] |  |  +- com.typesafe.scala-logging:scala-logging_2.13:jar:3.9.2:test
[INFO] |  |  +- org.apache.zookeeper:zookeeper:jar:3.5.9:compile
[INFO] |  |  |  +- org.apache.zookeeper:zookeeper-jute:jar:3.5.9:compile
[INFO] |  |  |  +- org.apache.yetus:audience-annotations:jar:0.5.0:compile
[INFO] |  |  |  \- io.netty:netty-transport-native-epoll:jar:4.1.70.Final:compile
[INFO] |  |  \- commons-cli:commons-cli:jar:1.4:test
[INFO] |  +- org.apache.kafka:kafka_2.13:jar:test:2.7.2:test
[INFO] |  \- org.junit.jupiter:junit-jupiter-api:jar:5.7.2:test
[INFO] |     +- org.opentest4j:opentest4j:jar:1.2.0:test
[INFO] |     \- org.junit.platform:junit-platform-commons:jar:1.7.2:test

【讨论】：