java.lang.NoClassDefFoundError: org/apache/kafka/common/message/KafkaLZ4BlockOutputStream

Question

我在使用 Spark 流 API 时收到 NoClassDefFoundErrorerror。这是我的流媒体代码。

我知道这是一些缺少 jar 和依赖项的问题，但我无法弄清楚那是什么。

我正在使用 kafka 0.9.0、spark 1.6.1 - 这些依赖项是否正常，或者我需要更改它们吗？我在下面附上了 pom.xml。

这是我正在使用的流 API。

JavaPairInputDStream directKafkaStream = KafkaUtils.createDirectStream(jsc, String.class, byte[].class, StringDecoder.class, DefaultDecoder.class, kafkaParams, topicSet);

这是我的代码片段。我在 while(itr.next()) 收到错误

directKafkaStream.foreachRDD(rdd -> {

    rdd.foreachPartition(itr -> {


        try {

            while (itr.hasNext()) {

java.lang.NoClassDefFoundError: org/apache/kafka/common/message/KafkaLZ4BlockOutputStream

这是我的 POM.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>com.abcd.rep.xyz</groupId>
        <artifactId>xyz</artifactId>
        <version>1.0</version>
        <relativePath>../pom.xml</relativePath>
    </parent>
        <artifactId>SparkPOC</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>
        <name>SparkPOCde</name>
        <url>http://maven.apache.org</url>
<properties>

<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>


<spark-version>1.6.1</spark-version>

<kafka-version>0.9.0.0</kafka-version>

</properties>


<dependencies>

<!-- http://mvnrepository.com/artifact/org.springframework/spring-core -->



<!-- http://mvnrepository.com/artifact/org.springframework/spring-jdbc -->
<dependency>

<groupId>log4j</groupId>

<artifactId>log4j</artifactId>

<version>1.2.17</version>

</dependency>

<!-- http://mvnrepository.com/artifact/org.apache.spark/spark-streaming_2.10 -->

<dependency>
<groupId>org.apache.spark</groupId>

<artifactId>spark-streaming_2.10</artifactId>

<version>${spark-version}</version>

</dependency>

<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka_2.10 -->



<dependency>

<groupId>org.apache.spark</groupId>

<artifactId>spark-streaming-kafka_2.10</artifactId>

<version>1.6.2</version>

 <exclusions>
        <exclusion>
            <groupId>io.netty</groupId>
            <artifactId>netty</artifactId>
        </exclusion>

        <exclusion>
            <groupId>io.jboss.netty</groupId>
            <artifactId>netty</artifactId>
        </exclusion>
    </exclusions> 

</dependency>

   <dependency>
            <groupId>com.abcd.rep.xyz</groupId>
            <artifactId>xyzCommon</artifactId>
            <version>1.0</version>
            <type>jar</type>
        </dependency>


<!-- http://mvnrepository.com/artifact/ojdbc/ojdbc -->


<!-- <dependency> <groupId>ojdbc</groupId> <artifactId>ojdbc</artifactId> <version>14</version> </dependency>-->


<!-- https://mvnrepository.com/artifact/org.mongodb/mongo-java-driver -->




<!-- http://mvnrepository.com/artifact/org.springframework.data/spring-data-mongodb -->




<!-- https://mvnrepository.com/artifact/com.googlecode.json-simple/json-simple -->





</dependencies>


<build>

<finalName>appname</finalName>


<resources>


<resource>

<directory>src/main/resources</directory>


<excludes>

<exclude>eventRules.json</exclude>

<exclude>log4j.xml</exclude>

<exclude>resources.properties</exclude>

</excludes>

</resource>

</resources>


<plugins>


<plugin>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-compiler-plugin</artifactId>

<version>3.5.1</version>


<configuration>

<source>1.8</source>

<target>1.8</target>

</configuration>

</plugin>


<plugin>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-shade-plugin</artifactId>

<version>2.4.3</version>


<executions>


<execution>

<phase>package</phase>


<goals>

<goal>shade</goal>

</goals>


<configuration>


<filters>


<filter>

<artifact>*:*</artifact>


-<excludes>

<exclude>META-INF/*.SF</exclude>

<exclude>META-INF/*.DSA</exclude>

<exclude>META-INF/*.RSA</exclude>

</excludes>

</filter>

</filters>


<transformers>


-<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">

<resource>reference.conf</resource>

</transformer>


-<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">

<mainClass>com.abcd.rep.xyz.SparkPOCde.EventConsumerServiceImpl</mainClass>

</transformer>

</transformers>

</configuration>

</execution>

</executions>

</plugin>

</plugins>

</build>

</project>

Answer 1

我使用版本 0.8.2.2 的 kafka jar 来解决此问题。

Answer 2

虽然我的 kafka 集群版本是 0.9.0.0 。 我使用这样的 maven pom 来处理带有 Spark Streaming 的 kafka。

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.10</artifactId>
<version>1.6.0</version>
</dependency>

，但我收到上述错误。 然后我尝试按如下方式添加依赖，它可以工作。

      <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.10</artifactId>
        <version>0.8.2.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>0.8.2.1</version>
    </dependency>

Answer 3

KafkaLZ4BlockOutputStream 在 kafka-clients jar 中。

直到 kafka-clients 版本 0.8.2.2 它位于 org/apache/kafka/common/message/KafkaLZ4BlockOutputStream

从 0.9.0.0 开始，它位于 /org/apache/kafka/common/record/