Spark NoClassDeftFoundError 上的 Apache Tika 1.11

Question

我正在尝试在 Spark 上使用 apache tika。但是，我遇到了配置问题。目前我最好的猜测是依赖项（其中 tika 有很多......）没有与 JAR 捆绑在一起以获取 spark。 如果这种直觉是正确的，我不确定最好的前进道路是什么。但我也不确定这是否是我的问题。

以下是一个非常简单的 spark 作业，它可以编译，但在到达 Tika 实例化时会遇到运行时错误。

我的pom.xml如下：

<project>
  <groupId>tika.test</groupId>
  <artifactId>tikaTime</artifactId>
  <modelVersion>4.0.0</modelVersion>
  <name>TikaTime</name>
  <packaging>jar</packaging>
  <version>1.0</version>
  <dependencies>
    <dependency> <!-- Spark dependency -->
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.10</artifactId>
      <version>1.5.2</version>
    </dependency>
    <dependency>
      <groupId>org.apache.tika</groupId>
      <artifactId>tika-parsers</artifactId>
      <version>1.11</version>
    </dependency>
    <dependency>
      <groupId>org.apache.tika</groupId>
      <artifactId>tika-core</artifactId>
      <version>1.11</version>
    </dependency>
  </dependencies>
</project>

我的示例代码在这里：

/* TikaTime.java */
import org.apache.spark.api.java.*;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.Function;
import java.io.FileInputStream;
import java.io.InputStream;
import java.io.File;
import java.io.IOException;
import org.apache.tika.*;

public class TikaTime {
  public static void main(String[] args) throws IOException {

    String logFile = "file.txt";
    File logfile = new File("/home/file.txt");
    SparkConf conf = new SparkConf().setAppName("TikaTime");
    JavaSparkContext sc = new JavaSparkContext(conf);
    JavaRDD<String> logData = sc.textFile(logFile).cache();

    long numAs = logData.filter(new Function<String, Boolean>() {
      public Boolean call(String s) { return s.contains("a"); }
    }).count();

    long numBs = logData.filter(new Function<String, Boolean>() {
      public Boolean call(String s) { return s.contains("b"); }
    }).count();

    System.out.println("Lines with a: " + numAs + ", lines with b: " + numBs);

    //Tika facade class.
    Tika tika = new Tika();
  }
}

错误堆栈跟踪如下：

    Lines with a: 2, lines with b: 1
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/tika/Tika
    at TikaTime.main(TikaTime.java:32)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.tika.Tika
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 10 more

想知道其他人之前是否遇到过这个问题。我很少使用 Maven，而且对 Spark 也有些陌生，所以我不确定我的直觉是否正确。

编辑：包括我的 spark 提交语法，以防它感兴趣。

~/spark151/spark-1.5.1/bin/spark-submit --class "TikaTime" --master local[4] target/tikaTime-1.0.jar

Answer 1

根据 Gagravarr 的回复和我最初的怀疑，问题是需要将 uber-jar 提供给 Spark。这是使用 maven-shade 插件完成的。新的pom.xml 如下所示。

<project>
<groupId>tika.test</groupId>
<artifactId>tikaTime</artifactId>
<modelVersion>4.0.0</modelVersion>
<name>TikaTime</name>
<packaging>jar</packaging>
<version>1.0</version>
<dependencies>
  <dependency> <!-- Spark dependency -->
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.10</artifactId>
    <version>1.5.2</version>
  </dependency>
  <dependency>
    <groupId>org.apache.tika</groupId>
    <artifactId>tika-parsers</artifactId>
    <version>1.11</version>
  </dependency>
  <dependency>
    <groupId>org.apache.tika</groupId>
    <artifactId>tika-core</artifactId>
    <version>1.11</version>
  </dependency>
</dependencies>
<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-shade-plugin</artifactId>
      <version>2.4.2</version>
      <executions>
        <execution>
          <phase>package</phase>
          <goals>
            <goal>shade</goal>
          </goals>
        </execution>
      </executions>
      <configuration>
          <filters>
            <filter>
              <artifact>*:*</artifact>
              <excludes>
                <exclude>META-INF/*.SF</exclude>
                <exclude>META-INF/*.DSA</exclude>
                <exclude>META-INF/*.RSA</exclude>
              </excludes>
            </filter>
          </filters>
          <finalName>uber-${project.artifactId}-${project.version}</finalName>
      </configuration>
    </plugin>
  </plugins>
</build>
</project>

注意：您还必须将由此创建的 uber-jar 提交给 spark 而不是原始的。