部署时出错：找不到类 org.apache.hadoop.fs.LocalFileSystem

Question

我正在尝试在 Scala/sbt 中编写 parquet 文件。我的代码在我的计算机上运行良好，但在使用 Jenkins 部署在服务器上时总是失败。 我有以下错误：

java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.LocalFileSystem not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2688)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3431)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:288)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
at org.apache.parquet.hadoop.util.HadoopOutputFile.fromPath(HadoopOutputFile.java:58)
at org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:677)
at com.github.mjakubowski84.parquet4s.ParquetWriter$.internalWriter(ParquetWriter.scala:129)
at com.github.mjakubowski84.parquet4s.ParquetWriterImpl.<init>(ParquetWriter.scala:186)
at com.github.mjakubowski84.parquet4s.ParquetWriter$BuilderImpl.build(ParquetWriter.scala:111)
at com.github.mjakubowski84.parquet4s.ParquetWriter$BuilderImpl.writeAndClose(ParquetWriter.scala:113)
at ParquetExport$.$anonfun$tryExport$1(ParquetExport.scala:307)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:658)
at scala.util.Success.$anonfun$map$1(Try.scala:255)
at scala.util.Success.map(Try.scala:213)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.LocalFileSystem not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2592)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2686)
... 29 more

我第一次尝试使用spark：

libraryDependencies += "org.apache.spark" %% "spark-core" % "3.3.0"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.3.0"

然后更改为使用这些的代码：

libraryDependencies ++= Seq(
"com.github.mjakubowski84" %% "parquet4s-core" % "2.6.0",
"org.apache.hadoop" % "hadoop-client" % "2.10.2"
)

仍然遇到同样的错误。

设置 Hadoop 配置没有帮助：

val hadoopConfig = new Configuration()
hadoopConfig.set("fs.hdfs.impl", classOf[org.apache.hadoop.hdfs.DistributedFileSystem].getname)
hadoopConfig.set("fs.hdfs.impl", classOf[org.apache.hadoop.fs.LocalFileSystem].getname)

也没有改变 ClassLoader： Thread.currentThread.setContextClassLoader(getClass.getClassLoader)

在本地一切正常，但在服务器上却不行。有什么想法吗？

Answer 1

我已经遇到了同样的问题。

如果您查看日志：

Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.LocalFileSystem not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2592)

您知道它找不到您为 hadoop 配置设置的类 LocalFileSystem 的名称。

尝试改变这一点：

val hadoopConfig = new Configuration()
hadoopConfig.set("fs.hdfs.impl", classOf[org.apache.hadoop.hdfs.DistributedFileSystem].getname)
hadoopConfig.set("fs.hdfs.impl", classOf[org.apache.hadoop.fs.LocalFileSystem].getname)

收件人：

 val hadoopConfig = new Configuration()
 hadoopConfig.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.DistributedFileSystem")
 hadoopConfig.set("fs.file.impl", "org.apache.hadoop.fs.LocalFileSystem")