【发布时间】：2019-10-31 19:12:40
【问题描述】：
我正在使用 IntelliJ ide 和语言 scala，我想使用 IAM 用户凭证访问存储在 AWS S3 中的文本文件。我没有仅使用依赖项在我的系统上下载 Hadoop。我已经使用 Aws 依赖项和 jets3t 依赖项完成了这项工作。但我想用火花来做。
我得到的基本错误是：
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found,
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3.S3FileSystem not found
// and similarly
Class org.apache.hadoop.fs.s3native.NativeS3FileSystem not found.
请帮我解决这个问题。
我尝试独立添加各种 Hadoop 依赖项、hadoop-aws 依赖项，但每个都给了我不同的错误。 就像添加 "org.apache.hadoop" % "hadoop-aws" % "3.2.0" 说的：
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StreamCapabilities
at java.lang.ClassLoader.defineClass1(Native Method)
添加 "org.apache.hadoop" % "hadoop-common" % "3.1.1" 说：
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.spark.SparkContext.withScope(SparkContext.scala:699)
at org.apache.spark.SparkContext.textFile(SparkContext.scala:828)
at spark_scala_s3$.main(spark_scala_s3.scala:40).
for the line:
val df = sc.textFile(s"s3a://my-week6-spark/$path").
添加 "org.apache.hadoop" % "hadoop-aws" % "2.7.3" 说：
com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400,
AWS Service: Amazon S3, AWS Request ID: 90C42E72BEEB31FB, AWS Error Code: null, AWS Error Message: Bad Request, S3 Extended Request ID: (someid).
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: AKIAUAOZGFGM2K5WEHFC:l0IHiYq4ApJEewbjKR00KwKA+Ra)
and
The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.
代码：
// Dependencies in sbt:
version := "0.1"
scalaVersion := "2.12.8"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.3"
// and others I added to see what works but gave errors as above
实际的scala文件代码
// just 1 import
import org.apache.spark.sql.SparkSession
val s = SparkSession.builder().appName("trial2").master("local").getOrCreate()
val sc = s.sparkContext
var accessKeyId: String = "acc key"
val secretAccessKey: String = "secret acc key"
sc.hadoopConfiguration.set("fs.s3.awsAccessKeyId", accessKeyId)
sc.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", accessKeyId)
sc.hadoopConfiguration.set("fs.s3a.access.key", accessKeyId)
sc.hadoopConfiguration.set("fs.s3.awsSecretAccessKey",
secretAccessKey)
sc.hadoopConfiguration.set("fs.s3n.awsSecretAccessKey",
secretAccessKey)
sc.hadoopConfiguration.set("fs.s3a.secret.key",secretAccessKey)
sc.hadoopConfiguration.set("fs.s3n.impl",
"org.apache.hadoop.fs.s3native.NativeS3FileSystem")
sc.hadoopConfiguration.set("fs.s3a.impl",
"org.apache.hadoop.fs.s3a.S3AFileSystem")
sc.hadoopConfiguration.set("fs.s3.impl",
"org.apache.hadoop.fs.s3.S3FileSystem")
try{
val df = sc.textFile(s"s3a://my-week6-spark/$path")
// my-week6-spark is bucket name
// path consist of file name
println("DF.show() 1\n",df.collect())
catch {
case exception: Exception => println("1 failed as
",exception)
}
try{
val df = sc.textFile(s"s3n://my-week6-spark/$path")
println("DF.show() 2\n",df.collect())
}catch {
case exception: Exception => println("2 failed as ", exception)
}
try{
val df = sc.textFile(s"s3://my-week6-spark/$path")
println("DF.show() 3\n",df.collect())
}catch {
case exception: Exception => println("3 failed as ", exception)
}
}
我希望文件的内容会被访问并按照它打印一些内容。但我得到了错误。
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StreamCapabilities
at java.lang.ClassLoader.defineClass1(Native Method)
如上所述，每个依赖项都会显示不同的错误。
我在网上找到了一个解决方案，它说设置 Hadoop 路径： 作为： 导出 hadoop_path = 一些路径。
但由于我没有安装 Hadoop，我无法提供安装它的任何路径。
【问题讨论】：
标签： scala apache-spark hadoop