从 EMR Spark 查询 Redshift [重复]

Question

我在主节点上启动了 spark-shell

spark-shell --jars minimal-json-0.9.4.jar spark-avro_2.11-3.1.0.jar spark-redshift_2.10-0.6.1-SNAPSHOT.jar

查询红移表

val DF = sqlContext.read.format("com.databricks.spark.redshift").option("url", jdbcURL).option("tempdir", s3TempDir).option("query", myQuery).option("temporary_aws_access_key_id", awsAccessKey).option("temporary_aws_secret_access_key", awsSecretKey).option("temporary_aws_session_token", token).load()

得到错误：

    java.lang.ClassNotFoundException: Failed to find data source: com.databricks.spark.redshift. Please find packages at https://cwiki.apache.org/confluence/display/SPARK/Third+Party+Projects
  at org.apache.spark.sql.execution.datasources.DataSource.lookupDataSource(DataSource.scala:148)
  at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:79)
  at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:79)
  at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:340)
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:149)
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:122)
  ... 54 elided
Caused by: java.lang.ClassNotFoundException: com.databricks.spark.redshift.DefaultSource
  at scala.reflect.internal.util.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:62)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$5$$anonfun$apply$1.apply(DataSource.scala:132)
  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$5$$anonfun$apply$1.apply(DataSource.scala:132)
  at scala.util.Try$.apply(Try.scala:192)
  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$5.apply(DataSource.scala:132)
  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$5.apply(DataSource.scala:132)
  at scala.util.Try.orElse(Try.scala:84)
  at org.apache.spark.sql.execution.datasources.DataSource.lookupDataSource(DataSource.scala:132)
  ... 59 more

Answer 1

你忘记了逗号：

spark-shell --jars minimal-json-0.9.4.jar,spark-avro_2.11-3.1.0.jar,spark-redshift_2.10-0.6.1-SNAPSHOT.jar