无法使用 spark-cassandra-connector 启动 spark-shell 查找日志

Question

$SPARK_HOME/bin/spark-shell --packages datastax:spark-cassandra-connector:2.0.0-M2-s_2.11 --conf spark.cassandra.connection.host=127.0.0.1
Ivy Default Cache set to: /home/gopalkrishna/.ivy2/cache
The jars for the packages stored in: /home/gopalkrishna/.ivy2/jars
:: loading settings :: url = jar:file:/home/gopalkrishna/INSTALL/spark-2.1.0-bin-hadoop2.6/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
datastax#spark-cassandra-connector added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
    confs: [default]
    found datastax#spark-cassandra-connector;2.0.0-M2-s_2.11 in spark-packages
downloading http://dl.bintray.com/spark-packages/maven/datastax/spark-cassandra-connector/2.0.0-M2-s_2.11/spark-cassandra-connector-2.0.0-M2-s_2.11.jar ...
    [SUCCESSFUL ] datastax#spark-cassandra-connector;2.0.0-M2-s_2.11!spark-cassandra-connector.jar (7998ms)
:: resolution report :: resolve 33340ms :: artifacts dl 8002ms
    :: modules in use:
    datastax#spark-cassandra-connector;2.0.0-M2-s_2.11 from spark-packages in [default]
    ---------------------------------------------------------------------
    |                  |            modules            ||   artifacts   |
    |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
    ---------------------------------------------------------------------
    |      default     |   7   |   1   |   1   |   0   ||   1   |   1   |
    ---------------------------------------------------------------------

:: problems summary ::
:::: WARNINGS
        module not found: commons-beanutils#commons-beanutils;1.8.0

    ==== local-m2-cache: tried

      file:/home/gopalkrishna/.m2/repository/commons-beanutils/commons-beanutils/1.8.0/commons-beanutils-1.8.0.pom

      -- artifact commons-beanutils#commons-beanutils;1.8.0!commons-beanutils.jar:

      file:/home/gopalkrishna/.m2/repository/commons-beanutils/commons-beanutils/1.8.0/commons-beanutils-1.8.0.jar

    ==== local-ivy-cache: tried

      /home/gopalkrishna/.ivy2/local/commons-beanutils/commons-beanutils/1.8.0/ivys/ivy.xml

      -- artifact commons-beanutils#commons-beanutils;1.8.0!commons-beanutils.jar:

      /home/gopalkrishna/.ivy2/local/commons-beanutils/commons-beanutils/1.8.0/jars/commons-beanutils.jar

    ==== central: tried

      https://repo1.maven.org/maven2/commons-beanutils/commons-beanutils/1.8.0/commons-beanutils-1.8.0.pom

      -- artifact commons-beanutils#commons-beanutils;1.8.0!commons-beanutils.jar:

      https://repo1.maven.org/maven2/commons-beanutils/commons-beanutils/1.8.0/commons-beanutils-1.8.0.jar

    ==== spark-packages: tried

      http://dl.bintray.com/spark-packages/maven/commons-beanutils/commons-beanutils/1.8.0/commons-beanutils-1.8.0.pom

      -- artifact commons-beanutils#commons-beanutils;1.8.0!commons-beanutils.jar:

      http://dl.bintray.com/spark-packages/maven/commons-beanutils/commons-beanutils/1.8.0/commons-beanutils-1.8.0.jar

        module not found: org.joda#joda-convert;1.2

    ==== local-m2-cache: tried

      file:/home/gopalkrishna/.m2/repository/org/joda/joda-convert/1.2/joda-convert-1.2.pom

      -- artifact org.joda#joda-convert;1.2!joda-convert.jar:

      file:/home/gopalkrishna/.m2/repository/org/joda/joda-convert/1.2/joda-convert-1.2.jar

    ==== local-ivy-cache: tried

      /home/gopalkrishna/.ivy2/local/org.joda/joda-convert/1.2/ivys/ivy.xml

      -- artifact org.joda#joda-convert;1.2!joda-convert.jar:

      /home/gopalkrishna/.ivy2/local/org.joda/joda-convert/1.2/jars/joda-convert.jar

    ==== central: tried

      https://repo1.maven.org/maven2/org/joda/joda-convert/1.2/joda-convert-1.2.pom

      -- artifact org.joda#joda-convert;1.2!joda-convert.jar:

      https://repo1.maven.org/maven2/org/joda/joda-convert/1.2/joda-convert-1.2.jar

    ==== spark-packages: tried

      http://dl.bintray.com/spark-packages/maven/org/joda/joda-convert/1.2/joda-convert-1.2.pom

      -- artifact org.joda#joda-convert;1.2!joda-convert.jar:

      http://dl.bintray.com/spark-packages/maven/org/joda/joda-convert/1.2/joda-convert-1.2.jar

        module not found: joda-time#joda-time;2.3

    ==== local-m2-cache: tried

      file:/home/gopalkrishna/.m2/repository/joda-time/joda-time/2.3/joda-time-2.3.pom

      -- artifact joda-time#joda-time;2.3!joda-time.jar:

      file:/home/gopalkrishna/.m2/repository/joda-time/joda-time/2.3/joda-time-2.3.jar

    ==== local-ivy-cache: tried

      /home/gopalkrishna/.ivy2/local/joda-time/joda-time/2.3/ivys/ivy.xml

      -- artifact joda-time#joda-time;2.3!joda-time.jar:

      /home/gopalkrishna/.ivy2/local/joda-time/joda-time/2.3/jars/joda-time.jar

    ==== central: tried

      https://repo1.maven.org/maven2/joda-time/joda-time/2.3/joda-time-2.3.pom

      -- artifact joda-time#joda-time;2.3!joda-time.jar:

      https://repo1.maven.org/maven2/joda-time/joda-time/2.3/joda-time-2.3.jar

    ==== spark-packages: tried

      http://dl.bintray.com/spark-packages/maven/joda-time/joda-time/2.3/joda-time-2.3.pom

      -- artifact joda-time#joda-time;2.3!joda-time.jar:

      http://dl.bintray.com/spark-packages/maven/joda-time/joda-time/2.3/joda-time-2.3.jar

        module not found: io.netty#netty-all;4.0.33.Final

    ==== local-m2-cache: tried

      file:/home/gopalkrishna/.m2/repository/io/netty/netty-all/4.0.33.Final/netty-all-4.0.33.Final.pom

      -- artifact io.netty#netty-all;4.0.33.Final!netty-all.jar:

      file:/home/gopalkrishna/.m2/repository/io/netty/netty-all/4.0.33.Final/netty-all-4.0.33.Final.jar

    ==== local-ivy-cache: tried

      /home/gopalkrishna/.ivy2/local/io.netty/netty-all/4.0.33.Final/ivys/ivy.xml

      -- artifact io.netty#netty-all;4.0.33.Final!netty-all.jar:

      /home/gopalkrishna/.ivy2/local/io.netty/netty-all/4.0.33.Final/jars/netty-all.jar

    ==== central: tried

      https://repo1.maven.org/maven2/io/netty/netty-all/4.0.33.Final/netty-all-4.0.33.Final.pom

      -- artifact io.netty#netty-all;4.0.33.Final!netty-all.jar:

      https://repo1.maven.org/maven2/io/netty/netty-all/4.0.33.Final/netty-all-4.0.33.Final.jar

    ==== spark-packages: tried

      http://dl.bintray.com/spark-packages/maven/io/netty/netty-all/4.0.33.Final/netty-all-4.0.33.Final.pom

      -- artifact io.netty#netty-all;4.0.33.Final!netty-all.jar:

      http://dl.bintray.com/spark-packages/maven/io/netty/netty-all/4.0.33.Final/netty-all-4.0.33.Final.jar

        module not found: com.twitter#jsr166e;1.1.0

    ==== local-m2-cache: tried

      file:/home/gopalkrishna/.m2/repository/com/twitter/jsr166e/1.1.0/jsr166e-1.1.0.pom

      -- artifact com.twitter#jsr166e;1.1.0!jsr166e.jar:

      file:/home/gopalkrishna/.m2/repository/com/twitter/jsr166e/1.1.0/jsr166e-1.1.0.jar

    ==== local-ivy-cache: tried

      /home/gopalkrishna/.ivy2/local/com.twitter/jsr166e/1.1.0/ivys/ivy.xml

      -- artifact com.twitter#jsr166e;1.1.0!jsr166e.jar:

      /home/gopalkrishna/.ivy2/local/com.twitter/jsr166e/1.1.0/jars/jsr166e.jar

    ==== central: tried

      https://repo1.maven.org/maven2/com/twitter/jsr166e/1.1.0/jsr166e-1.1.0.pom

      -- artifact com.twitter#jsr166e;1.1.0!jsr166e.jar:

      https://repo1.maven.org/maven2/com/twitter/jsr166e/1.1.0/jsr166e-1.1.0.jar

    ==== spark-packages: tried

      http://dl.bintray.com/spark-packages/maven/com/twitter/jsr166e/1.1.0/jsr166e-1.1.0.pom

      -- artifact com.twitter#jsr166e;1.1.0!jsr166e.jar:

      http://dl.bintray.com/spark-packages/maven/com/twitter/jsr166e/1.1.0/jsr166e-1.1.0.jar

        module not found: org.scala-lang#scala-reflect;2.11.8

    ==== local-m2-cache: tried

      file:/home/gopalkrishna/.m2/repository/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.pom

      -- artifact org.scala-lang#scala-reflect;2.11.8!scala-reflect.jar:

      file:/home/gopalkrishna/.m2/repository/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.jar

    ==== local-ivy-cache: tried

      /home/gopalkrishna/.ivy2/local/org.scala-lang/scala-reflect/2.11.8/ivys/ivy.xml

      -- artifact org.scala-lang#scala-reflect;2.11.8!scala-reflect.jar:

      /home/gopalkrishna/.ivy2/local/org.scala-lang/scala-reflect/2.11.8/jars/scala-reflect.jar

    ==== central: tried

      https://repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.pom

      -- artifact org.scala-lang#scala-reflect;2.11.8!scala-reflect.jar:

      https://repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.jar

    ==== spark-packages: tried

      http://dl.bintray.com/spark-packages/maven/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.pom

      -- artifact org.scala-lang#scala-reflect;2.11.8!scala-reflect.jar:

      http://dl.bintray.com/spark-packages/maven/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.jar

        ::::::::::::::::::::::::::::::::::::::::::::::

        ::          UNRESOLVED DEPENDENCIES         ::

        ::::::::::::::::::::::::::::::::::::::::::::::

        :: commons-beanutils#commons-beanutils;1.8.0: not found

        :: org.joda#joda-convert;1.2: not found

        :: joda-time#joda-time;2.3: not found

        :: io.netty#netty-all;4.0.33.Final: not found

        :: com.twitter#jsr166e;1.1.0: not found

        :: org.scala-lang#scala-reflect;2.11.8: not found

        ::::::::::::::::::::::::::::::::::::::::::::::


:::: ERRORS
    Server access error at url https://repo1.maven.org/maven2/datastax/spark-cassandra-connector/2.0.0-M2-s_2.11/spark-cassandra-connector-2.0.0-M2-s_2.11.pom (javax.net.ssl.SSLException: Received fatal alert: protocol_version)

    Server access error at url https://repo1.maven.org/maven2/datastax/spark-cassandra-connector/2.0.0-M2-s_2.11/spark-cassandra-connector-2.0.0-M2-s_2.11.jar (javax.net.ssl.SSLException: Received fatal alert: protocol_version)

    Server access error at url https://repo1.maven.org/maven2/datastax/spark-cassandra-connector/2.0.0-M2-s_2.11/spark-cassandra-connector-2.0.0-M2-s_2.11-sources.jar (javax.net.ssl.SSLException: Received fatal alert: protocol_version)

    Server access error at url https://repo1.maven.org/maven2/datastax/spark-cassandra-connector/2.0.0-M2-s_2.11/spark-cassandra-connector-2.0.0-M2-s_2.11-src.jar (javax.net.ssl.SSLException: Received fatal alert: protocol_version)

    Server access error at url https://repo1.maven.org/maven2/datastax/spark-cassandra-connector/2.0.0-M2-s_2.11/spark-cassandra-connector-2.0.0-M2-s_2.11-javadoc.jar (javax.net.ssl.SSLException: Received fatal alert: protocol_version)

    Server access error at url https://repo1.maven.org/maven2/commons-beanutils/commons-beanutils/1.8.0/commons-beanutils-1.8.0.pom (javax.net.ssl.SSLException: Received fatal alert: protocol_version)

    Server access error at url https://repo1.maven.org/maven2/commons-beanutils/commons-beanutils/1.8.0/commons-beanutils-1.8.0.jar (javax.net.ssl.SSLException: Received fatal alert: protocol_version)

    Server access error at url https://repo1.maven.org/maven2/org/joda/joda-convert/1.2/joda-convert-1.2.pom (javax.net.ssl.SSLException: Received fatal alert: protocol_version)

    Server access error at url https://repo1.maven.org/maven2/org/joda/joda-convert/1.2/joda-convert-1.2.jar (javax.net.ssl.SSLException: Received fatal alert: protocol_version)

    Server access error at url https://repo1.maven.org/maven2/joda-time/joda-time/2.3/joda-time-2.3.pom (javax.net.ssl.SSLException: Received fatal alert: protocol_version)

    Server access error at url https://repo1.maven.org/maven2/joda-time/joda-time/2.3/joda-time-2.3.jar (javax.net.ssl.SSLException: Received fatal alert: protocol_version)

    Server access error at url https://repo1.maven.org/maven2/io/netty/netty-all/4.0.33.Final/netty-all-4.0.33.Final.pom (javax.net.ssl.SSLException: Received fatal alert: protocol_version)

    Server access error at url https://repo1.maven.org/maven2/io/netty/netty-all/4.0.33.Final/netty-all-4.0.33.Final.jar (javax.net.ssl.SSLException: Received fatal alert: protocol_version)

    Server access error at url https://repo1.maven.org/maven2/com/twitter/jsr166e/1.1.0/jsr166e-1.1.0.pom (javax.net.ssl.SSLException: Received fatal alert: protocol_version)

    Server access error at url https://repo1.maven.org/maven2/com/twitter/jsr166e/1.1.0/jsr166e-1.1.0.jar (javax.net.ssl.SSLException: Received fatal alert: protocol_version)

    Server access error at url https://repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.pom (javax.net.ssl.SSLException: Received fatal alert: protocol_version)

    Server access error at url https://repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.jar (javax.net.ssl.SSLException: Received fatal alert: protocol_version)


:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: commons-beanutils#commons-beanutils;1.8.0: not found, unresolved dependency: org.joda#joda-convert;1.2: not found, unresolved dependency: joda-time#joda-time;2.3: not found, unresolved dependency: io.netty#netty-all;4.0.33.Final: not found, unresolved dependency: com.twitter#jsr166e;1.1.0: not found, unresolved dependency: org.scala-lang#scala-reflect;2.11.8: not found]
    at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1078)
    at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:296)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:160)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Answer 1

但是你为什么使用 2.0.0-M2 版本呢？它已有 3.5 年历史，是新功能的预览版，不应该在生产环境中使用...

您需要使用 connector documentation 中指出的最新版本之一，例如：

bin/spark-shell --packages com.datastax.spark:spark-cassandra-connector_2.11:2.4.2

附： Spark 包站点有时会出现问题，因此您可以尝试 2.4.0 而不是 2.4.2