无法在 datastax 上启动 spark答案

【问题标题】：cannot start the spark on datastax无法在 datastax 上启动 spark
【发布时间】：2015-07-25 05:50:53
【问题描述】：

我正在尝试在 datastax (4.6) 上启动火花。当我执行 ./dse spark 时，我得到了：

java.lang.SecurityException: Authentication failed! Credentials required
    at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticationFailure(JMXPluggableAuthenticator.java:211)
    at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticate(JMXPluggableAuthenticator.java:163)
    at sun.management.jmxremote.ConnectorBootstrap$AccessFileCheckerAuthenticator.authenticate(ConnectorBootstrap.java:219)
    at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:232)
    at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
    at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
    at sun.rmi.transport.Transport$1.run(Transport.java:177)
    at sun.rmi.transport.Transport$1.run(Transport.java:174)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
    at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(Unknown Source)
    at sun.rmi.transport.StreamRemoteCall.executeCall(Unknown Source)
    at sun.rmi.server.UnicastRef.invoke(Unknown Source)
    at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
    at javax.management.remote.rmi.RMIConnector.getConnection(Unknown Source)
    at javax.management.remote.rmi.RMIConnector.connect(Unknown Source)
    at javax.management.remote.JMXConnectorFactory.connect(Unknown Source)
    at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:151)
    at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:107)
    at com.datastax.bdp.tools.DseTool.<init>(DseTool.java:92)
    at com.datastax.bdp.tools.DseTool.main(DseTool.java:1048)
Welcome to
    ____              __
   / __/__  ___ _____/ /__
  _\ \/ _ \/ _ `/ __/  '_/
 /___/ .__/\_,_/_/ /_/\_\   version 1.1.0
    /_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_21)
Type in expressions to have them evaluated.
Type :help for more information.
Creating SparkContext...
Initializing SparkContext with MASTER: spark://master:7077
Created spark context..
Spark context available as sc.
15/05/18 09:27:03 WARN Remoting: Tried to associate with unreachable      remote address [akka.tcp://sparkMaster@master:7077]. Address is now gated for 60000 ms, all messages to this address will be delivered to dead letters.
15/05/18 09:27:03 WARN AppClient$ClientActor: Could not connect to     akka.tcp://sparkMaster@master:7077: akka.remote.InvalidAssociation: Invalid    address: akka.tcp://sparkMaster@master:7077
HiveSQLContext available as hc.
CassandraSQLContext available as csc.
Type in expressions to have them evaluated.
Type :help for more information.

scala> 15/05/18 09:28:03 ERROR SparkDeploySchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
15/05/18 09:28:03 ERROR TaskSchedulerImpl: Exiting due to error from cluster scheduler: All masters are unresponsive! Giving up.

奇怪的是

1，我已禁用密码授权器，但仍然得到 java.lang.SecurityException：身份验证失败！需要凭据

2，我设置了SPARK_MASTER=my_hostname，和cassandra节点是同一个主机，但是还是得到All masters are unresponsive

有人可以帮我解决这个问题吗？谢谢

【问题讨论】：

已经解决了这个问题。我使用的datastax的版本是4.6.1，在这个版本中，我们在spark/lib/中有snappy-java-1.0.5.1.jar，就是这个问题。在我将这个 jar 文件更改为 snappy-java-1.0.4.1.jar 后，它就可以工作了。

标签： cassandra apache-spark datastax

【解决方案1】：

确保为 dse_auth 密钥空间正确设置复制因子：

http://docs.datastax.com/en/datastax_enterprise/4.0/datastax_enterprise/sec/secConfSysAuthKeyspRepl.html

程序

根据您的环境，根据以下示例之一设置复制因子：简单策略示例：

ALTER KEYSPACE "system_auth"
   WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 3 };

NetworkTopologyStrategy 示例：

ALTER KEYSPACE "system_auth"
   WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'dc1' : 3, 'dc2' : 2};

【讨论】：

你使用的是$ dse -u username -p password spark还是~/.dserc文件？
我试过 -u -p 但没用。现在我已禁用身份验证。

【解决方案2】：

要通过身份验证在 dse 上启动 spark，您也可以通过这种方式传递凭据

dse spark -Dcassandra.username=user -Dcassandra.password=pass

【讨论】：