Spark 0.90 独立连接被拒绝

Question

我正在使用 spark 0.90 独立模式。

当我尝试在独立模式下使用流式应用程序时，我遇到了连接被拒绝异常。

我在/etc/hosts 中添加了主机名，也尝试单独使用 IP。在这两种情况下，工人都毫无问题地向主人注册。

有没有办法解决这个问题？

14/02/28 07:15:01 INFO Master: akka.tcp://driverClient@127.0.0.1:55891 got disassociated, removing it.
14/02/28 07:15:04 INFO Master: Registering app Twitter Streaming
14/02/28 07:15:04 INFO Master: Registered app Twitter Streaming with ID app-20140228071504-0000
14/02/28 07:34:42 INFO Master: akka.tcp://spark@127.0.0.1:33688 got disassociated, removing it.
14/02/28 07:34:42 INFO LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.165.35.96%3A38903-6#-1146558090] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/02/28 07:34:42 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@10.165.35.96:8910] -> [akka.tcp://spark@127.0.0.1:33688]: Error [Association failed with [akka.tcp://spark@127.0.0.1:33688]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://spark@127.0.0.1:33688]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:33688

Answer 1

我在 Spark 中以集群模式运行时遇到了类似的问题。我的问题是服务器以主机名“fluentd:7077”而不是 FQDN 启动。我编辑了

/sbin/start-master.sh

以反映我的远程节点如何与 -ip 标志连接。

/usr/lib/jvm/jdk1.7.0_51/bin/java -cp :/home/vagrant/spark-0.9.0-incubating-bin-   hadoop2/conf:/home/vagrant/spark-0.9.0-incuba
ting-bin-hadoop2/assembly/target/scala-2.10/spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar -Dspark.akka.logLifecycleEvents=true -Djava.library.path= -Xms512m -Xmx512m org.ap
ache.spark.deploy.master.Master --ip fluentd.alex.dev --port 7077 --webui-port 8080

希望这会有所帮助。