Apache Spark 工作器超时

Question

我在使用 Spark 时遇到了一个又一个的问题，我相信这与网络或权限或两者有关。 master 或 worker 日志或抛出的错误中没有任何内容表明存在问题。

15/12/29 19:19:58 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/12/29 19:20:13 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/12/29 19:20:28 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/12/29 19:20:43 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/12/29 19:20:58 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/12/29 19:21:11 INFO AppClient$ClientEndpoint: Executor updated: app-20151229141057-0000/8 is now EXITED (Command exited with code 1)
15/12/29 19:21:11 INFO SparkDeploySchedulerBackend: Executor app-20151229141057-0000/8 removed: Command exited with code 1
15/12/29 19:21:11 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 8
15/12/29 19:21:11 INFO AppClient$ClientEndpoint: Executor added: app-20151229141057-0000/10 on worker-20151229141026-127.0.0.1-48818 (127.0.0.1:48818) with 2 cores
15/12/29 19:21:11 INFO SparkDeploySchedulerBackend: Granted executor ID app-20151229141057-0000/10 on hostPort 127.0.0.1:48818 with 2 cores, 1024.0 MB RAM
15/12/29 19:21:11 INFO AppClient$ClientEndpoint: Executor updated: app-20151229141057-0000/10 is now LOADING
15/12/29 19:21:11 INFO AppClient$ClientEndpoint: Executor updated: app-20151229141057-0000/10 is now RUNNING
15/12/29 19:21:12 INFO AppClient$ClientEndpoint: Executor updated: app-20151229141057-0000/9 is now EXITED (Command exited with code 1)
15/12/29 19:21:12 INFO SparkDeploySchedulerBackend: Executor app-20151229141057-0000/9 removed: Command exited with code 1
15/12/29 19:21:12 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 9
15/12/29 19:21:12 INFO AppClient$ClientEndpoint: Executor added: app-20151229141057-0000/11 on worker-20151229141023-127.0.0.1-35452 (127.0.0.1:35452) with 2 cores
15/12/29 19:21:12 INFO SparkDeploySchedulerBackend: Granted executor ID app-20151229141057-0000/11 on hostPort 127.0.0.1:35452 with 2 cores, 1024.0 MB RAM
15/12/29 19:21:12 INFO AppClient$ClientEndpoint: Executor updated: app-20151229141057-0000/11 is now LOADING
15/12/29 19:21:12 INFO AppClient$ClientEndpoint: Executor updated: app-20151229141057-0000/11 is now RUNNING
15/12/29 19:21:13 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

我正在尝试使用 spark 1.52 在 Ubuntu 14.04 上运行独立设置。一切似乎都配置正确，但工作似乎永远不会完成，每个工作人员都超时了。

这是我正在执行作业的远程机器...

代码只是他们的示例之一。我也尝试过 Pi 估计示例并且遇到了同样的问题。

def main(args: Array[String]) {
  val logFile = "/Users/user/spark.txt" // Should be some file on your system
  val conf = new SparkConf().setAppName("Simple App").setMaster("spark://46.101.xxx.xxx:7077")
  val sc = new SparkContext(conf)
  val logData = sc.textFile(logFile, 2).cache()
  val numAs = logData.filter(line => line.contains("a")).count()
  val numBs = logData.filter(line => line.contains("b")).count()
  println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
}

以前有人遇到过这个问题吗？如果有人能让我找到正确的方向来解决这个问题，我将不胜感激。

--edit - 附加信息。

#spark-env.sh
export SPARK_LOCAL_IP="46.101.xxx.xxx"
export SPARK_MASTER_IP="46.101.xxx.xxx"
export SPARK_PUBLIC_DNS="46.101.xxx.xxx"

使用 Scala 2.10.6 和 2.11.latest 尝试了 Java 7 和 Java 8。

Master 以 ./start-master.sh 开头 Worker 以 ./start-slave.sh spark://46.101.xxx.xxx:7077 开始

在 Ubuntu 14.04.3 LTS 上运行。 （数字海洋） - 没有防火墙。可以从远程机器telnet到master和worker。 Master和Worker都在同一台机器上。

测试了 Spark 1.5.2 和 1.5.0。在客户端计算机（请求）和远程服务器（主服务器和工作服务器）之间保持 Java、Scala 和 Spark 的版本相同。

Answer 1

您的应用程序似乎找不到工作人员。当您启动集群时，您是否启动了任何从属服务器并将它们连接到主服务器？

要启动您的工作人员并将它们连接到主服务器，请运行以下命令：

./bin/spark-class org.apache.spark.deploy.worker.Worker spark://ip:port

spark://ip:port 是主人的。