找不到文件 - 用于在 ec2 上提交 Apache Spark 作业的 API

Question

上下文 - 目标 - 来自向 Spark EC2 集群提交 Spark 作业的任何机器的 API 调用 作业运行得非常好 - 本地主机上运行的 Python 文件 - Apache Spark 但是，无法在 Apache Spark EC2 上运行它。

澄清 -

Submitting jobs to Spark EC2 cluster remotely 指远程提交作业到 Spark EC2 -（但不是通过 API 调用）

API 调用

 curl -X POST http://ec2-54-209-108-127.compute-1.amazonaws.com:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{
  "action" : "CreateSubmissionRequest",
  "appArgs" : [ "" ],
  "appResource" : "wordcount.py",
  "clientSparkVersion" : "1.5.0",
  "environmentVariables" : {
    "SPARK_ENV_LOADED" : "1"
  },
  "mainClass" : "",
  "sparkProperties" : {
    "spark.jars" : "wordcount.py",
    "spark.driver.supervise" : "true",
    "spark.app.name" : "MyJob",
    "spark.eventLog.enabled": "true",
    "spark.submit.deployMode" : "cluster",
    "spark.master" : "spark://ec2-54-209-108-127.compute-1.amazonaws.com:6066"
  }}'
{
  "action" : "CreateSubmissionResponse",
  "message" : "Driver successfully submitted as driver-20160712145703-0003",
  "serverSparkVersion" : "1.6.1",
  "submissionId" : "driver-20160712145703-0003",
  "success" : true
}

为了获得响应，以下 API 返回错误 - 找不到文件

curl  http://ec2-54-209-108-127.compute-1.amazonaws.com:6066/v1/submissions/status/driver-20160712145703-0003
{
  "action" : "SubmissionStatusResponse",
  "driverState" : "ERROR",
  "message" : "Exception from the cluster:\njava.io.FileNotFoundException: wordcount.py (No such file or directory)\n\tjava.io.FileInputStream.open(Native Method)\n\tjava.io.FileInputStream.<init>(FileInputStream.java:146)\n\torg.spark-project.guava.io.Files$FileByteSource.openStream(Files.java:124)\n\torg.spark-project.guava.io.Files$FileByteSource.openStream(Files.java:114)\n\torg.spark-project.guava.io.ByteSource.copyTo(ByteSource.java:202)\n\torg.spark-project.guava.io.Files.copy(Files.java:436)\n\torg.apache.spark.util.Utils$.org$apache$spark$util$Utils$$copyRecursive(Utils.scala:539)\n\torg.apache.spark.util.Utils$.copyFile(Utils.scala:510)\n\torg.apache.spark.util.Utils$.doFetchFile(Utils.scala:595)\n\torg.apache.spark.util.Utils$.fetchFile(Utils.scala:394)\n\torg.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:150)\n\torg.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:79)",
  "serverSparkVersion" : "1.6.1",
  "submissionId" : "driver-20160712145703-0003",
  "success" : true,
  "workerHostPort" : "172.31.17.189:59433",
  "workerId" : "worker-20160712083825-172.31.17.189-59433"
}

等待建议和改进。 p.s. - Apache Spark 的新手..

更新API调用（设置主类、appArgs、appResource、clientSparkVersion为更新值）->

curl -X POST http://ec2-54-209-108-127.compute-1.amazonaws.com:6066/v1/submissions/create{
"action" : "CreateSubmissionRequest",
"appArgs" : [ "/wordcount.py" ],
"appResource" : "file:/wordcount.py",
"clientSparkVersion" : "1.6.1",
"environmentVariables" : {
"SPARK_ENV_LOADED" : "1"
},
"mainClass" : "org.apache.spark.deploy.SparkSubmit",
"sparkProperties" : {
"spark.driver.supervise" : "false",
"spark.app.name" : "Simple App",
"spark.eventLog.enabled": "true",
"spark.submit.deployMode" : "cluster",
"spark.master" : "spark://ec2-54-209-108-127.compute-1.amazonaws.com:6066"
}
}

Answer 1

显然，错误意味着 在 EC2 实例上运行的 Spark 收到提交给它的作业 它将任务委托给 Worker 由于 python 应用程序和相关文件不存在于应用程序应该运行的 Worker 上 它给出了 - 找不到文件错误 因此，必须在 EC2 中上传应用程序，然后确保使用

在所有 Worker 节点上复制该操作

sudo /root/spark-ec2/copy-dir /root/wordcount.py 
RSYNC'ing /root/wordcount.py to slaves...
ec2-54-175-163-32.compute-1.amazonaws.com

因此，文件未找到错误停止。但是重新提交Spark Job后，Submission的状态是

{
  "action": "SubmissionStatusResponse",
  "driverState": "FAILED",
  "serverSparkVersion": "1.6.1",
  "submissionId": "driver-20160713094138-0010",
  "success": true,
  "workerHostPort": "172.31.17.189:59433",
  "workerId": "worker-20160712083825-172.31.17.189-59433"
}

因此不确定确切的错误是什么，是否已解决以及新的错误是什么