【发布时间】：2021-12-13 05:54:34
【问题描述】：
按照this website 中的说明，我正在尝试通过 REST API /v1/submissions 向 Spark 提交作业。
我尝试在示例中提交SparkPi：
$ ./create.sh
{
"action" : "CreateSubmissionResponse",
"message" : "Driver successfully submitted as driver-20211212044718-0003",
"serverSparkVersion" : "3.1.2",
"submissionId" : "driver-20211212044718-0003",
"success" : true
}
$ ./status.sh driver-20211212044718-0003
{
"action" : "SubmissionStatusResponse",
"driverState" : "SUBMITTED",
"serverSparkVersion" : "3.1.2",
"submissionId" : "driver-20211212044718-0003",
"success" : true
}
create.sh:
curl -X POST http://172.17.197.143:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{
"appResource": "/home/ruc/spark-3.1.2/examples/jars/spark-examples_2.12-3.1.2.jar",
"sparkProperties": {
"spark.master": "spark://172.17.197.143:7077",
"spark.driver.memory": "1g",
"spark.driver.cores": "1",
"spark.app.name": "REST API - PI",
"spark.jars": "/home/ruc/spark-3.1.2/examples/jars/spark-examples_2.12-3.1.2.jar",
"spark.driver.supervise": "true"
},
"clientSparkVersion": "3.1.2",
"mainClass": "org.apache.spark.examples.SparkPi",
"action": "CreateSubmissionRequest",
"environmentVariables": {
"SPARK_ENV_LOADED": "1"
},
"appArgs": [
"400"
]
}'
status.sh:
export DRIVER_ID=$1
curl http://172.17.197.143:6066/v1/submissions/status/$DRIVER_ID
但是当我尝试获取作业的状态时（即使在几分钟后），我得到的是“已提交”而不是“正在运行”或“已完成”。
然后我查了一下日志，发现
21/12/12 04:47:18 INFO master.Master: Driver submitted org.apache.spark.deploy.worker.DriverWrapper
21/12/12 04:47:18 WARN master.Master: Driver driver-20211212044718-0003 requires more resource than any of Workers could have.
# ...
21/12/12 04:49:02 WARN master.Master: Driver driver-20211212044718-0003 requires more resource than any of Workers could have.
但是，在我的spark-env.sh 中，我有
export SPARK_WORKER_MEMORY=10g
export SPARK_WORKER_CORES=2
我不知道发生了什么。怎样才能让它正常运行？
【问题讨论】：
-
确保您有 10G 内存作为 spark-env.sh，并尝试将 appArgs 从 400 减少到 10。看来您没有资源来运行应用程序。
-
@Yassine Abdul-Rahman
/proc/meminfo表明我至少有 4GB 的可用内存，然后我将 spark-env.sh 更改为 1g 并将 appArgs 减少到 10。但它仍然没有工作。 -
如果这个 spark server 有一个 UI，你能不能去 executors 选项卡检查驱动程序和 worker 是否正确产生了足够的线程来处理？
标签： rest apache-spark