【发布时间】：2016-08-18 05:28:37
【问题描述】：
现在我正在使用 spark-submit 在集群模式下启动应用程序。来自主服务器的响应提供了一个带有 submitId 的 json 对象，我用它来识别应用程序并在必要时将其终止。但是，我还没有找到一种简单的方法来从主服务器响应或驱动程序 ID 中检索 worker rest url（可能网络可以抓取主 Web ui，但这会很丑）。相反，我必须等到应用程序完成，然后从历史服务器中查找应用程序统计信息。
有没有什么方法可以使用 driver-id 从以集群模式部署的应用程序（通常在 worker-node:4040）中识别 worker url？
16/08/12 11:39:47 INFO RestSubmissionClient: Submitting a request to launch an application in spark://192.yyy:6066.
16/08/12 11:39:47 INFO RestSubmissionClient: Submission successfully created as driver-20160812114003-0001. Polling submission state...
16/08/12 11:39:47 INFO RestSubmissionClient: Submitting a request for the status of submission driver-20160812114003-0001 in spark://192.yyy:6066.
16/08/12 11:39:47 INFO RestSubmissionClient: State of driver driver-20160812114003-0001 is now RUNNING.
16/08/12 11:39:47 INFO RestSubmissionClient: Driver is running on worker worker-20160812113715-192.xxx-46215 at 192.xxx:46215.
16/08/12 11:39:47 INFO RestSubmissionClient: Server responded with CreateSubmissionResponse:
{
"action" : "CreateSubmissionResponse",
"message" : "Driver successfully submitted as driver-20160812114003-0001",
"serverSparkVersion" : "1.6.1",
"submissionId" : "driver-20160812114003-0001",
"success" : true
}
编辑：这是 DEBUG 处 log4j 控制台输出的典型输出外观
Spark 提交命令：
./apps/spark-2.0.0-bin-hadoop2.7/bin/spark-submit --master mesos://masterurl:7077
--verbose --class MainClass --deploy-mode cluster
~/path/myjar.jar args
Spark 提交输出：
Using properties file: null
Parsed arguments:
master mesos://masterurl:7077
deployMode cluster
executorMemory null
executorCores null
totalExecutorCores null
propertiesFile null
driverMemory null
driverCores null
driverExtraClassPath null
driverExtraLibraryPath null
driverExtraJavaOptions null
supervise false
queue null
numExecutors null
files null
pyFiles null
archives null
mainClass MyApp
primaryResource file:/path/myjar.jar
name MyApp
childArgs [args]
jars null
packages null
packagesExclusions null
repositories null
verbose true
Spark properties used, including those specified through
--conf and those from the properties file null:
Main class:
org.apache.spark.deploy.rest.RestSubmissionClient
Arguments:
file:/path/myjar.jar
MyApp
args
System properties:
SPARK_SUBMIT -> true
spark.driver.supervise -> false
spark.app.name -> MyApp
spark.jars -> file:/path/myjar.jar
spark.submit.deployMode -> cluster
spark.master -> mesos://masterurl:7077
Classpath elements:
16/08/17 13:26:49 INFO RestSubmissionClient: Submitting a request to launch an application in mesos://masterurl:7077.
16/08/17 13:26:49 DEBUG RestSubmissionClient: Sending POST request to server at http://masterurl:7077/v1/submissions/create:
{
"action" : "CreateSubmissionRequest",
"appArgs" : [ args ],
"appResource" : "file:/path/myjar.jar",
"clientSparkVersion" : "2.0.0",
"environmentVariables" : {
"SPARK_SCALA_VERSION" : "2.10"
},
"mainClass" : "SimpleSort",
"sparkProperties" : {
"spark.jars" : "file:/path/myjar.jar",
"spark.driver.supervise" : "false",
"spark.app.name" : "MyApp",
"spark.submit.deployMode" : "cluster",
"spark.master" : "mesos://masterurl:7077"
}
}
16/08/17 13:26:49 DEBUG RestSubmissionClient: Response from the server:
{
"action" : "CreateSubmissionResponse",
"serverSparkVersion" : "2.0.0",
"submissionId" : "driver-20160817132658-0004",
"success" : true
}
16/08/17 13:26:49 INFO RestSubmissionClient: Submission successfully created as driver-20160817132658-0004. Polling submission state...
16/08/17 13:26:49 INFO RestSubmissionClient: Submitting a request for the status of submission driver-20160817132658-0004 in mesos://masterurl:7077.
16/08/17 13:26:49 DEBUG RestSubmissionClient: Sending GET request to server at http://masterurl:7077/v1/submissions/status/driver-20160817132658-0004.
16/08/17 13:26:49 DEBUG RestSubmissionClient: Response from the server:
{
"action" : "SubmissionStatusResponse",
"driverState" : "RUNNING",
"serverSparkVersion" : "2.0.0",
"submissionId" : "driver-20160817132658-0004",
"success" : true
}
16/08/17 13:26:49 INFO RestSubmissionClient: State of driver driver-20160817132658-0004 is now RUNNING.
16/08/17 13:26:49 INFO RestSubmissionClient: Server responded with CreateSubmissionResponse:
{
"action" : "CreateSubmissionResponse",
"serverSparkVersion" : "2.0.0",
"submissionId" : "driver-20160817132658-0004",
"success" : true
}
【问题讨论】：
标签： apache-spark