如何在集群模式下设置驱动程序python路径（pyspark）

Question

我的程序在客户端模式下运行良好，但是当我尝试在集群模式下运行时如果失败，原因是集群节点上的 python 版本不同

当我的应用程序以集群模式运行时，我正在尝试设置 python 驱动程序路径

下面是我在集群模式下的 spark 提交命令

spark-submit --master yarn --deploy-mode cluster --num-executors 10 --executor-cores 3 --driver-memory 50G --executor-memory 20G \
--conf spark.dynamicAllocation.enabled=false  \
--conf spark.kryoserializer.buffer.max=1024 --conf spark.yarn.keytab=keytab_path --conf spark.yarn.principal=${10} \
--conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=/bin/python3
--jars path_to_jars \
--py-files Pipeline.egg-info,<path>/app.py <application_path>/app.py arguments

下面是错误

22/08/04 06:09:34 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
22/08/04 06:09:34 INFO yarn.ApplicationMaster: Waiting for spark context initialization...
22/08/04 06:09:34 ERROR yarn.ApplicationMaster: User application exited with status 1
22/08/04 06:09:34 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 13, (reason: User application exited with status 1)
22/08/04 06:09:34 ERROR yarn.ApplicationMaster: Uncaught exception:
org.apache.spark.SparkException: Exception thrown in awaitResult:
        at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226)
        at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:447)
        at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:275)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:805)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:804)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
        at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:804)
        at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: org.apache.spark.SparkUserAppException: User application exited with 1
        at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:106)
        at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
22/08/04 06:09:34 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://test-scc/user/tst_rdip_cross/.sparkStaging/application_1643123069214_48871
22/08/04 06:09:35 INFO util.ShutdownHookManager: Shutdown hook called

在探索控制台日志和应用程序日志上我们没有找到错误的原因，然后我们探索了纱线日志，发现python版本在集群节点上不兼容

请有人能帮助我

提前致谢

Answer 1

当你在--conf中提交你的spark作业时，你可以使用spark.pyspark.python和spark.pyspark.driver.python来设置你的python路径，如果你使用的spark版本>=2.1.0.