无法在python中以纱线集群模式读取jceks文件

Question

我正在使用jceks文件解密我的密码，在yarn集群模式下无法读取加密密码

我尝试过不同的方法，比如包含

spark-submit --deploy-mode cluster 
--file /localpath/credentials.jceks#credentials.jceks
--conf spark.hadoop.hadoop.security.credential.provider.path=jceks://file////localpath/credentials.jceks test.py

spark1 = SparkSession.builder.appName("xyz").master("yarn").enableHiveSupport().config("hive.exec.dynamic.partition", "true").config("hive.exec.dynamic.partition.mode", "nonstrict").getOrCreate()
x = spark1.sparkContext._jsc.hadoopConfiguration()
x.set("hadoop.security.credential.provider.path", "jceks://file///credentials.jceks")
a = x.getPassword("<password alias>")
passw = ""
for i in range(a.__len__()):
   passw = passw + str(a.__getitem__(i))

我收到以下错误：

attributeError: 'NoneType' 对象没有属性 'len'

当我打印一个时，它没有

Answer 1

FWIW，如果你尝试将你的 jceks 文件放到 hdfs 上，yarn worker 将能够在集群模式下运行时找到它，至少它对我有用。希望它对你有用。

hadoop fs -put ~/.jceks /user/<uid>/.jceks

spark1 = SparkSession.builder.appName("xyz").master("yarn").enableHiveSupport().config("hive.exec.dynamic.partition", "true").config("hive.exec.dynamic.partition.mode", "nonstrict").getOrCreate()
x = spark1.sparkContext._jsc.hadoopConfiguration()
jceks_hdfs_path = "jceks://hdfs@<host>/user/<uid>/.jceks"
x.set("hadoop.security.credential.provider.path", jceks_hdfs_path)
a = x.getPassword("<password alias>")
passw = ""
for i in range(a.__len__()):
   passw = passw + str(a.__getitem__(i))

运行 spark-submit 时，您无需在参数中指定 --files 和 --conf。希望对您有所帮助。

Answer 2

可以这样引用jceks

#从 spark staging 目录中选择 jceks

jceks_location=“jceks://“ + str(“/user/your_user_name/.sparkStaging/“ + str(spark.sparkContest.applicationId) + “/credentials.jceks“)

x.set("hadoop.security.credential.provider.path",jceks_location)