【发布时间】:2022-11-29 18:55:26
【问题描述】:
我想使用不重复的代码(Python 函数)创建如下所示的 JSON Python 列表。
预期输出:
Steps=[
{
'Name': 'Download Config File',
'ActionOnFailure': 'CONTINUE',
'HadoopJarStep': {
'Jar': 'command-runner.jar',
'Args': [
'aws',
's3',
'cp',
's3://sample-qa/jars/Loads/',
'/home/hadoop/',
'--recursive'
]
}
},
{
'Name': 'Spark Job',
'ActionOnFailure': 'CONTINUE',
'HadoopJarStep': {
'Jar': 'command-runner.jar',
'Args': [
'spark-submit',
'--deploy-mode', 'cluster',
'--executor-memory', '10g',
'--conf', 'spark.serializer=org.apache.spark.serializer.KryoSerializer',
'--conf', 'spark.sql.hive.convertMetastoreParquet=false',
'--master', 'yarn',
'--class', 'com.general.Loads',
'/home/hadoop/Loads-assembly-0.1.jar',
'20120304',
'sample-qa'
]
}
}
{
'Name': 'Spark Job',
'ActionOnFailure': 'CONTINUE',
'HadoopJarStep': {
'Jar': 'command-runner.jar',
'Args': [
'spark-submit',
'--deploy-mode', 'cluster',
'--executor-memory', '10g',
'--conf', 'spark.serializer=org.apache.spark.serializer.KryoSerializer',
'--conf', 'spark.sql.hive.convertMetastoreParquet=false',
'--master', 'yarn',
'--class', 'com.general.Loads',
'/home/hadoop/Loads-assembly-0.1.jar',
'20220130',
'sample-qa'
]
}
},
{
'Name': 'Spark Job',
'ActionOnFailure': 'CONTINUE',
'HadoopJarStep': {
'Jar': 'command-runner.jar',
'Args': [
'spark-submit',
'--deploy-mode', 'cluster',
'--executor-memory', '10g',
'--conf', 'spark.serializer=org.apache.spark.serializer.KryoSerializer',
'--conf', 'spark.sql.hive.convertMetastoreParquet=false',
'--master', 'yarn',
'--class', 'com.general.Loads',
'/home/hadoop/Loads-assembly-0.1.jar',
'20220214',
'sample-qa'
]
}
}
]
尝试过:
def lambda_handler(event, context):
steps = [
{
"Name": "Download Config File",
"ActionOnFailure": "CONTINUE",
"HadoopJarStep": {
"Jar": "command-runner.jar",
"Args": [
"aws",
"s3",
"cp",
"s3://sample-qa/jars/Loads/",
"/home/hadoop/",
"--recursive",
],
},
},
]
def addSteps(date):
step = {
"Name": "Spark Job",
"ActionOnFailure": "CONTINUE",
"HadoopJarStep": {
"Jar": "command-runner.jar",
"Args": [
'spark-submit',
'--deploy-mode', 'cluster',
'--executor-memory', '10g',
'--conf', 'spark.serializer=org.apache.spark.serializer.KryoSerializer',
'--conf', 'spark.sql.hive.convertMetastoreParquet=false',
'--master', 'yarn',
'--class', 'com.general.Loads',
'/home/hadoop/Loads-assembly-0.1.jar',
date,
'sample-qa'
],
},
}
return step
for date in ['20210801','20210807','20210814']:
addingstep = addSteps(date)
steps.append(addingstep)
steps =json.dumps(steps)
print(steps)
我尝试过此实现,但出现错误,请在下面找到。
错误:AttributeError: 'str' 对象没有属性 'append'
创建此列表的任何其他可能性?如何做到这一点?
【问题讨论】:
-
删除
steps =json.dumps(steps),或将其移至循环后。
标签: python python-3.x list aws-lambda amazon-emr