AWS Glue 数据目标 Upsert 生成异常答案

【问题标题】：AWS Glue Data Target Upsert generates an ExceptionAWS Glue 数据目标 Upsert 生成异常
【发布时间】：2022-06-17 16:47:04
【问题描述】：

我正在使用 AWS Glue 将数据加载到使用 Glue Studio 的 Redshift 数据库中。

如果 Data Target 是 Insert Only 数据被插入没有任何问题，这是生成的代码：

# Script generated for node Amazon Redshift
AmazonRedshift_node = glueContext.write_dynamic_frame.from_catalog(
    frame=SelectFields_node2,
    database="redshift_mast_code",
    table_name="dev_mcd_rs_iot_mast_code",
    redshift_tmp_dir="s3://glue-temp-dir-dev/",
    additional_options={
        "aws_iam_role": "arn:aws:iam::...myRole"
    },
    transformation_ctx="AmazonRedshift_node",
)

但是，当我使用 Upsert（更新和插入）选项时，出现异常。

2022-04-03 15:19:09,674 ERROR [main] glue.ProcessLauncher (Logging.scala:logError(73)): Error from Python:Traceback (most recent call last):
  File "/tmp/glue-scripts-tmp", line 129, in <module>
    transformation_ctx="AmazonRedshift_node",
TypeError: from_jdbc_conf() got an unexpected keyword argument 'additional_options'

这是生成的代码：

# Script generated for node Amazon Redshift
pre_query = "drop table if exists mcd_rs_iot.stage_table_941d406a69c8480aa44ed085a2adeb40;create table mcd_rs_iot.stage_table_941d406a69c8480aa44ed085a2adeb40 as select * from mcd_rs_iot.mast_code where 1=2;"
post_query = "begin;delete from mcd_rs_iot.mast_code using mcd_rs_iot.stage_table_941d406a69c8480aa44ed085a2adeb40 where mcd_rs_iot.stage_table_941d406a69c8480aa44ed085a2adeb40.cd = mcd_rs_iot.mast_code.cd; insert into mcd_rs_iot.mast_code select * from mcd_rs_iot.stage_table_941d406a69c8480aa44ed085a2adeb40; drop table mcd_rs_iot.stage_table_941d406a69c8480aa44ed085a2adeb40; end;"
AmazonRedshift_node = glueContext.write_dynamic_frame.from_jdbc_conf(
    frame=SelectFields_node2,
    catalog_connection="Redshift-Connection",
    connection_options={
        "database": "dev",
        "dbtable": "mcd_rs_iot.stage_table_941d406a69c8480aa44ed085a2adeb40",
        "preactions": pre_query,
        "postactions": post_query,
    },
    redshift_tmp_dir="s3://glue-temp-dir-dev/",
    additional_options={
        "aws_iam_role": "arn:aws:iam::...myRole"
    },
    transformation_ctx="AmazonRedshift_node",

如何在 Glue Studio 中使用 Upsert 选项？

胶水版本：3.0 支持 Spark 3.1 Scala 2、Python 3

【问题讨论】：

标签： aws-glue aws-glue3.0

【解决方案1】：

您正在设置 from_jdbc_conf 函数中不允许的附加选项：https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-dynamic-frame-writer.html

只需将其从配置中删除即可。

【讨论】：