本地执行与 Terraform 和 AWS CLI 不一致答案

【问题标题】：Local-exec with Terraform and AWS CLI inconsistent本地执行与 Terraform 和 AWS CLI 不一致
【发布时间】：2020-09-18 12:24:07
【问题描述】：

我正在尝试使用 AWS cli 修改 DMS S3 Endpoint 额外连接属性，因为它在 terraform 资源中不起作用。

我有这个代码：

resource "aws_dms_endpoint" "s3_endpoint" {

  count                       = length(var.zones)
  endpoint_id                 = "${var.s3_endpoint_id_prefix}-${var.environment}-${var.shortApp}-${var.zones[count.index]}"
  endpoint_type               = "target"
  engine_name                 = "s3"
  ssl_mode                    = "none"
  kms_key_arn = var.kms_key_arn
  
  s3_settings {
    bucket_folder           = "${var.zones[count.index]}/dms-export/${var.shortApp}"
    bucket_name             = var.bucket_name
    service_access_role_arn = var.service_access_role_arn

  }

  tags = merge(var.tags, { "global.project" = "${local.global_project_tag}" })

   provisioner "local-exec" {
    command = "aws dms modify-endpoint --endpoint-arn ${self.endpoint_arn} --extra-connection-attributes ServiceAccessRoleArn=${var.service_access_role_arn};bucketFolder=${var.zones[count.index]}/dms-export/${var.shortApp};bucketName=${var.bucket_name};cannedAclForObjects=BUCKET_OWNER_FULL_CONTROL;cdcPath=undefined;compressionType=NONE;csvDelimiter=,;csvRowDelimiter=\n;dataFormat=csv;datePartitionEnabled=false;includeOpForFullLoad=true;timestampColumnName=CDCTIMESTAMP"

  }

}

代码没有出现任何错误，但并非所有额外的连接属性都被传递。结果（摘自 json 响应）是这样的：

"ExtraConnectionAttributes": "ServiceAccessRoleArn=arn:aws:iam::<crossedout>:role/SDP_DMS_MIGRATION_TO_S3_ROLE;bucketFolder=test/dms-export/pc;bucketName=<crossedout>-nonprod-sdp-lz-stag-pnc;cannedAclForObjects=BUCKET_OWNER_FULL_CONTROL;cdcPath=undefined;compressionType=NONE;csvDelimiter=,;csvRowDelimiter=\\n;datePartitionEnabled=false;",

但是其他属性没有通过。也没有错误。

编辑：AWS CLI 工作

aws dms modify-endpoint --endpoint-arn arn:aws:dms:eu-central-1:***********:endpoint:*********** --extra-connection-attributes 'ServiceAccessRoleArn=arn:aws:iam::***********:role/SDP_DMS_MIGRATION_TO_S3_ROLE;bucketFolder=komo/dms-export/pc;bucketName=***********-nonprod-sdp-lz-stag-pnc;cannedAclForObjects=BUCKET_OWNER_FULL_CONTROL;cdcPath=undefined;compressionType=NONE;csvDelimiter=,;csvRowDelimiter=\n;dataFormat=csv;datePartitionEnabled=false;includeOpForFullLoad=true;timestampColumnName=CDCTIMESTAMP'

【问题讨论】：

您是否能够证明附加连接属性没有生效，即不仅仅是响应完整性问题？
我也可以在管理控制台中看到它们不存在。当我手动使用 AWS CLI 时，它可以工作（编辑了我的帖子）。
如果将缺少的属性放在 extra-connection-attributes 选项的首位会怎样？

标签： amazon-web-services terraform aws-cli

【解决方案1】：

cli dms docs 显示属性名称。它们的外壳与您使用的不同。

文档展示

"ServiceAccessRoleArn": "string",
"ExternalTableDefinition": "string",
"CsvRowDelimiter": "string",
"CsvDelimiter": "string",
"BucketFolder": "string",
"BucketName": "string",
"CompressionType": "none"|"gzip",
"EncryptionMode": "sse-s3"|"sse-kms",
"ServerSideEncryptionKmsKeyId": "string",
"DataFormat": "csv"|"parquet",
"EncodingType": "plain"|"plain-dictionary"|"rle-dictionary",
"DictPageSizeLimit": integer,
"RowGroupLength": integer,
"DataPageSize": integer,
"ParquetVersion": "parquet-1-0"|"parquet-2-0",
"EnableStatistics": true|false,
"IncludeOpForFullLoad": true|false,
"CdcInsertsOnly": true|false,
"TimestampColumnName": "string",
"ParquetTimestampInMillisecond": true|false,
"CdcInsertsAndUpdates": true|false,
"DatePartitionEnabled": true|false,
"DatePartitionSequence": "YYYYMMDD"|"YYYYMMDDHH"|"YYYYMM"|"MMYYYYDD"|"DDMMYYYY",
"DatePartitionDelimiter": "SLASH"|"UNDERSCORE"|"DASH"|"NONE"

你有：

ServiceAccessRoleArn
bucketFolder
bucketName
cannedAclForObjects
cdcPath
compressionType
csvDelimiter
csvRowDelimiter
dataFormat
datePartitionEnabled
includeOpForFullLoad
timestampColumnName

更容易查看：

yours | docs
:-- | :--
bucketFolder         | BucketFolder
bucketName           | BucketName
compressionType      | CompressionType
csvDelimiter         | CsvDelimiter
csvRowDelimiter      | CsvRowDelimiter
dataFormat           | DataFormat
datePartitionEnabled | DatePartitionEnabled
includeOpForFullLoad | IncludeOpForFullLoad
serviceAccessRoleArn | ServiceAccessRoleArn
timestampColumnName  | TimestampColumnName

我在文档中没有看到以下内容：

CannedAclForObjects
CdcPath

把这一切放在一起试试：

provisioner "local-exec" {
    command = format("aws dms modify-endpoint --endpoint-arn %s --extra-connection-attributes %s", self.endpoint_arn, join(";", [
        "ServiceAccessRoleArn=${var.service_access_role_arn}",
        "BucketFolder=${var.zones[count.index]}/dms-export/${var.shortApp}",
        "BucketName=${var.bucket_name}",
        "CannedAclForObjects=BUCKET_OWNER_FULL_CONTROL",
        "CdcPath=undefined",
        "CompressionType=NONE",
        "CsvDelimiter=,",
        "CsvRowDelimiter=\n",
        "DataFormat=csv",
        "DatePartitionEnabled=false",
        "IncludeOpForFullLoad=true",
        "TimestampColumnName=CDCTIMESTAMP",
    ]))
}

; 上的 join 也将使您的命令更易于阅读。它采用该列表并将它们与; 分隔符组合在一起。

【讨论】：