AWS Glue Cloudformation 排除模式排除：字符串答案

【问题标题】：AWS Glue Cloudformation exclude patterns Exclusions: StringAWS Glue Cloudformation 排除模式排除：字符串
【发布时间】：2020-03-10 01:14:21
【问题描述】：

我已经在 AWS 控制台上成功设置了一个胶水爬虫。现在我有一个 Cloudformation 模板来模拟整个过程，除了我无法将 Exclusions: 字段添加到模板中。背景：在 AWS Glue API 中，Exclusions: 字段表示 glob 模式，用于排除与数据存储（在我的示例中为 S3 数据存储）中特定模式匹配的文件或文件夹。

尽管脚本中的所有其他值与爬虫配置一起填充（即 S3Target、爬虫名称、IAM 角色和分组行为，所有这些胶水设置），但我费了很大力气还是无法在胶水爬虫控制台上填充 glob 模式/fields 从 CFN 模板成功填充，除了 Exclusions 字段，也称为 Glue 控制台上的排除模式。我的 CFN 模板通过了验证，我已经运行了爬虫，希望尽管隐藏的排除 glob 仍然会产生影响，但不幸的是我似乎无法填充 Exclusions 字段？

Here's the S3Target Exclusion AWS Glue API guide

Here's an AWS sample YAML CFN for a Glue Crawler

Here's a helpful YAML string array guide

YAML

 CFNCrawlerSecDeraNUM:
    Type: AWS::Glue::Crawler
    Properties:
      Name: !Ref CFNCrawlerName
      Role: !GetAtt CFNRoleSecDERA.Arn
      #Classifiers: none, use the default classifier
      Description: AWS Glue crawler to crawl SecDERA data
      #Schedule: none, use default run-on-demand
      DatabaseName: !Ref CFNDatabaseName
      Targets:
        S3Targets:
          - Exclusions:
              - "*/readme.htm"
              - "*/sub.txt"
              - "*/pre.txt"
              - "*/tag.txt"
          - Path: "s3://sec-input"
      TablePrefix: !Ref CFNTablePrefixName
      SchemaChangePolicy:
        UpdateBehavior: "UPDATE_IN_DATABASE"
        DeleteBehavior: "LOG"
        # Added single schema grouping Glue API option
      Configuration: "{\"Version\":1.0,\"CrawlerOutput\":{\"Partitions\":{\"AddOrUpdateBehavior\":\"InheritFromTable\"},\"Tables\":{\"AddOrUpdateBehavior\":\"MergeNewColumns\"}},\"Grouping\":{\"TableGroupingPolicy\":\"CombineCompatibleSchemas\"}}"

JSON

"CFNCrawlerSecDeraNUM": {
    "Type": "AWS::Glue::Crawler",
    "Properties": {
        "Name": {
            "Ref": "CFNCrawlerName"
        },
        "Role": {
            "Fn::GetAtt": [
                "CFNRoleSecDERA",
                "Arn"
            ]
        },
        "Description": "AWS Glue crawler to crawl SecDERA data",
        "DatabaseName": {
            "Ref": "CFNDatabaseName"
        },
        "Targets": {
            "S3Targets": [
                {
                    "Exclusions": [
                        "*/readme.htm",
                        "*/sub.txt",
                        "*/pre.txt",
                        "*/tag.txt"
                    ]
                },
                {
                    "Path": "s3://sec-input"
                }
            ]
        },
        "TablePrefix": {
            "Ref": "CFNTablePrefixName"
        },
        "SchemaChangePolicy": {
            "UpdateBehavior": "UPDATE_IN_DATABASE",
            "DeleteBehavior": "LOG"
        },
        "Configuration": "{\"Version\":1.0,\"CrawlerOutput\":{\"Partitions\":{\"AddOrUpdateBehavior\":\"InheritFromTable\"},\"Tables\":{\"AddOrUpdateBehavior\":\"MergeNewColumns\"}},\"Grouping\":{\"TableGroupingPolicy\":\"CombineCompatibleSchemas\"}}"
    }
}

【问题讨论】：

标签： amazon-web-services amazon-cloudformation aws-glue

【解决方案1】：

您将Exclusions 作为新的S3Target 对象传递给S3Targets 列表。

尝试改变这个：

  Targets:
    S3Targets:
      - Exclusions:
          - "*/readme.htm"
          - "*/sub.txt"
          - "*/pre.txt"
          - "*/tag.txt"
      - Path: "s3://sec-input"

到这里：

  Targets:
    S3Targets:
      - Path: "s3://sec-input"
        Exclusions:
          - "*/readme.htm"
          - "*/sub.txt"
          - "*/pre.txt"
          - "*/tag.txt"

【讨论】：