为什么 AWS CloudFormation 会抛出“遇到不受支持的属性 InstanceGroups”？

Question

当我部署以下 AWS CloudFormation 脚本时，我收到以下错误：“遇到不支持的属性 InstanceGroups”

我过去使用 InstanceGroups 没有任何问题。以下是其他人如何使用它的示例：https://noise.getoto.net/tag/amazon-emr/

我使用的是以前设置过的 EMR 5.17.0。

{
  "Description": "Spark ETL EMR CloudFormation",
  "Resources": {
    "EMRCluster": {
      "Type": "AWS::EMR::Cluster",
      "Properties": {
        "Applications": [
          {
            "Name": "Hadoop"
          },
          {
            "Name": "Spark"
          },
          {
            "Name": "Ganglia"
          },
          {
            "Name": "Zeppelin"
          }
        ],
        "AutoScalingRole": "EMR_AutoScaling_DefaultRole",
        "BootstrapActions": [
          {
            "Path": "s3://somepath/scripts/install_pip36_dependencies.sh",
            "Args": [
              "relay==0.0.1"
            ],
            "Name": "install_pip36_dependencies"
          }
        ],
        "Configurations": [
          {
            "Classification": "yarn-site",
            "Properties": {
              "yarn.scheduler.fair.preemption": "False",
              "yarn.resourcemanager.am.max-attempts": "1"
            },
            "Configurations": []
          },
          {
            "Classification": "core-site",
            "Properties": {
              "fs.s3.canned.acl": "BucketOwnerFullControl"
            },
            "Configurations": []
          }
        ],
        "EbsRootVolumeSize": 10,
        "InstanceGroups": [
          {
            "Name": "Master",
            "Market": "ON_DEMAND",
            "InstanceRole": "MASTER",
            "InstanceType": "m5.2xlarge",
            "InstanceCount": 1,
            "EbsConfiguration": {
              "EbsBlockDeviceConfigs": [
                {
                  "VolumeSpecification": {
                    "SizeInGB": 100,
                    "VolumeType": "64"
                  },
                  "VolumesPerInstance": 1
                }
              ],
              "EbsOptimized": "True"
            }
          },
          {
            "Name": "Core",
            "Market": "ON_DEMAND",
            "InstanceGroupType": "CORE",
            "InstanceType": "m5.2xlarge",
            "InstanceCount": 5,
            "EbsConfiguration": {
              "EbsBlockDeviceConfigs": [
                {
                  "VolumeSpecification": {
                    "SizeInGB": 100,
                    "VolumeType": "gp2"
                  },
                  "VolumesPerInstance": 1
                }
              ],
              "EbsOptimized": "True"
            }
          },
          {
            "Name": "Task - 3",
            "Market": "ON_DEMAND",
            "InstanceGroupType": "TASK",
            "InstanceType": "m5.2xlarge",
            "InstanceCount": 2,
            "EbsConfiguration": {
              "EbsBlockDeviceConfigs": [
                {
                  "VolumeSpecification": {
                    "SizeInGB": 32,
                    "VolumeType": "gp2"
                  },
                  "VolumesPerInstance": 1
                }
              ],
              "EbsOptimized": "True"
            }
          }
        ],
        "LogUri": "s3://somepath/emr-logs/",
        "Name": "EMR CF",
        "ReleaseLabel": "emr-5.17.0",
        "ServiceRole": "EMR_DefaultRole",
        "VisibleToAllUsers": "True"
      }
    }
  }
}

当 CF 脚本被加载时，它应该创建一个 AWS EMR 集群

Answer 1

AWS 建议您在Instances 下设置MasterInstanceGroup 和CoreInstanceGroup

我给你一个Instances属性的例子EMR Cluster with Hadoop, Hbase, Spark, Ganglia and Zookeeper：

      Instances:
        Ec2KeyName: !Ref KeyName
        Ec2SubnetId: !ImportValue MySubnetPrivateA
        EmrManagedMasterSecurityGroup: !ImportValue EmrMasterSgId
        AdditionalMasterSecurityGroups:
          - !ImportValue EmrMasterAdditionalSgId
        EmrManagedSlaveSecurityGroup: !ImportValue EmrSlaveSgId
        AdditionalSlaveSecurityGroups:
          - !ImportValue EmrSlaveAdditionalSgId
        ServiceAccessSecurityGroup: !ImportValue EmrServiceSgId
        MasterInstanceGroup:
          InstanceCount: 1
          InstanceType: !Ref MasterInstanceType
          Market: ON_DEMAND
          Name: Master
        CoreInstanceGroup:
          InstanceCount: !Ref NumberOfCoreInstances
          InstanceType: !Ref CoreInstanceType
          Market: ON_DEMAND
          Name: Core
        TerminationProtected: false
      VisibleToAllUsers: true
      JobFlowRole: !Ref EMRClusterinstanceProfile
      ReleaseLabel: !Ref ReleaseLabel
      LogUri: !Ref LogUri
      Name: !Ref EMRClusterName
      AutoScalingRole: EMR_AutoScaling_DefaultRole
      ServiceRole: !Ref EMRClusterServiceRole
      Tags:
        -
          Key: "cluster_name"
          Value: "master.emr.my.com"

您可以查看完整的 AWS 模板here。