【问题标题】:Azure ARM template dependsOn not working (succeeds on redeploy)Azure ARM 模板依赖于不工作(重新部署成功)
【发布时间】:2020-11-18 16:55:30
【问题描述】:

当我在 identity 字段中使用带有 SystemAssigned 的 ARM 模板创建 AKS 群集时,会创建一个格式为 MC_<rg_name>_<cluster_name>_location 的辅助资源组。在该组中,有一个 <cluster-name>-agentpool 形式的托管标识,Kubelet 使用该标识。

手臂模板将在帖子的底部,但总体结构如下。我将它们分开部署的原因是因为我在 subscription 级别部署东西:

Deployment A
  - Microsoft.ContainerService/managedClusters
Deployment B (dependsOn A)
  - Microsoft.Authorization/roleAssignments
    - contains reference to `nodeResourceGroup` with API version so I made explicit dependsOn for A

我计划使用Azure Pod Identity,因此我需要为该托管标识分配两个角色:ManagedIdentityOperatorVirtualMachineContributor。我有一个变量,它构建了引用中使用的此标识的路径:

"agentpool-account":[concat(subscription().id, '/resourceGroups/', variables('managedClusterResourceGroup'), '/providers/Microsoft.ManagedIdentity/userAssignedIdentities/', parameters('cluster-name'), '-agentpool')]
"principalId": "[reference(parameters('agentpool-account'), '2018-11-30', 'full').properties.principalId]",

但是,在我第一次部署此模板时(即在创建集群时),RoleAssignment 部署将由于 Resource Group not found 错误而失败。查看部署活动确认RoleAssignment 已部署MC_group 被创建,尽管在集群部署上有明确的dependsOn。如果我重新部署模板,它将成功,因为 MC_group 现在根据 ARM 存在。

我想知道是否有其他人遇到过这个问题,以及如何解决这个问题的任何提示都会很棒。我遇到了一个链接https://bmoore-msft.blog/2020/07/26/resource-not-found-dependson-is-not-working/,但似乎对我不起作用。

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.1",
  "parameters": {
    "cluster-name": {
      "metadata": {
        "description": "The name of the cluster"
      },
      "type": "string"
    },
  },
  "resources": [
    {
      "apiVersion": "2019-10-01",
      "location": "centralus",
      "name": "test",
      "type": "Microsoft.Resources/resourceGroups"
    },
    {
      "apiVersion": "2020-06-01",
      "dependsOn": [
        "[resourceId('Microsoft.Resources/resourceGroups', 'test')]"
      ],
      "name": "cluster-deployment",
      "properties": {
        "expressionEvaluationOptions": {
          "scope": "outer"
        },
        "mode": "Incremental",
        "parameters": {},
        "template": {
          "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
          "contentVersion": "1.0.0.0",
          "outputs": {},
          "parameters": {},
          "resources": [
            {
              "apiVersion": "2019-06-01",
              "dependsOn": [],
              "identity": {
                "type": "SystemAssigned"
              },
              "location": "centralus",
              "name": "[parameters('cluster-name')]",
              "properties": {
                "addonProfiles": {
                  "azurePolicy": {
                    "enabled": false
                  },
                  "httpApplicationRouting": {
                    "enabled": false
                  }
                },
                "agentPoolProfiles": [
                  {
                    "availabilityZones": [
                      "1",
                      "2",
                      "3"
                    ],
                    "count": 3,
                    "maxPods": 110,
                    "mode": "System",
                    "name": "agentpool",
                    "osDiskSizeGB": 0,
                    "osType": "Linux",
                    "storageProfile": "ManagedDisks",
                    "type": "VirtualMachineScaleSets",
                    "vmSize": "Standard_D16s_v3"
                  }
                ],
                "apiServerAccessProfile": {
                  "enablePrivateCluster": false
                },
                "dnsPrefix": "[concat(parameters('cluster-name'), '-dns')]",
                "enableRBAC": true,
                "kubernetesVersion": "1.17.11",
                "networkProfile": {
                  "loadBalancerSku": "standard",
                  "networkPlugin": "kubenet",
                  "networkPolicy": "calico"
                }
              },
              "tags": {},
              "type": "Microsoft.ContainerService/managedClusters"
            }
          ],
          "variables": {}
        }
      },
      "resourceGroup": "test",
      "type": "Microsoft.Resources/deployments"
    },
    {
      "apiVersion": "2020-06-01",
      "dependsOn": [
        "cluster-deployment"
      ],
      "name": "identity-assignment",
      "properties": {
        "expressionEvaluationOptions": {
          "scope": "outer"
        },
        "mode": "Incremental",
        "template": {
          "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
          "contentVersion": "1.0.0.0",
          "outputs": {},
          "parameters": {},
          "resources": [
            {
              "apiVersion": "2017-09-01",
              "name": "[guid('test', 'ManagedIdentityOperator')]",
              "properties": {
                "principalId": "[reference(variables('agentpoolResourceId'), '2018-11-30', 'full').properties.principalId]",
                "roleDefinitionId": "[variables('managedIdentityOperatorRoleId')]",
                "scope": "[concat(subscription().id, '/resourceGroups/test')]"
              },
              "type": "Microsoft.Authorization/roleAssignments"
            },
            {
              "apiVersion": "2017-09-01",
              "name": "[guid('test', 'VirtualMachineContributor')]",
              "properties": {
                "principalId": "[reference(variables('agentpoolResourceId'), '2018-11-30', 'full').properties.principalId]",
                "roleDefinitionId": "[variables('virtualMachineContributorRoleId')]",
                "scope": "[concat(subscription().id, '/resourceGroups/test')]"
              },
              "type": "Microsoft.Authorization/roleAssignments"
            }
          ]
        }
      },
      "resourceGroup": "test",
      "type": "Microsoft.Resources/deployments"
    }
  ],
  "variables": {
    "agentPoolResourceId": "[concat(subscription().id, '/resourceGroups/', variables('managedClusterResourceGroup'), '/providers/Microsoft.ManagedIdentity/userAssignedIdentities/', parameters('cluster-name'), '-agentpool')]",
    "managedClusterResourceGroup": "[concat('MC_test_', parameters('cluster-name'), '_centralus')]",
    "managedIdentityOperatorRoleId": "[concat(subscription().id, '/providers/Microsoft.Authorization/roleDefinitions/f1a07417-d97a-45cb-824c-7a7467783830')]",
    "virtualMachineContributorRoleId": "[concat(subscription().id, '/providers/Microsoft.Authorization/roleDefinitions/9980e02c-c2be-4d73-94e8-173b1dc7cf3c')]"
  }
}

【问题讨论】:

    标签: azure azure-resource-manager arm-template azure-resource-group


    【解决方案1】:

    试试附件...我怀疑您遇到的问题是 systemAssigned 身份(又名 MSI)在执行 roleAssignment 时尚未全局复制。要解决此问题,您可以将 principalType 属性添加到 roleAssigment 并且即使可能尚未找到主体(即 MSI)也会强制分配。这可能是简单的解决方法。

    除此之外,我稍微更改了模板以删除第二个部署,这样可能会显示出比您实际需要的更大的差异,但只是一种不同的处理方式。

    {
        "$schema": "https://schema.management.azure.com/schemas/2018-05-01/subscriptionDeploymentTemplate.json#",
        "contentVersion": "1.0.0.1",
        "parameters": {
            "cluster-name": {
                "type": "string",
                "defaultValue": "mc1"
            },
            "resourceGroupName": {
                "type": "string",
                "defaultValue": "test"
            }
        },
        "resources": [
            {
                "type": "Microsoft.Resources/resourceGroups",
                "apiVersion": "2019-10-01",
                "location": "centralus",
                "name": "[parameters('resourceGroupName')]"
            },
            {
                "type": "Microsoft.Resources/deployments",
                "apiVersion": "2020-06-01",
                "name": "cluster-deployment",
                "resourceGroup": "[parameters('resourceGroupName')]",
                "dependsOn": [
                    "[subscriptionResourceId('Microsoft.Resources/resourceGroups', 'test')]"
                ],
                "properties": {
                    "expressionEvaluationOptions": {
                        "scope": "inner"
                    },
                    "mode": "Incremental",
                    "parameters": {
                        "cluster-name":{
                            "value": "[parameters('cluster-name')]"
                        }
                    },
                    "template": {
                        "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
                        "contentVersion": "1.0.0.0",
                        "parameters": {
                            "cluster-name": {
                                "type": "string"
                            }
                        },
                        "variables": {
                            "managedIdentityOperatorRoleId": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'f1a07417-d97a-45cb-824c-7a7467783830')]",
                            "virtualMachineContributorRoleId": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '9980e02c-c2be-4d73-94e8-173b1dc7cf3c')]"
                        },
                        "resources": [
                            {
                                "type": "Microsoft.ContainerService/managedClusters",
                                "apiVersion": "2020-11-01",
                                "name": "[parameters('cluster-name')]",
                                "location": "centralus",
                                "identity": {
                                    "type": "SystemAssigned"
                                },
                                "properties": {
                                    "addonProfiles": {
                                        "azurePolicy": {
                                            "enabled": false
                                        },
                                        "httpApplicationRouting": {
                                            "enabled": false
                                        }
                                    },
                                    "agentPoolProfiles": [
                                        {
                                            "availabilityZones": [
                                                "1",
                                                "2",
                                                "3"
                                            ],
                                            "count": 3,
                                            "maxPods": 110,
                                            "mode": "System",
                                            "name": "agentpool",
                                            "osDiskSizeGB": 0,
                                            "osType": "Linux",
                                            "storageProfile": "ManagedDisks",
                                            "type": "VirtualMachineScaleSets",
                                            "vmSize": "Standard_D16s_v3"
                                        }
                                    ],
                                    "apiServerAccessProfile": {
                                        "enablePrivateCluster": false
                                    },
                                    "dnsPrefix": "[concat(parameters('cluster-name'), '-dns')]",
                                    "enableRBAC": true,
                                    "kubernetesVersion": "1.17.11",
                                    "networkProfile": {
                                        "loadBalancerSku": "standard",
                                        "networkPlugin": "kubenet",
                                        "networkPolicy": "calico"
                                    }
                                }
                            },
                            {
                                "type": "Microsoft.Authorization/roleAssignments",
                                "apiVersion": "2017-09-01",
                                "name": "[guid('test', 'ManagedIdentityOperator')]",
                                "properties": {
                                    "principalId": "[reference(parameters('cluster-name'), '2020-11-01', 'full').identity.principalId]",
                                    "roleDefinitionId": "[variables('managedIdentityOperatorRoleId')]",
                                    "scope": "[resourceGroup().id]",
                                    "principalType": "ServicePrincipal"
                                }
                            },
                            {
                                "type": "Microsoft.Authorization/roleAssignments",
                                "apiVersion": "2017-09-01",
                                "name": "[guid('test', 'VirtualMachineContributor')]",
                                "properties": {
                                    "principalId": "[reference(parameters('cluster-name'), '2020-11-01', 'full').identity.principalId]",
                                    "roleDefinitionId": "[variables('virtualMachineContributorRoleId')]",
                                    "scope": "[resourceGroup().id]",
                                    "principalType": "ServicePrincipal"
                                }
                            }
                        ]
                    }
                }
            }
        ]
    }
    

    如果您对其他更改有任何疑问...

    【讨论】:

    • 感谢您的回复!在之前的迭代中,我实际上使用了对系统分配标识的精确引用。但是,当我开始设置AAD Pod Identity 时,我意识到(默认情况下)预期的托管标识不是系统分配的标识,而是用户分配的<cluster-name>-agentpool 形式的标识。 agentpool 帐户是位于 MC_group 内的资源,而验证 MC_group 的存在对我来说是失败的。
    • 我似乎找到了绕过它的方法,它的过程类似于我在您的博客上找到的过程。我基本上将ManagedCluster 部署变成了inner 范围部署并输出reference('cluster-name').nodeResourceGroup。然后,我将该值作为参数汇集到同一父模板中的单独角色分配部署中,并且在第一次通过时工作。
    • 添加"principalType": "ServicePrincipal" 就可以了...谢谢布赖恩! (真的应该在文档中)
    • 你说得对,应该 - 你在哪里看,我看看能不能更新...
    猜你喜欢
    • 1970-01-01
    • 2019-02-11
    • 1970-01-01
    • 1970-01-01
    • 2022-11-23
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2017-12-19
    相关资源
    最近更新 更多