在混合节点池（Windows/Linux）上的 AKS 上忽略 nodeSelector 约束？

Question

尝试通过运行以下命令使用 Helm 安装基本的 Nginx 入口：

 helm install nginx-ingress --namespace ingress-basic ingress-nginx/ingress-nginx \
          --set controller.service.loadBalancerIP='52.232.109.226' \
          --set controller.nodeSelector."beta\.kubernetes\.io/os"='linux' \
          --set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"='linux' \
          --set controller.replicaCount=1 \
          --set rbac.create=true

安装后不久，我注意到 pod 被调度到 Windows 节点而不是 Linux 节点：

wesley@Azure:~$ kubectl get pods -n ingress-basic -o wide
NAME                                                 READY   STATUS              RESTARTS   AGE   IP       NODE           NOMINATED NODE   READINESS GATES
nginx-ingress-ingress-nginx-admission-create-jcp6x   0/1     ContainerCreating   0          18s   <none>   akswin000002   <none>           <none>

wesley@Azure:~$ kubectl get nodes -o wide
NAME                                STATUS   ROLES   AGE     VERSION    INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION      CONTAINER-RUNTIME
aks-agentpool-59412422-vmss000000   Ready    agent   5h32m   v1.17.11   10.240.0.4     <none>        Ubuntu 16.04.7 LTS               4.15.0-1096-azure   docker://19.3.12
aks-linuxpool-59412422-vmss000000   Ready    agent   5h32m   v1.17.11   10.240.0.128   <none>        Ubuntu 16.04.7 LTS               4.15.0-1096-azure   docker://19.3.12
akswin000000                        Ready    agent   5h28m   v1.17.11   10.240.0.35    <none>        Windows Server 2019 Datacenter   10.0.17763.1397     docker://19.3.11
akswin000001                        Ready    agent   5h28m   v1.17.11   10.240.0.66    <none>        Windows Server 2019 Datacenter   10.0.17763.1397     docker://19.3.11
akswin000002                        Ready    agent   5h28m   v1.17.11   10.240.0.97    <none>        Windows Server 2019 Datacenter   10.0.17763.1397     docker://19.3.11

在 nginx pod 上运行 describe 显示字段 Node-selectors 仍需设置为 <none>.

Name:               nginx-ingress-ingress-nginx-admission-create-jcp6x
Namespace:          ingress-basic
Priority:           0
PriorityClassName:  <none>
Node:               akswin000002/10.240.0.97
Start Time:         Fri, 16 Oct 2020 20:09:36 +0000
Labels:             app.kubernetes.io/component=admission-webhook
                    app.kubernetes.io/instance=nginx-ingress
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=ingress-nginx
                    app.kubernetes.io/version=0.40.2
                    controller-uid=d03091cd-8138-4923-a369-afeca669099c
                    helm.sh/chart=ingress-nginx-3.7.1
                    job-name=nginx-ingress-ingress-nginx-admission-create
Annotations:        <none>
**Status:             Pending**
IP:                
Controlled By:      Job/nginx-ingress-ingress-nginx-admission-create
Containers:
  create:
    Container ID: 
    Image:         docker.io/jettech/kube-webhook-certgen:v1.3.0
    Image ID:     
    Port:          <none>
    Host Port:     <none>
    Args:
      create
      --host=nginx-ingress-ingress-nginx-controller-admission,nginx-ingress-ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
      --namespace=$(POD_NAMESPACE)
      --secret-name=nginx-ingress-ingress-nginx-admission
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:
      POD_NAMESPACE:  ingress-basic (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from nginx-ingress-ingress-nginx-admission-token-8x6ct (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  nginx-ingress-ingress-nginx-admission-token-8x6ct:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  nginx-ingress-ingress-nginx-admission-token-8x6ct
    Optional:    false
QoS Class:       BestEffort
**Node-Selectors:  <none>**
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age                  From                   Message
  ----     ------                  ----                 ----                   -------
  Normal   SandboxChanged          5m (x5401 over 2h)   kubelet, akswin000002  Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  31s (x5543 over 2h)  kubelet, akswin000002  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "nginx-ingress-ingress-nginx-admission-create-jcp6x": Error response from daemon: container a734c23d20338d7fed800752c19f5e94688fd38fe82c9e90fc14533bae90c6bc encountered an error during hcsshim::System::CreateProcess: failure in a Windows system call: The user name or password is incorrect. (0x52e) extra info: {"CommandLine":"cmd /S /C pauseloop.exe","User":"2000","WorkingDirectory":"C:\\","Environment":{"PATH":"c:\\Windows\\System32;c:\\Windows"},"CreateStdInPipe":true,"CreateStdOutPipe":true,"CreateStdErrPipe":true,"ConsoleSize":[0,0]}

我希望将 pod 安排到 Linux 节点上。有谁知道为什么会这样？我没有看到任何污点或任何东西，这只是一个新的旋转集群。目前唯一的解决方法似乎是将 windows 节点缩放回 0。安装 nginx ingress，然后再次扩展 windows 节点。

• Kubernetes 版本：1.17.11

Answer 1

适用于以下。添加 admissionWebhooks.patch.nodeSelector。 https://docs.microsoft.com/en-us/azure/aks/ingress-basic#create-an-ingress-controller

helm install nginx-ingress ingress-nginx/ingress-nginx \
    --namespace ingress-basic \
    --set controller.replicaCount=2 \
    --set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
    --set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux \
    --set controller.admissionWebhooks.patch.nodeSelector."beta\.kubernetes\.io/os"=linux