kube-dns pod 故障排除：等待初始化超时

Question

我的 kube-dns pod 崩溃了：

kube-dns-6d4fc847dc-6bh59                               1/3     CrashLoopBackOff   5844       7d13h

这些是来自 kubedns 容器的日志

kubectl logs kube-dns-6d4fc847dc-6bh59 --namespace kube-system  -c kubedns

I1021 00:56:37.547936       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:38.047923       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
E1021 00:56:38.048390       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:192: Failed to list *v1.Service: Get https://100.64.0.1:443/api/v1/services?resourceVersion=0: dial tcp 100.64.0.1:443: i/o timeout
E1021 00:56:38.048405       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:189: Failed to list *v1.Endpoints: Get https://100.64.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 100.64.0.1:443: i/o timeout
I1021 00:56:38.547922       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:39.047917       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:39.547923       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:40.047935       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:40.547928       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:41.047931       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:41.547929       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:42.047924       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:42.547931       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:43.047909       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:43.547921       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:44.047923       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:44.547932       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:45.047935       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:45.547927       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:46.047928       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:46.547929       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:47.047925       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:47.547933       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:48.047914       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:48.547930       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:49.047927       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:49.547929       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:50.047925       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:50.547928       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:51.047933       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:51.547928       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:52.047920       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:52.547930       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:53.047913       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:53.547931       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:54.047925       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:54.547930       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:55.047928       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:55.547938       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:56.047935       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:56.547937       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:57.047940       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:57.547941       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:58.047915       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:58.547934       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:59.047932       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:59.547937       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:00.047942       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:00.547934       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:01.047925       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:01.547929       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:02.047915       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:02.547924       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:03.047906       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:03.547927       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:04.047934       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:04.547927       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:05.047933       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:05.547925       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:06.047928       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:06.547926       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:07.047926       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:07.547930       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
F1021 00:57:08.047911       1 dns.go:209] Timeout waiting for initialization

请帮助。

Answer 1

看起来您的 kube-dns pod 无法与您的 kube-apiserver 通信。 kube-apiserver 上的端口443 已被弃用一段时间。您的 ConfigMap 或配置文件将其指向 <kube-master:443>。

您可以尝试使用 --kube-master-url 选项在您的 kube-dns 部署中覆盖它。像这样的：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  strategy:
    rollingUpdate:
      maxSurge: 10%
      maxUnavailable: 0
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      volumes:
      - name: kube-dns-config
        configMap:
          name: kube-dns
          optional: true
      containers:
      - name: kubedns
        image: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.7
        ...
        args:
        - --kube-master-url=https://<master-ip>:6443
        - --domain=cluster.local.
        - --dns-port=10053
        - --config-dir=/kube-dns-config
        - --v=2

您可以从以下输出中获取：

$ kubectl cluster-info