Coredns 不时解决答案

【问题标题】：Coredns doesn't resolve time to timeCoredns 不时解决
【发布时间】：2019-06-18 12:18:51
【问题描述】：

k8s：1.13.1 核心：1.2.6

我有时会在尝试解决服务时遇到“无法解决”

[root@k8s-kube-master-vm01 ~]# kubectl exec -it  tiller-deploy-dc85f7fbd-vkqjs -n kube-system nslookup kubernetes-dashboard 
nslookup: can't resolve '(null)': Name does not resolve

Name:      kubernetes-dashboard
Address 1: 10.233.43.194
[root@k8s-kube-master-vm01 ~]# kubectl exec -it  tiller-deploy-dc85f7fbd-vkqjs -n kube-system nslookup kubernetes-dashboard 
nslookup: can't resolve '(null)': Name does not resolve

Name:      kubernetes-dashboard
Address 1: 10.233.43.194
[root@k8s-kube-master-vm01 ~]# kubectl exec -it  tiller-deploy-dc85f7fbd-vkqjs -n kube-system nslookup kubernetes-dashboard 
nslookup: can't resolve '(null)': Name does not resolve

nslookup: can't resolve 'kubernetes-dashboard': Try again
command terminated with exit code 1

核心日志：

[root@k8s-kube-master-vm01 ~]# for p in $(kubectl get pods --namespace=kube-system -l k8s-app=coredns -o name); do kubectl logs --namespace=kube-system $p; done  | grep kubernetes-dashboard
2019-06-18T12:05:22.549Z [INFO] 10.233.124.35:41451 - 32323 "AAAA IN kubernetes-dashboard.kube-system.svc.cluster.local. udp 68 false 512" NOERROR qr,aa,rd,ra 161 0.000192773s
2019-06-18T12:05:22.55Z [INFO] 10.233.124.35:41451 - 32054 "A IN kubernetes-dashboard.kube-system.svc.cluster.local. udp 68 false 512" NOERROR qr,aa,rd,ra 134 0.000112697s
2019-06-18T12:05:25.053Z [INFO] 10.233.124.35:41451 - 32323 "AAAA IN kubernetes-dashboard.kube-system.svc.cluster.local. udp 68 false 512" NOERROR qr,rd,ra 161 0.000115262s
2019-06-18T12:05:25.053Z [INFO] 10.233.124.35:41451 - 32054 "A IN kubernetes-dashboard.kube-system.svc.cluster.local. udp 68 false 512" NOERROR qr,rd,ra 134 0.000037126s
2019-06-18T12:05:10.187Z [INFO] 10.233.124.35:50002 - 59429 "AAAA IN kubernetes-dashboard.kube-system.svc.cluster.local. udp 68 false 512" NOERROR qr,aa,rd,ra 161 0.000160883s
2019-06-18T12:05:10.187Z [INFO] 10.233.124.35:50002 - 59123 "A IN kubernetes-dashboard.kube-system.svc.cluster.local. udp 68 false 512" NOERROR qr,aa,rd,ra 134 0.00006191s
2019-06-18T12:05:16.493Z [INFO] 10.233.124.35:40094 - 37873 "AAAA IN kubernetes-dashboard.kube-system.svc.cluster.local. udp 68 false 512" NOERROR qr,rd,ra 161 0.000113602s
2019-06-18T12:05:16.493Z [INFO] 10.233.124.35:40094 - 37550 "A IN kubernetes-dashboard.kube-system.svc.cluster.local. udp 68 false 512" NOERROR qr,aa,rd,ra 134 0.000154653s

外部地址根本没有解析：

[root@k8s-kube-master-vm01 ~]# for p in $(kubectl get pods --namespace=kube-system -l k8s-app=coredns -o name); do kubectl logs --namespace=kube-system $p; done  | grep google.com
2019-06-18T10:18:31.888Z [INFO] 10.233.124.35:45081 - 28149 "AAAA IN google.com.svc.cluster.local. udp 46 false 512" NXDOMAIN qr,aa,rd,ra 139 0.000211934s
2019-06-18T10:18:31.888Z [INFO] 10.233.124.35:45081 - 27872 "A IN google.com.svc.cluster.local. udp 46 false 512" NXDOMAIN qr,aa,rd,ra 139 0.000136701s
2019-06-18T10:18:34.391Z [INFO] 10.233.124.35:45081 - 28149 "AAAA IN google.com.svc.cluster.local. udp 46 false 512" NXDOMAIN qr,rd,ra 139 0.000136248s
2019-06-18T10:18:34.391Z [INFO] 10.233.124.35:45081 - 27872 "A IN google.com.svc.cluster.local. udp 46 false 512" NXDOMAIN qr,rd,ra 139 0.000035216s
2019-06-18T10:20:23.899Z [INFO] 10.233.124.35:57585 - 35314 "AAAA IN google.com.svc.cluster.local. udp 46 false 512" NXDOMAIN qr,aa,rd,ra 139 0.000202737s
2019-06-18T10:20:23.899Z [INFO] 10.233.124.35:57585 - 35055 "A IN google.com.svc.cluster.local. udp 46 false 512" NXDOMAIN qr,aa,rd,ra 139 0.000252218s
2019-06-18T10:20:26.4Z [INFO] 10.233.124.35:57585 - 35314 "AAAA IN google.com.svc.cluster.local. udp 46 false 512" NXDOMAIN qr,rd,ra 139 0.000131456s
2019-06-18T10:20:26.4Z [INFO] 10.233.124.35:57585 - 35055 "A IN google.com.svc.cluster.local. udp 46 false 512" NXDOMAIN qr,rd,ra 139 0.000062335s
2019-06-18T12:11:18.188Z [INFO] 10.233.124.35:50407 - 11706 "AAAA IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,aa,rd,ra 151 0.000202781s
2019-06-18T12:11:18.188Z [INFO] 10.233.124.35:50407 - 11325 "A IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,aa,rd,ra 151 0.00015313s
2019-06-18T12:11:20.691Z [INFO] 10.233.124.35:50407 - 11706 "AAAA IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,rd,ra 151 0.000148718s
2019-06-18T12:11:20.691Z [INFO] 10.233.124.35:50407 - 11325 "A IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,rd,ra 151 0.000124545s
2019-06-18T12:11:24.528Z [INFO] 10.233.124.35:54579 - 11413 "AAAA IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,rd,ra 151 0.000074136s
2019-06-18T12:11:24.528Z [INFO] 10.233.124.35:54579 - 11083 "A IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,rd,ra 151 0.000033644s
2019-06-18T12:11:27.03Z [INFO] 10.233.124.35:54579 - 11413 "AAAA IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,rd,ra 151 0.000136221s
2019-06-18T12:11:27.03Z [INFO] 10.233.124.35:54579 - 11083 "A IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,rd,ra 151 0.000190085s
2019-06-18T10:18:31.894Z [INFO] 10.233.124.35:49521 - 63928 "AAAA IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,aa,rd,ra 151 0.000237032s
2019-06-18T10:18:31.895Z [INFO] 10.233.124.35:49521 - 63636 "A IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,aa,rd,ra 151 0.000177646s
2019-06-18T10:20:23.905Z [INFO] 10.233.124.35:41078 - 52148 "AAAA IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,aa,rd,ra 151 0.000218841s
2019-06-18T10:20:23.906Z [INFO] 10.233.124.35:41078 - 51789 "A IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,aa,rd,ra 151 0.000090002s

我尝试查看 /etc/resolv.conf，但 coredns pod 中没有实用程序：

[root@k8s-kube-master-vm01 ~]# kubectl -n kube-system exec coredns-6fd7dbf94c-czjxj  -it sh
OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"sh\": executable file not found in $PATH": unknown
command terminated with exit code 126
[root@k8s-kube-master-vm01 ~]# kubectl -n kube-system exec coredns-6fd7dbf94c-czjxj  -it cat /etc/hosts
OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"cat\": executable file not found in $PATH": unknown
command terminated with exit code 126
[root@k8s-kube-master-vm01 ~]# kubectl -n kube-system exec coredns-6fd7dbf94c-czjxj  -it /bin/cat /etc/hosts
OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"/bin/cat\": stat /bin/cat: no such file or directory": unknown
command terminated with exit code 126

【问题讨论】：

标签： kubernetes coredns

【解决方案1】：

检查您是否在 /etc/hosts 中输入了主 IP 和主机名

nano /etc/hosts
master YOUR_IP

如果没有帮助你试试

# systemctl stop kubelet
# systemctl stop docker
# iptables --flush
# iptables -tnat --flush
# systemctl start kubelet
# systemctl start docker

【讨论】：

所有的master和nodes都在/etc/hosts中。

【解决方案2】：

请开始调试您的 coredns pod 并检查 DNS pod 是否正在运行/重新启动 - 遵循 examples:

for p in $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name); do kubectl logs --namespace=kube-system $p; done

更改 coredns configmap 并添加日志选项：

kubectl -n kube-system edit configmap coredns

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        log
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        proxy . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }

【讨论】：