【发布时间】：2019-12-17 05:55:46
【问题描述】：
我运行一个带有几个节点池的小型 GKE 集群（每个节点池 2-8 个节点，一些是可抢占的）。我开始看到节点本身存在很多健康问题，并且 pod 操作需要很长时间（30 多分钟）。这包括终止 pod、启动 pod、在 pod 中启动 initContainers、在 pod 中启动主容器等。示例如下。 集群运行一些 NodeJS、PHP 和 Nginx 容器，以及单个 Elastic、Redis 和 NFS pod。此外，还有一些基于 PHP 的 CronJobs。它们共同组成了一个位于 CDN 后面的网站。
- 我的问题是：如何在 GKE 上进行调试，原因可能是什么？
我尝试通过 SSH 连接到支持节点的 VM 实例以检查日志，但我的 SSH 连接总是超时，不确定这是否正常。
症状：节点在Ready 和NotReady 之间摆动：
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-cluster-default-pool-4fa127c-l3xt Ready <none> 62d v1.13.6-gke.13
gke-cluster-default-pool-791e6c2-7b01 NotReady <none> 45d v1.13.6-gke.13
gke-cluster-preemptible-0f81875-cc5q Ready <none> 3h40m v1.13.6-gke.13
gke-cluster-preemptible-0f81875-krqk NotReady <none> 22h v1.13.6-gke.13
gke-cluster-preemptible-0f81875-mb05 Ready <none> 5h42m v1.13.6-gke.13
gke-cluster-preemptible-2453785-1c4v Ready <none> 22h v1.13.6-gke.13
gke-cluster-preemptible-2453785-nv9q Ready <none> 134m v1.13.6-gke.13
gke-cluster-preemptible-2453785-s7r2 NotReady <none> 22h v1.13.6-gke.13
症状：节点有时会重新启动：
2019-08-09 14:23:54.000 CEST
Node gke-cluster-preemptible-0f81875-mb05 has been rebooted, boot id: e601f182-2eab-46b0-a953-7787f95d438
症状：集群不健康：
2019-08-09T11:29:03Z Cluster is unhealthy
2019-08-09T11:33:25Z Cluster is unhealthy
2019-08-09T11:41:08Z Cluster is unhealthy
2019-08-09T11:45:10Z Cluster is unhealthy
2019-08-09T11:49:11Z Cluster is unhealthy
2019-08-09T11:53:23Z Cluster is unhealthy
症状：节点日志中出现各种 PLEG 健康错误（这种类型的条目很多很多）：
12:53:10.573176 1315163 kubelet.go:1854] skipping pod synchronization - [PLEG is not healthy: pleg was last seen active 3m26.30454685s ago; threshold is 3m0s]
12:53:18.126428 1036 setters.go:520] Node became not ready: {Type:Ready Status:False LastHeartbeatTime:2019-08-09 12:53:18.126363615 +0000 UTC m=+3924434.187952856 LastTransitionTime:2019-08-09 12:53:18.126363615 +0000 UTC m=+3924434.187952856 Reason:KubeletNotReady Message:PLEG is not healthy: pleg was last seen active 3m5.837134315s ago; threshold is 3m0s}
12:53:38.627284 1036 kubelet.go:1854] skipping pod synchronization - [PLEG is not healthy: pleg was last seen active 3m26.338024015s ago; threshold is 3m0s]
症状：Pod 发出“网络未就绪”错误：
2019-08-09T12:42:45Z network is not ready: [runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized]
2019-08-09T12:42:47Z network is not ready: [runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized]
2019-08-09T12:42:49Z network is not ready: [runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized]
症状：Pod 抱怨“超出上下文期限”：
2019-08-09T08:04:07Z error determining status: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2019-08-09T08:04:15Z error determining status: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2019-08-09T08:04:20Z error determining status: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2019-08-09T08:04:26Z error determining status: rpc error: code = DeadlineExceeded desc = context deadline exceeded
显然发生了一些特别奇怪的事情，但是 IOPS、入口请求、cpu/内存饱和的数量相当微不足道。我预计会有一些症状指出我可以进一步调试的方向。但似乎这些错误无处不在。
【问题讨论】：
标签： kubernetes google-kubernetes-engine