创建 pod 沙箱失败：rpc 错误：code = Unknown desc = failed to setup sandbox container答案

【问题标题】：Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container创建 pod 沙箱失败：rpc 错误：code = Unknown desc = failed to setup sandbox container
【发布时间】：2021-04-02 01:29:57
【问题描述】：

我们正在尝试创建 POD，但是 Pod 的状态在 ContainerCreating 中停留了很长时间。

这是我们运行命令后得到的输出：kubectl describe pod

Name:           demo-6c59fb8f77-9x6sr
Namespace:      default
Priority:       0
Node:           k8-slave2/10.0.0.5
Start Time:     Wed, 23 Dec 2020 10:16:23 +0000
Labels:         app=demo
                pod-template-hash=6c59fb8f77
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/demo-6c59fb8f77
Containers:
  private-docker-registry:
    Container ID:
    Image:          private-docker-registry:5000/mahin/mof-docker-demo:v1
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-p94zw (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-p94zw:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-p94zw
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  

  Events:
      Type     Reason                  Age                  From               Message
      ----     ------                  ----                 ----               -------
      Normal   Scheduled               10m                  default-scheduler  Successfully assigned default/demo-6c59fb8f77-9x6sr to k8-slave2
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "8eee497a2176c7f5782222f804cc63a4abac7f4a2fc7813016793857ae1b1dff" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "95e72bfc6f6c13de7f5c96eb76b012c2e6639ca03f4c2f270b23ed1a09b90413" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "566370012e4a1d32af2ef9035ff64d743cd81f36f25d2724e7b033e393b8247e" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7d499e40f572cfc29ecfb44f8376493df56a44213b1c1e9333b65499a0c288cd" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "53241e64de1e4470712b4061e2c82f44916d654bc532f8f1d12e5d5d4e136914" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fd168faab4546f988dc38fc56df2f71cf80c922e86d3f869be15a43f08328f99" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e578afe329abb0cba64802dfa480e00f2bbbb8c80be537791c24a31c853eb62f" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a3cb32dba55907ca907fc4f38f7ca05ef6db10a6af2dd1fa3c4db166e4ab9ffe" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7e4368ba8ec460b3c94de24ab0a04b6c799eb28df885cbbacfc3bb3ffa8c1e67" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m (x4 over 10m)    kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c4aaa8f8cd2dc1eff788baf04774c4ecc845568d00ed1b386df311ec224eb6f3" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Normal   SandboxChanged          56s (x551 over 10m)  kubelet            Pod sandbox changed, it will be killed and re-created.



azureuser@k8-master:~$ kubectl get pods --all-namespaces
NAMESPACE              NAME                                         READY   STATUS              RESTARTS   AGE
default                demo-6c59fb8f77-2jq6k                        0/1     ContainerCreating   0          5m23s
kube-system            coredns-f9fd979d6-q8s9b                      1/1     Running             2          27h
kube-system            coredns-f9fd979d6-qnm4j                      1/1     Running             2          27h
kube-system            etcd-k8-master                               1/1     Running             2          27h
kube-system            kube-apiserver-k8-master                     1/1     Running             3          27h
kube-system            kube-controller-manager-k8-master            1/1     Running             3          27h
kube-system            kube-flannel-ds-kqz4t                        0/1     CrashLoopBackOff    92         27h
kube-system            kube-flannel-ds-szqzn                        1/1     Running             3          27h
kube-system            kube-flannel-ds-v9q47                        0/1     CrashLoopBackOff    142        27h
kube-system            kube-proxy-4mb47                             1/1     Running             2          27h
kube-system            kube-proxy-54m9b                             1/1     Running             2          27h
kube-system            kube-proxy-wdxfz                             1/1     Running             1          27h
kube-system            kube-scheduler-k8-master                     1/1     Running             3          27h
kubernetes-dashboard   dashboard-metrics-scraper-7b59f7d4df-zmlvs   0/1     ContainerCreating   0          27h
kubernetes-dashboard   kubernetes-dashboard-665f4c5ff-cnsvn         0/1     ContainerCreating   0          6h3m

为了修复 flannel crashloopbackoff，我们重置了 Kubeadm，过了一段时间这个问题又出现了。

目前我们正在使用一个主节点和两个工作节点。

我的集群详情如下： azureuser@k8-master:~$ kubectl 配置视图 api版本：v1 集群： - 簇：证书授权数据：DATA+OMITTED 服务器：https://52.150.11.168:6443 名称：kubernetes 上下文： - 语境：集群：Kubernetes 用户：kubernetes-admin 名称：kubernetes-admin@kubernetes 当前上下文：kubernetes-admin@kubernetes 种类：配置优先： {} 用户： - 名称：kubernetes-admin 用户：客户证书数据：已编辑客户密钥数据：已编辑

Docker 版本：

azureuser@k8-master:~$ sudo docker version
[sudo] password for azureuser: 
Client:
 Version:           19.03.6
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        369ce74a3c
 Built:             Wed Oct 14 19:00:27 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.6
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       369ce74a3c
  Built:            Wed Oct 14 16:52:50 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu1~18.04.2
  GitCommit:
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:
 docker-init:
  Version:          0.18.0
  GitCommit:

kubeadm 版本：

azureuser@k8-master:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-11T13:15:05Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

每当我尝试安排 pod 创建时，法兰绒都会崩溃。

【问题讨论】：

您好，您需要查看 flanel pods 的日志吗？为什么是 crashloopbackoffing
请将错误输出粘贴为文本，而不是图片。你能分享更多细节，比如 Kubernetes 版本，你的环境吗？您是否尝试过除 flanner 之外的其他 CNI，例如 calico？您能否提供一些步骤来复制此问题？您是否更改了集群中的节点数？您能否提供描述其他受影响豆荚的一些豆荚？你有足够的资源（RAM、CPU）吗？是否有任何污点、防火墙更改？
不，我们没有使用除 Flannel 之外的任何其他 CNI。并对您提到的帖子进行了更改。是的，我们有足够的资源。
你检查过你提到的目录中是否有文件吗？ /run/flannel/subnet.env ?如果没有，您可以按照this article 中的说明手动创建它或重新部署flannel。您还可以分享受影响的法兰绒 pod 日志kubectl logs <flannelPod> -n kube-system 吗？使用kubeadm init之前部署flannel了吗？
是的，我在执行 kubeadm init 后安装了 flannel。

标签： kubernetes kubectl kubeadm

【解决方案1】：

背景

我认为您的问题是由您的2 Flannel CNI pods CrashLoopBackOff 状态引起的。

你的错误

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "8eee497a2176c7f5782222f804cc63a4abac7f4a2fc7813016793857ae1b1dff" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory

指出由于缺少 /run/flannel/subnet.env 文件而无法创建 pod。在Flannel Github 文档中可以找到：

Flannel 在每台主机上运行一个名为 flanneld的小型单一二进制代理，并负责从更大的预配置地址空间中为每台主机分配子网租约。

意思是，为了正常工作，Flannel pod 应该在每个节点上运行，因为它包含子网信息。从您的输出中，我可以看到 只有 1 个 正常工作 out of 3 Flannel pods。

NAMESPACE              NAME                                         READY   STATUS              RESTARTS   AGE
...
kube-system            kube-flannel-ds-kqz4t                        0/1     CrashLoopBackOff    92         27h
kube-system            kube-flannel-ds-szqzn                        1/1     Running             3          27h
kube-system            kube-flannel-ds-v9q47                        0/1     CrashLoopBackOff    142        27h

如果提到的 pod 被安排在 flannel pod 不工作的节点上，由于CNI network issues，它将不会被创建。除了您的 demo pod 之外，kubernetes-dashboard pod 也有与 ContainerCreating 状态相同的问题。

结论

您的demo pod 无法调度，因为 Kubernetes 遇到一些与 flannel 配置文件 (...network: open /run/flannel/subnet.env: no such file or directory) 相关的网络问题。

对于27 hours，您的法兰绒吊舱重新启动计数非常高。您必须确定原因并修复它。这可能是缺乏资源、基础架构的网络问题或许多其他原因。一旦所有 flannel pod 都可以正常工作，您就不会遇到此错误。

解决方案

你必须让 flannel pods 在每个节点上都能正常工作。

其他故障排除详细信息

详细调查请提供

$ kubectl describe kube-flannel-ds-kqz4t -n kube-system
$ kubectl describe kube-flannel-ds-v9q47 -n kube-system

日志详细信息也会有所帮助

$ kubectl logs kube-flannel-ds-kqz4t -n kube-system
$ kubectl logs kube-flannel-ds-v9q47 -n kube-system

请将kubectl get pods --all-namespaces 替换为kubectl get pods -o wide -A 和kubectl get nodes -o wide 的输出。

如果您提供这些信息，应该可以确定 flannel pod 问题的根本原因，我将使用确切的解决方案编辑此答案。

【讨论】：

@Giridharan M 你能解决你的问题吗？
嗨@PjoterS，我有同样的问题，介意你提供一些帮助吗？我对k get po -o wide -A 的输出显示 coredns 'Completed' 和 flannel-ds 'CrashLoopBackOff'；第二个命令显示所有节点“就绪”
@Spencer Trinh 如果您在堆栈上创建带有一些细节的新问题会更好（例如描述 o flanne pod、环境细节和复制问题的步骤）。
我在github.com/kubernetes/kubernetes/issues/70202 找到了解决方案，但是每次重启我都需要重新创建这个文件。这是正常行为吗？