Ubuntu 升级到 19.10 后 Minikube 将无法工作

Question

我只是将 Ubuntu 从 19.04 升级到 19.10

现在 Minikube 无法启动。

所以，过了一会儿，我刚刚完全删除了 Minikube 。

minikube stop; minikube delete
docker stop $(docker ps -aq)
rm -r ~/.kube ~/.minikube
sudo rm /usr/local/bin/localkube /usr/local/bin/minikube
systemctl stop '*kubelet*.mount'
sudo rm -rf /etc/kubernetes/
docker system prune -af --volumes

现在我想重新安装所有内容，但无法正常工作。

我下载了minikube，移到/usr/local/bin

curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \\n  && chmod +x minikube && sudo mv ./minikube /usr/local/bin

我启动 minikube

sudo minikube start --vm-driver=none

一切正常，minukube启动成功。

~ sudo minikube start --vm-driver=none
????  minikube v1.4.0 on Ubuntu 19.10
????  Running on localhost (CPUs=4, Memory=7847MB, Disk=280664MB) ...
ℹ️   OS release is Ubuntu 19.10
????  Preparing Kubernetes v1.16.0 on Docker 18.09.6 ...
    ▪ kubelet.resolv-conf=/run/systemd/resolve/resolv.conf
????  Pulling images ...
????  Launching Kubernetes ... 
????  Configuring local host environment ...

⚠️  The 'none' driver provides limited isolation and may reduce system security and reliability.
⚠️  For more information, see:
????  https://minikube.sigs.k8s.io/docs/reference/drivers/none/

⚠️  kubectl and minikube configuration will be stored in /root
⚠️  To use kubectl or minikube commands as your own user, you may need to relocate them. For example, to overwrite your own settings, run:

    ▪ sudo mv /root/.kube /root/.minikube $HOME
    ▪ sudo chown -R $USER $HOME/.kube $HOME/.minikube

????  This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
⌛  Waiting for: apiserver proxy etcd scheduler controller dns
????  Done! kubectl is now configured to use "minikube"

我终于做到了：

   ~ sudo mv /root/.kube /root/.minikube $HOME
➜  ~ sudo chown -R $USER $HOME/.kube $HOME/.minikube

但是当我想检查 pod 时：

kubectl get po

我明白了：

➜  ~ kubectl get po
Error in configuration: 
* unable to read client-cert /root/.minikube/client.crt for minikube due to open /root/.minikube/client.crt: permission denied
* unable to read client-key /root/.minikube/client.key for minikube due to open /root/.minikube/client.key: permission denied
* unable to read certificate-authority /root/.minikube/ca.crt for minikube due to open /root/.minikube/ca.crt: permission denied

如果使用 sudo：

 ~ sudo kubectl get po
[sudo] password for julien: 
The connection to the server localhost:8080 was refused - did you specify the right host or port?

这是minikube logs的结果

https://gist.github.com/xoco70/8a9c7042238400e370796cb23cb11c88

我该怎么办？

编辑：

重启后，当启动 minikube 时：

sudo minikube start --vm-driver=none

我明白了：

Error starting cluster: cmd failed: sudo env PATH=/var/lib/minikube/binaries/v1.16.0:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap

: running command: sudo env PATH=/var/lib/minikube/binaries/v1.16.0:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap
 output: [init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
    [WARNING FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
    [WARNING FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
    [WARNING FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
    [WARNING FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
    [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
    [WARNING Swap]: running with swap on is not supported. Please disable swap
    [WARNING FileExisting-ethtool]: ethtool not found in system path
    [WARNING FileExisting-socat]: socat not found in system path
    [WARNING Hostname]: hostname "minikube" could not be reached
    [WARNING Hostname]: hostname "minikube": lookup minikube on 8.8.8.8:53: no such host
    [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
    [WARNING Port-10250]: Port 10250 is in use
error execution phase preflight: [preflight] Some fatal errors occurred:
    [ERROR Port-8443]: Port 8443 is in use
    [ERROR Port-10251]: Port 10251 is in use
    [ERROR Port-10252]: Port 10252 is in use
    [ERROR Port-2379]: Port 2379 is in use
    [ERROR Port-2380]: Port 2380 is in use
    [ERROR DirAvailable--var-lib-minikube-etcd]: /var/lib/minikube/etcd is not empty
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
: running command: sudo env PATH=/var/lib/minikube/binaries/v1.16.0:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap
.: exit status 1

????  Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
????  https://github.com/kubernetes/minikube/issues/new/choose
❌  Problems detected in kube-apiserver [3e0d8c59345d]:
    I1025 07:09:56.349120       1 log.go:172] http: TLS handshake error from 127.0.0.1:46254: remote error: tls: bad certificate
    I1025 07:09:56.353714       1 log.go:172] http: TLS handshake error from 127.0.0.1:46082: remote error: tls: bad certificate
    I1025 07:09:56.353790       1 log.go:172] http: TLS handshake error from 127.0.0.1:46080: remote error: tls: bad certificate

Answer 1

好的，所以我在将 minikube 升级到 19.10 后重现并遇到了同样的错误。

我是如何在 19.04 启动集群的：

#Install kubectl
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && chmod +x ./kubectl && sudo mv ./kubectl /usr/local/bin/kubectl

#Install minikube. Make sure to check for latest version
curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

#Install Docker
curl -fsSL get.docker.com -o get-docker.sh && chmod +x get-docker.sh
sh get-docker.sh
sudo usermod -aG docker $USER


export MINIKUBE_WANTUPDATENOTIFICATION=false
export MINIKUBE_WANTREPORTERRORPROMPT=false
export MINIKUBE_HOME=$HOME
export CHANGE_MINIKUBE_NONE_USER=true
export KUBECONFIG=$HOME/.kube/config
sudo minikube start --vm-driver none
sudo chown -R $USER $HOME/.kube $HOME/.minikube


vkr@ubuntu-minikube:~$ docker version
Client: Docker Engine - Community
 Version:           19.03.3
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        a872fc2f86
 Built:             Tue Oct  8 01:00:44 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.3
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       a872fc2f86
  Built:            Tue Oct  8 00:59:17 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683


vkr@ubuntu-minikube:~$ kubectl get pods -A
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
kube-system   coredns-5644d7b6d9-cv8c5           1/1     Running   0          2m25s
kube-system   coredns-5644d7b6d9-gk725           1/1     Running   0          2m25s
kube-system   etcd-minikube                      1/1     Running   0          75s
kube-system   kube-addon-manager-minikube        1/1     Running   0          75s
kube-system   kube-apiserver-minikube            1/1     Running   0          98s
kube-system   kube-controller-manager-minikube   1/1     Running   0          88s
kube-system   kube-proxy-59jp9                   1/1     Running   0          2m25s
kube-system   kube-scheduler-minikube            1/1     Running   0          82s
kube-system   storage-provisioner                1/1     Running   0          2m24s

升级到 19.10 并清理 minikube 安装后：

vkr@ubuntu-minikube:~$ kubectl get all -A
Error in configuration: 
* unable to read client-cert /root/.minikube/client.crt for minikube due to open /root/.minikube/client.crt: permission denied
* unable to read client-key /root/.minikube/client.key for minikube due to open /root/.minikube/client.key: permission denied
* unable to read certificate-authority /root/.minikube/ca.crt for minikube due to open /root/.minikube/ca.crt: permission denied

有很多讨论表明您应该使用 root 代替 none driver，因为 minikube 直接在您的机器上运行 kubernetes 系统组件...

Running minikube as normal user

Can't start minikube-- permissions

https://minikube.sigs.k8s.io/docs/reference/drivers/none/:

使用 none 驱动要求 minikube 以 root 身份运行，直到 #3760可以 解决

但是..这里有一个小技巧给你..

1) 擦除所有内容

vkr@ubuntu-minikube:~$ minikube stop
✋  Stopping "minikube" in none ...
?  "minikube" stopped.
vkr@ubuntu-minikube:~$ minikube delete
?  Uninstalling Kubernetes v1.16.0 using kubeadm ...
?  Deleting "minikube" in none ...
?  The "minikube" cluster has been deleted.
vkr@ubuntu-minikube:~$ rm -rf ~/.kube
vkr@ubuntu-minikube:~$ rm -rf ~/.minikube
vkr@ubuntu-minikube:~$ sudo rm -rf /var/lib/minikube
vkr@ubuntu-minikube:~$ sudo rm -rf /etc/kubernetes
vkr@ubuntu-minikube:~$ sudo rm -rf /root/.minikube
vkr@ubuntu-minikube:~$ sudo rm -rf /usr/local/bin/minikube

2) 安装minikube，导出变量，检查

vkr@ubuntu-minikube:~$  curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
vkr@ubuntu-minikube:~$   export MINIKUBE_WANTUPDATENOTIFICATION=false
vkr@ubuntu-minikube:~$  export MINIKUBE_WANTREPORTERRORPROMPT=false
vkr@ubuntu-minikube:~$  export MINIKUBE_HOME=$HOME
vkr@ubuntu-minikube:~$  export CHANGE_MINIKUBE_NONE_USER=true
vkr@ubuntu-minikube:~$  export KUBECONFIG=$HOME/.kube/config
vkr@ubuntu-minikube:~$  sudo minikube start --vm-driver none

?  minikube v1.4.0 on Ubuntu 19.10
?  Running on localhost (CPUs=2, Memory=7458MB, Disk=9749MB) ...
ℹ️   OS release is Ubuntu 19.10
?  Preparing Kubernetes v1.16.0 on Docker 19.03.3 ...
    ▪ kubelet.resolv-conf=/run/systemd/resolve/resolv.conf
?  Downloading kubelet v1.16.0
?  Downloading kubeadm v1.16.0
?  Pulling images ...
?  Launching Kubernetes ... 
?  Configuring local host environment ...

⚠️  The 'none' driver provides limited isolation and may reduce system security and reliability.
⚠️  For more information, see:
?  https://minikube.sigs.k8s.io/docs/reference/drivers/none/

⚠️  kubectl and minikube configuration will be stored in /root
⚠️  To use kubectl or minikube commands as your own user, you may need to relocate them. For example, to overwrite your own settings, run:

    ▪ sudo mv /root/.kube /root/.minikube $HOME
    ▪ sudo chown -R $USER $HOME/.kube $HOME/.minikube

?  This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
⌛  Waiting for: apiserver proxy etcd scheduler controller dns
?  Done! kubectl is now configured to use "minikube"

接下来我要做的是将/root/.kube 和/root/.minikube 中的所有内容复制到$HOME，授予用户权限，最后编辑$HOME/.kube/config，指定新的证书路径（$HOME/.minikube/ 而不是/root/.minikube/）。现在看起来像

vkr@ubuntu-minikube:~$ cat $KUBECONFIG
apiVersion: v1
...
    certificate-authority: /root/.minikube/ca.crt
...
    client-certificate: /root/.minikube/client.crt
    client-key: /root/.minikube/client.key

让我们做吧:)

vkr@ubuntu-minikube:~$ sudo cp -r /root/.kube  /root/.minikube $HOME
vkr@ubuntu-minikube:~$ sudo chown -R $USER $HOME/.kube
vkr@ubuntu-minikube:~$ sudo chown -R $USER $HOME/.minikube
sed 's/root/home\/vkr/g' $KUBECONFIG > tmp; mv tmp $KUBECONFIG

最终结果..

vkr@ubuntu-minikube:~$ kubectl get all -A
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-5644d7b6d9-bt897           1/1     Running   0          81m
kube-system   pod/coredns-5644d7b6d9-hkm5t           1/1     Running   0          81m
kube-system   pod/etcd-minikube                      1/1     Running   0          80m
kube-system   pod/kube-addon-manager-minikube        1/1     Running   0          80m
kube-system   pod/kube-apiserver-minikube            1/1     Running   0          80m
kube-system   pod/kube-controller-manager-minikube   1/1     Running   0          80m
kube-system   pod/kube-proxy-wm52p                   1/1     Running   0          81m
kube-system   pod/kube-scheduler-minikube            1/1     Running   0          80m
kube-system   pod/storage-provisioner                1/1     Running   0          81m

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  81m
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   81m

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           beta.kubernetes.io/os=linux   81m

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   2/2     2            2           81m

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-5644d7b6d9   2         2         2       81m

Answer 2

一个有点令人困惑的问题，你的故事中有些不匹配。

您在询问如何“修复 Minikube 并保留我的集群”，同时提到您删除了本地集群，但仍然存在问题。

我只能在这里假设您最终破坏了集群并且不需要恢复它。

看，您说尽管您的集群尚未启动（Error restarting cluster:exit status 1,Sorry that minikube crashed. 错误） - 命令 sudo kubectl get po -A 仍然有效。而且你在普通用户下做同样的事情会有一些问题。

这里的问题 - “sudo kubectl get po -A 仍然有效”是什么意思？你看到豆荚了吗？你能看到其他物体吗？

如果是这样，您应该首先查看etcd direction。此外，您在启动过程中有etcd 错误。

: running command: sudo ln -s /data/minikube /var/lib/minikube/etcd
 output: ln: failed to create symbolic link '/var/lib/minikube/etcd/minikube': File exists
: running command: sudo ln -s /data/minikube /var/lib/minikube/etcd
.: exit status 1

Etcd 是 k8s 的主键值数据存储，它存储和复制所有 kubernetes 集群状态，很可能因此我认为存在问题。

也很高兴知道您是如何删除集群的。我敢打赌你已经完成了minikube delete 并且忘记了手动删除不必要的配置。正确方法：

minikube stop
minikube delete
rm -rf ~/.kube
rm -rf ~/.minikube
sudo rm -rf /var/lib/minikube
sudo rm /var/lib/kubeadm.yaml
sudo rm -rf /etc/kubernetes

删除所有内容并从头开始安装。