Minikube 是否在 Ubuntu 19.10 虚拟机中运行？

Question

我在 VMWare Workstation 中运行的 Ubuntu 19.10 上安装了 kubectl 和 minikube，如 k8s docs 所示。当我使用 --vm-driver=none 启动 minikube 时（因为我在虚拟机上运行）我收到此错误：

minikube v1.6.2 on Ubuntu 19.10
  Selecting 'none' driver from user configuration (alternates: [])
  Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
  Using the running none "minikube" VM ...
  Waiting for the host to be provisioned ...
  Preparing Kubernetes v1.17.0 on Docker '19.03.2' ...
     kubelet.resolv-conf=/run/systemd/resolve/resolv.conf
  Launching Kubernetes ...

Error starting cluster: running cmd: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.17.0:$PATH kubeadm init phase certs all --config /var/tmp/minikube/kubeadm.yaml": /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.17.0:$PATH kubeadm init phase certs all --config /var/tmp/minikube/kubeadm.yaml": exit status 1
stdout:
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk

stderr:
W0110 13:53:06.531592    6051 common.go:77] your configuration file uses a deprecated API spec: "kubeadm.k8s.io/v1beta1". Please use 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.
W0110 13:53:06.532094    6051 common.go:77] your configuration file uses a deprecated API spec: "kubeadm.k8s.io/v1beta1". Please use 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.
W0110 13:53:06.533419    6051 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0110 13:53:06.533440    6051 validation.go:28] Cannot validate kubelet config - no validator is available
error execution phase certs/apiserver-kubelet-client: [certs] certificate apiserver-kubelet-client not signed by CA certificate ca: crypto/rsa: verification error
To see the stack trace of this error execute with --v=5 or higher


????  minikube is exiting due to an error. If the above message is not useful, open an issue:
????  https://github.com/kubernetes/minikube/issues/new/choose
❌  Problems detected in kube-addon-manager ["c7ccb7eb00cc"]:
    error: You must be logged in to the server (error when retrieving current configuration of:
    error: You must be logged in to the server (the server has asked for the client to provide credentials)
    error: You must be logged in to the server (the server has asked for the client to provide credentials)
❌  Problems detected in kube-apiserver ["c91043e54554"]:
    I0110 20:52:44.283421       1 log.go:172] http: TLS handshake error from 127.0.0.1:47838: remote error: tls: bad certificate
    I0110 20:52:44.483603       1 log.go:172] http: TLS handshake error from 127.0.0.1:47846: remote error: tls: bad certificate
    I0110 20:52:44.549909       1 log.go:172] http: TLS handshake error from 127.0.0.1:47848: remote error: tls: bad certificate

这是使用版本 5 内核的 Ubuntu 19.10 的问题，还是在 VM 中运行的结果，是否有解决方法？

谢谢。

Answer 1

在this thread VKR 中提供了如何使其工作的详细说明。我将他的答案发布为社区 wiki：

---

好的，所以我在将 minikube 升级到 19.10 后重现并遇到了同样的错误。

我是如何在 19.04 启动集群的：

#Install kubectl
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && chmod +x ./kubectl && sudo mv ./kubectl /usr/local/bin/kubectl

#Install minikube. Make sure to check for latest version
curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

#Install Docker
curl -fsSL get.docker.com -o get-docker.sh && chmod +x get-docker.sh
sh get-docker.sh
sudo usermod -aG docker $USER


export MINIKUBE_WANTUPDATENOTIFICATION=false
export MINIKUBE_WANTREPORTERRORPROMPT=false
export MINIKUBE_HOME=$HOME
export CHANGE_MINIKUBE_NONE_USER=true
export KUBECONFIG=$HOME/.kube/config
sudo minikube start --vm-driver none
sudo chown -R $USER $HOME/.kube $HOME/.minikube


vkr@ubuntu-minikube:~$ docker version
Client: Docker Engine - Community
 Version:           19.03.3
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        a872fc2f86
 Built:             Tue Oct  8 01:00:44 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.3
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       a872fc2f86
  Built:            Tue Oct  8 00:59:17 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683


vkr@ubuntu-minikube:~$ kubectl get pods -A
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
kube-system   coredns-5644d7b6d9-cv8c5           1/1     Running   0          2m25s
kube-system   coredns-5644d7b6d9-gk725           1/1     Running   0          2m25s
kube-system   etcd-minikube                      1/1     Running   0          75s
kube-system   kube-addon-manager-minikube        1/1     Running   0          75s
kube-system   kube-apiserver-minikube            1/1     Running   0          98s
kube-system   kube-controller-manager-minikube   1/1     Running   0          88s
kube-system   kube-proxy-59jp9                   1/1     Running   0          2m25s
kube-system   kube-scheduler-minikube            1/1     Running   0          82s
kube-system   storage-provisioner                1/1     Running   0          2m24s

升级到 19.10 并清理 minikube 安装后：

vkr@ubuntu-minikube:~$ kubectl get all -A
Error in configuration: 
* unable to read client-cert /root/.minikube/client.crt for minikube due to open /root/.minikube/client.crt: permission denied
* unable to read client-key /root/.minikube/client.key for minikube due to open /root/.minikube/client.key: permission denied
* unable to read certificate-authority /root/.minikube/ca.crt for minikube due to open /root/.minikube/ca.crt: permission denied

有很多讨论表明您应该使用 root 代替 none driver，因为 minikube 直接在您的机器上运行 kubernetes 系统组件...

Running minikube as normal user

Can't start minikube-- permissions

https://minikube.sigs.k8s.io/docs/reference/drivers/none/:

使用 none 驱动要求 minikube 以 root 身份运行，直到 #3760可以 解决

但是..这里有一个小技巧给你..

1) 擦除所有内容

vkr@ubuntu-minikube:~$ minikube stop
✋  Stopping "minikube" in none ...
?  "minikube" stopped.
vkr@ubuntu-minikube:~$ minikube delete
?  Uninstalling Kubernetes v1.16.0 using kubeadm ...
?  Deleting "minikube" in none ...
?  The "minikube" cluster has been deleted.
vkr@ubuntu-minikube:~$ rm -rf ~/.kube
vkr@ubuntu-minikube:~$ rm -rf ~/.minikube
vkr@ubuntu-minikube:~$ sudo rm -rf /var/lib/minikube
vkr@ubuntu-minikube:~$ sudo rm -rf /etc/kubernetes
vkr@ubuntu-minikube:~$ sudo rm -rf /root/.minikube
vkr@ubuntu-minikube:~$ sudo rm -rf /usr/local/bin/minikube

2) 安装minikube，导出变量，检查

vkr@ubuntu-minikube:~$  curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
vkr@ubuntu-minikube:~$   export MINIKUBE_WANTUPDATENOTIFICATION=false
vkr@ubuntu-minikube:~$  export MINIKUBE_WANTREPORTERRORPROMPT=false
vkr@ubuntu-minikube:~$  export MINIKUBE_HOME=$HOME
vkr@ubuntu-minikube:~$  export CHANGE_MINIKUBE_NONE_USER=true
vkr@ubuntu-minikube:~$  export KUBECONFIG=$HOME/.kube/config
vkr@ubuntu-minikube:~$  sudo minikube start --vm-driver none

?  minikube v1.4.0 on Ubuntu 19.10
?  Running on localhost (CPUs=2, Memory=7458MB, Disk=9749MB) ...
ℹ️   OS release is Ubuntu 19.10
?  Preparing Kubernetes v1.16.0 on Docker 19.03.3 ...
    ▪ kubelet.resolv-conf=/run/systemd/resolve/resolv.conf
?  Downloading kubelet v1.16.0
?  Downloading kubeadm v1.16.0
?  Pulling images ...
?  Launching Kubernetes ... 
?  Configuring local host environment ...

⚠️  The 'none' driver provides limited isolation and may reduce system security and reliability.
⚠️  For more information, see:
?  https://minikube.sigs.k8s.io/docs/reference/drivers/none/

⚠️  kubectl and minikube configuration will be stored in /root
⚠️  To use kubectl or minikube commands as your own user, you may need to relocate them. For example, to overwrite your own settings, run:

    ▪ sudo mv /root/.kube /root/.minikube $HOME
    ▪ sudo chown -R $USER $HOME/.kube $HOME/.minikube

?  This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
⌛  Waiting for: apiserver proxy etcd scheduler controller dns
?  Done! kubectl is now configured to use "minikube"

我接下来要做的是将/root/.kube 和/root/.minikube 中的所有内容复制到$HOME，授予用户权限，最后编辑$HOME/.kube/config，指定新的证书路径（$HOME/.minikube/ 而不是/root/.minikube/）。现在看起来像

vkr@ubuntu-minikube:~$ cat $KUBECONFIG
apiVersion: v1
...
    certificate-authority: /root/.minikube/ca.crt
...
    client-certificate: /root/.minikube/client.crt
    client-key: /root/.minikube/client.key

让我们做吧:)

vkr@ubuntu-minikube:~$ sudo cp -r /root/.kube  /root/.minikube $HOME
vkr@ubuntu-minikube:~$ sudo chown -R $USER $HOME/.kube
vkr@ubuntu-minikube:~$ sudo chown -R $USER $HOME/.minikube
sed 's/root/home\/vkr/g' $KUBECONFIG > tmp; mv tmp $KUBECONFIG

最终结果..

vkr@ubuntu-minikube:~$ kubectl get all -A
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-5644d7b6d9-bt897           1/1     Running   0          81m
kube-system   pod/coredns-5644d7b6d9-hkm5t           1/1     Running   0          81m
kube-system   pod/etcd-minikube                      1/1     Running   0          80m
kube-system   pod/kube-addon-manager-minikube        1/1     Running   0          80m
kube-system   pod/kube-apiserver-minikube            1/1     Running   0          80m
kube-system   pod/kube-controller-manager-minikube   1/1     Running   0          80m
kube-system   pod/kube-proxy-wm52p                   1/1     Running   0          81m
kube-system   pod/kube-scheduler-minikube            1/1     Running   0          80m
kube-system   pod/storage-provisioner                1/1     Running   0          81m

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  81m
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   81m

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           beta.kubernetes.io/os=linux   81m

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   2/2     2            2           81m

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-5644d7b6d9   2         2         2       81m

---

如果有帮助，请告诉我。

Answer 2

是的，它应该可以工作。

以下说明应适用于 Debian / Ubuntu / Mint Linux 发行版。如果使用 RHEL / Arch / Other 发行版或 Ubuntu 服务器等非桌面发行版，或可能省略许多预期工具的轻量级发行版，您的体验可能会有所不同。

安装 VirtualBox：

找到您的 Linux 发行版并下载 .deb 软件包，此处使用图形安装程序就足够了。如果您使用像 apt 这样的包管理器从您的终端安装，您可能会得到一个相当过时的版本。

https://www.virtualbox.org/wiki/Linux_Downloads

安装后，检查您的安装以确保其正常工作：

VBoxManage —version

作为替代方案，您可以使用（或者可能必须使用）KVM 代替 VirtualBox。在这篇文章中可以找到一些很棒的说明：

https://computingforgeeks.com/install-kvm-centos-rhel-ubuntu-debian-sles-arch/

安装 Kubectl

在您的终端中运行以下命令：

curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl

chmod +x ./kubectl

sudo mv ./kubectl /usr/local/bin/kubectl

检查您的安装：

kubectl version

另见官方文档： https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-linux

安装 Minikube

在您的终端中运行以下命令：

curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube

sudo install minikube /usr/local/bin

检查您的安装：

minikube version

启动 Minikube：

minikube start

另见官方文档：

https://kubernetes.io/docs/tasks/tools/install-minikube/