如何调试失败的 kubectl 调用？

Question

我正在使用 Suse 12.01 Enterprise 并试图让 Minikube 正常工作。虚拟机已经工作，minikube shell 工具可以通信了。

但是 kubectl 仍然无法与 kubernetes master 对话。我正在尝试对其进行调试，而获取附加信息的最佳方法似乎是使用-v 9 运行随机命令。这样做我得到以下输出：

$ kubectl get pots -v 9
I0310 14:02:27.727767   29330 loader.go:354] Config loaded from file /home/D069407/.kube/config
I0310 14:02:27.728479   29330 round_trippers.go:299] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.5.3 (linux/amd64) kubernetes/029c3a4" https://192.168.99.104:8443/api
I0310 14:03:42.704009   29330 round_trippers.go:318] GET https://192.168.99.104:8443/api  in 74975 milliseconds
I0310 14:03:42.704037   29330 round_trippers.go:324] Response Headers:
I0310 14:03:42.704103   29330 helpers.go:221] Connection error: Get https://192.168.99.104:8443/api: Service Unavailable
F0310 14:03:42.704111   29330 helpers.go:116] Unable to connect to the server: Service Unavailable

信息不多，但我猜curl -k -vvvv .... 会给我更多信息。但是，只是执行与日志中相同的 curl 会导致身份验证错误，因为 api 服务器执行客户端身份验证（对吗？）。所以这行不通。

我现在如何继续调试？是否有一种秘密的忍者方法可以在不将它们添加到 shell 调用的情况下提供 curl auth 参数？ kubectl 是否真的在执行另一个请求，只是将 curl 打印到日志中以提供一些关于它正在调用的内容的提示？

*edit：与同事讨论时，我们都同意这一定是 minikube vm 内部问题。

minikube logs 是这样开始的：

-- Logs begin at Fri 2017-03-10 12:43:34 UTC, end at Fri 2017-03-10 14:44:11 UTC. --
Mar 10 12:45:45 minikube systemd[1]: Starting Localkube...
Mar 10 12:45:45 minikube localkube[3496]: I0310 12:45:45.977140    3496 start.go:77] Feature gates:%!(EXTRA string=)
Mar 10 12:45:45 minikube localkube[3496]: localkube host ip address: 10.0.2.15
Mar 10 12:45:45 minikube localkube[3496]: I0310 12:45:45.981395    3496 server.go:215] Using iptables Proxier.
Mar 10 12:45:45 minikube localkube[3496]: W0310 12:45:45.981764    3496 server.go:468] Failed to retrieve node info: Get http://127.0.0.1:8080/api/v1/nodes/minikube: dial tcp 127.0.0.1:8080:
getsockopt: connection refused
Mar 10 12:45:45 minikube localkube[3496]: W0310 12:45:45.981879    3496 proxier.go:249] invalid nodeIP, initialize kube-proxy with 127.0.0.1 as nodeIP
Mar 10 12:45:45 minikube localkube[3496]: W0310 12:45:45.981947    3496 proxier.go:254] clusterCIDR not specified, unable to distinguish between internal and external traffic
Mar 10 12:45:45 minikube localkube[3496]: I0310 12:45:45.982082    3496 server.go:227] Tearing down userspace rules.
Mar 10 12:45:45 minikube localkube[3496]: Starting etcd...
Mar 10 12:45:45 minikube localkube[3496]: E0310 12:45:45.991070    3496 reflector.go:188] pkg/proxy/config/api.go:30: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?r
esourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
Mar 10 12:45:45 minikube localkube[3496]: E0310 12:45:45.991108    3496 reflector.go:188] pkg/proxy/config/api.go:33: Failed to list *api.Endpoints: Get http://127.0.0.1:8080/api/v1/endpoint
s?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
Mar 10 12:45:45 minikube localkube[3496]: name = kubeetcd
Mar 10 12:45:45 minikube localkube[3496]: data dir = /var/lib/localkube/etcd
Mar 10 12:45:45 minikube localkube[3496]: member dir = /var/lib/localkube/etcd/member
Mar 10 12:45:45 minikube localkube[3496]: heartbeat = 100ms
Mar 10 12:45:45 minikube localkube[3496]: election = 1000ms
Mar 10 12:45:45 minikube localkube[3496]: snapshot count = 10000
Mar 10 12:45:45 minikube localkube[3496]: advertise client URLs = http://0.0.0.0:2379
Mar 10 12:45:45 minikube localkube[3496]: initial advertise peer URLs = http://0.0.0.0:2380
Mar 10 12:45:45 minikube localkube[3496]: initial cluster = kubeetcd=http://0.0.0.0:2380
Mar 10 12:45:45 minikube localkube[3496]: starting member fcf2ad36debdd5bb in cluster 7f055ae3b0912328
Mar 10 12:45:45 minikube localkube[3496]: fcf2ad36debdd5bb became follower at term 0
Mar 10 12:45:45 minikube localkube[3496]: newRaft fcf2ad36debdd5bb [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
Mar 10 12:45:45 minikube localkube[3496]: fcf2ad36debdd5bb became follower at term 1
Mar 10 12:45:46 minikube localkube[3496]: starting server... [version: 3.0.14, cluster version: to_be_decided]
Mar 10 12:45:46 minikube localkube[3496]: Starting apiserver...
Mar 10 12:45:46 minikube localkube[3496]: Starting controller-manager...
Mar 10 12:45:46 minikube localkube[3496]: Starting scheduler...
Mar 10 12:45:46 minikube localkube[3496]: Starting kubelet...
Mar 10 12:45:46 minikube localkube[3496]: added member fcf2ad36debdd5bb [http://0.0.0.0:2380] to cluster 7f055ae3b0912328
Mar 10 12:45:46 minikube localkube[3496]: Starting proxy...
Mar 10 12:45:46 minikube localkube[3496]: Starting storage-provisioner...

不过，在minikube ssh 内部，服务 api 可以工作。检查curl 127.0.0.1:8080/api 并收到不错的 json。

*编辑：根据反馈提供更多信息。

卷曲 inside minikube vm:

$ curl localhost:8080
{
"paths": [
    "/api",
    "/api/v1",
    "/apis",
    "/apis/apps",
    "/apis/apps/v1beta1",
    "/apis/authentication.k8s.io",
    "/apis/authentication.k8s.io/v1beta1",
    "/apis/authorization.k8s.io",
    "/apis/authorization.k8s.io/v1beta1",
    "/apis/autoscaling",
    "/apis/autoscaling/v1",
    "/apis/batch",
    "/apis/batch/v1",
    "/apis/batch/v2alpha1",
    "/apis/certificates.k8s.io",
    "/apis/certificates.k8s.io/v1alpha1",
    "/apis/extensions",
    "/apis/extensions/v1beta1",
    "/apis/policy",
    "/apis/policy/v1beta1",
    "/apis/rbac.authorization.k8s.io",
    "/apis/rbac.authorization.k8s.io/v1alpha1",
    "/apis/storage.k8s.io",
    "/apis/storage.k8s.io/v1beta1",
    "/healthz",
    "/healthz/poststarthook/bootstrap-controller",
    "/healthz/poststarthook/extensions/third-party-resources",
    "/healthz/poststarthook/rbac/bootstrap-roles",
    "/logs",
    "/metrics",
    "/swaggerapi/",
    "/ui/",
    "/version"
]
}$ curl localhost:8080/api
{
"kind": "APIVersions",
"versions": [
    "v1"
],
"serverAddressByClientCIDRs": [
    {
    "clientCIDR": "0.0.0.0/0",
    "serverAddress": "10.0.2.15:8443"
    }
]
}

kube 配置（虚拟机外）：

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /home/<user>/.minikube/ca.crt
    server: https://192.168.99.104:8443
name: minikube
contexts:
- context:
    cluster: minikube
    user: minikube
name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
user:
    client-certificate: /home/<user>/.minikube/apiserver.crt
    client-key: /home/<user>/.minikube/apiserver.key

Answer 1

您说您尝试过curl 127.0.0.1:8080/api，但关于它尝试通过 https 连接的日志。所以你应该试试curl https://127.0.0.1:8080/api。

我在源代码中搜索了术语 Service Unavailable，并在 for example 中搜索了 HTTP 503 返回码的错误描述。

如果您尝试curl https://127.0.0.1:8080/api，我假设您会收到 HTTP 503。

编辑：由于您使用 minikube，我们可以假设它运行正常。在这种情况下，更有可能存在配置问题。您的日志显示kubectl 尝试连接到localhost。 AFAIK 这是默认主机。当您启动 minikube 时，它​​不会在主机上运行，​​而是在虚拟机上运行。因此localhost 看起来不对，你应该看看 kube 配置。