kube-apiserver 启动时无法获取或设置密钥

Question

我试图用 packer 和 terraform 代替 kube-up.sh 脚本设置一个高可用的 kubernetes 集群。原因：我想要更大的机器，不同的设置等。我的大部分配置来自 coreos kubernetes 部署教程。

关于我的设置：

CoreOS

一切都在 gce 上运行。 我有 3 个 etcd 和一个 skydns 实例。他们正在工作并且能够互相联系。

我有一个实例作为 kubernetes 主实例，它正在运行带有清单的 kubelet。

我现在的实际问题是 kube-api 服务器无法自行连接。我可以从主机系统运行 curl 命令并获得有效响应。 /version 等。

443 和 8080 没有从 docker 转发也有点奇怪。还是这是正常现象？

我以为我错误配置了一些主端点。所以我尝试了 localhost 和所有清单的外部 ip。 => 不工作。

kube-api 容器中的错误：

I0925 14:51:47.505859       1 plugins.go:69] No cloud provider specified.
I0925 14:51:47.973450       1 master.go:273] Node port range unspecified. Defaulting to 30000-32767.
E0925 14:51:48.009367       1 reflector.go:136] Failed to list *api.ResourceQuota: Get http://127.0.0.1:8080/api/v1/resourcequotas: dial tcp 127.0.0.1:8080: connection refused
E0925 14:51:48.010730       1 reflector.go:136] Failed to list *api.Secret: Get http://127.0.0.1:8080/api/v1/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token: dial tcp 127.0.0.1:8080: connection refused
E0925 14:51:48.010996       1 reflector.go:136] Failed to list *api.ServiceAccount: Get http://127.0.0.1:8080/api/v1/serviceaccounts: dial tcp 127.0.0.1:8080: connection refused
E0925 14:51:48.011083       1 reflector.go:136] Failed to list *api.LimitRange: Get http://127.0.0.1:8080/api/v1/limitranges: dial tcp 127.0.0.1:8080: connection refused
E0925 14:51:48.012697       1 reflector.go:136] Failed to list *api.Namespace: Get http://127.0.0.1:8080/api/v1/namespaces: dial tcp 127.0.0.1:8080: connection refused
E0925 14:51:48.012753       1 reflector.go:136] Failed to list *api.Namespace: Get http://127.0.0.1:8080/api/v1/namespaces: dial tcp 127.0.0.1:8080: connection refused
[restful] 2015/09/25 14:51:48 log.go:30: [restful/swagger] listing is available at https://104.155.60.74:443/swaggerapi/
[restful] 2015/09/25 14:51:48 log.go:30: [restful/swagger] https://104.155.60.74:443/swaggerui/ is mapped to folder /swagger-ui/
I0925 14:51:48.136166       1 server.go:441] Serving securely on 0.0.0.0:443
I0925 14:51:48.136248       1 server.go:483] Serving insecurely on 127.0.0.1:8080

控制器容器有几乎相同的错误。每个其他容器都很好。

我的配置：

/etc/kubelet.env

KUBE_KUBELET_OPTS="\
  --api_servers=http://127.0.0.1:8080 \
  --register-node=false \
  --allow-privileged=true \
  --config=/etc/kubernetes/manifests \
  --tls_cert_file=/etc/kubernetes/ssl/apiserver.pem \
  --tls_private_key_file=/etc/kubernetes/ssl/apiserver-key.pem \
  --cloud-provider=gce \
  --cluster_dns=10.10.38.10 \
  --cluster_domain=cluster.local \
  --cadvisor-port=0"

/etc/kubernetes/manifests/

apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
  - name: kube-apiserver
    image: gcr.io/google_containers/hyperkube:v1.0.6
    command:
    - /hyperkube
    - apiserver
    - --bind-address=0.0.0.0
    - --etcd_servers=http://10.10.125.10:2379,http://10.10.82.201:2379,http://10.10.63.185:2379
    - --allow-privileged=true
    - --service-cluster-ip-range=10.40.0.0/16
    - --secure_port=443
    - --advertise-address=104.155.60.74
    - --admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota
    - --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem
    - --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
    - --client-ca-file=/etc/kubernetes/ssl/ca.pem
    - --service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem
    ports:
    - containerPort: 443
      hostPort: 443
      name: https
    - containerPort: 8080
      hostPort: 8080
      name: local
    volumeMounts:
    - mountPath: /etc/kubernetes/ssl
      name: ssl-certs-kubernetes
      readOnly: true
    - mountPath: /etc/ssl/certs
      name: ssl-certs-host
      readOnly: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/ssl
    name: ssl-certs-kubernetes
  - hostPath:
      path: /usr/share/ca-certificates
    name: ssl-certs-host

/etc/kubernetes/manifests/kube-controller-manager.yml

apiVersion: v1
kind: Pod
metadata:
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - name: kube-controller-manager
    image: gcr.io/google_containers/hyperkube:v1.0.6
    command:
    - /hyperkube
    - controller-manager
    - --master=https://104.155.60.74:443
    - --service-account-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
    - --root-ca-file=/etc/kubernetes/ssl/ca.pem
    - --cloud_provider=gce
    livenessProbe:
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10252
      initialDelaySeconds: 15
      timeoutSeconds: 1
    volumeMounts:
    - mountPath: /etc/kubernetes/ssl
      name: ssl-certs-kubernetes
      readOnly: true
    - mountPath: /etc/ssl/certs
      name: ssl-certs-host
      readOnly: true
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/ssl
    name: ssl-certs-kubernetes
  - hostPath:
      path: /usr/share/ca-certificates
    name: ssl-certs-host

码头工人ps

CONTAINER ID        IMAGE                                       COMMAND                CREATED             STATUS              PORTS               NAMES
3e37b2ea2277        gcr.io/google_containers/hyperkube:v1.0.6   "/hyperkube controll   31 minutes ago      Up 31 minutes                           k8s_kube-controller-manager.afecd3c9_kube-controller-manager-kubernetes-km0.c.stylelounge-1042.inte
rnal_kube-system_621db46bf7b0764eaa46d17dfba8e90f_519cd0da   
43917185d91b        gcr.io/google_containers/hyperkube:v1.0.6   "/hyperkube proxy --   31 minutes ago      Up 31 minutes                           k8s_kube-proxy.a2db3197_kube-proxy-kubernetes-km0.c.stylelounge-1042.internal_kube-system_67c22e99a
eb1ef9c2997c942cfbe48b9_c82a8a60                             
f548279e90f9        gcr.io/google_containers/hyperkube:v1.0.6   "/hyperkube apiserve   31 minutes ago      Up 31 minutes                           k8s_kube-apiserver.2bcb2c35_kube-apiserver-kubernetes-km0.c.stylelounge-1042.internal_kube-system_8
67c500deb54965609810fd0771fa92d_a306feae                     
94b1942a09f0        gcr.io/google_containers/hyperkube:v1.0.6   "/hyperkube schedule   31 minutes ago      Up 31 minutes                           k8s_kube-scheduler.603b59f4_kube-scheduler-kubernetes-km0.c.stylelounge-1042.internal_kube-system_3
9e2c582fd067b44ebe8cefaee036c0e_e0ddf6a2                     
9de4a4264ef6        gcr.io/google_containers/podmaster:1.1      "/podmaster --etcd-s   31 minutes ago      Up 31 minutes                           k8s_controller-manager-elector.89f472b4_kube-podmaster-kubernetes-km0.c.stylelounge-1042.internal_k
ube-system_e23fc0902c7e6da7b315ad34130b9807_7c8d2901         
af2df45f4081        gcr.io/google_containers/podmaster:1.1      "/podmaster --etcd-s   31 minutes ago      Up 31 minutes                           k8s_scheduler-elector.608b6780_kube-podmaster-kubernetes-km0.c.stylelounge-1042.internal_kube-syste
m_e23fc0902c7e6da7b315ad34130b9807_b11e601d                  
ac0e068456c7        gcr.io/google_containers/pause:0.8.0        "/pause"               31 minutes ago      Up 31 minutes                           k8s_POD.e4cc795_kube-controller-manager-kubernetes-km0.c.stylelounge-1042.internal_kube-system_621d
b46bf7b0764eaa46d17dfba8e90f_e9760e28                        
2773ba48d011        gcr.io/google_containers/pause:0.8.0        "/pause"               31 minutes ago      Up 31 minutes                           k8s_POD.e4cc795_kube-podmaster-kubernetes-km0.c.stylelounge-1042.internal_kube-system_e23fc0902c7e6
da7b315ad34130b9807_4fba9edb                                 
987531f1951d        gcr.io/google_containers/pause:0.8.0        "/pause"               31 minutes ago      Up 31 minutes                           k8s_POD.e4cc795_kube-apiserver-kubernetes-km0.c.stylelounge-1042.internal_kube-system_867c500deb549
65609810fd0771fa92d_d15d2d66                                 
f4453b948186        gcr.io/google_containers/pause:0.8.0        "/pause"               31 minutes ago      Up 31 minutes                           k8s_POD.e4cc795_kube-proxy-kubernetes-km0.c.stylelounge-1042.internal_kube-system_67c22e99aeb1ef9c2
997c942cfbe48b9_07e540c8                                     
ce01cfda007e        gcr.io/google_containers/pause:0.8.0        "/pause"               31 minutes ago      Up 31 minutes                           k8s_POD.e4cc795_kube-scheduler-kubernetes-km0.c.stylelounge-1042.internal_kube-system_39e2c582fd067
b44ebe8cefaee036c0e_e6cb6500          

这里是 curl 命令：

kubernetes-km0 ~ # docker logs a404a310b55e
I0928 09:14:05.019135       1 plugins.go:69] No cloud provider specified.
I0928 09:14:05.192451       1 master.go:273] Node port range unspecified. Defaulting to 30000-32767.
I0928 09:14:05.192900       1 master.go:295] Will report 10.10.247.127 as public IP address.
E0928 09:14:05.226222       1 reflector.go:136] Failed to list *api.LimitRange: Get http://127.0.0.1:8080/api/v1/limitranges: dial tcp 127.0.0.1:8080: connection refused
E0928 09:14:05.226428       1 reflector.go:136] Failed to list *api.Namespace: Get http://127.0.0.1:8080/api/v1/namespaces: dial tcp 127.0.0.1:8080: connection refused
E0928 09:14:05.226479       1 reflector.go:136] Failed to list *api.Namespace: Get http://127.0.0.1:8080/api/v1/namespaces: dial tcp 127.0.0.1:8080: connection refused
E0928 09:14:05.226593       1 reflector.go:136] Failed to list *api.Secret: Get http://127.0.0.1:8080/api/v1/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token: dial tcp 127.0.0.1:8080: connection refused
E0928 09:14:05.226908       1 reflector.go:136] Failed to list *api.ServiceAccount: Get http://127.0.0.1:8080/api/v1/serviceaccounts: dial tcp 127.0.0.1:8080: connection refused
[restful] 2015/09/28 09:14:05 log.go:30: [restful/swagger] listing is available at https://10.10.247.127:443/swaggerapi/
[restful] 2015/09/28 09:14:05 log.go:30: [restful/swagger] https://10.10.247.127:443/swaggerui/ is mapped to folder /swagger-ui/
E0928 09:14:05.232632       1 reflector.go:136] Failed to list *api.ResourceQuota: Get http://127.0.0.1:8080/api/v1/resourcequotas: dial tcp 127.0.0.1:8080: connection refused
I0928 09:14:05.368697       1 server.go:441] Serving securely on 0.0.0.0:443
I0928 09:14:05.368788       1 server.go:483] Serving insecurely on 127.0.0.1:8080
kubernetes-km0 ~ # curl http://127.0.0.1:8080/api/v1/limitranges
{
  "kind": "LimitRangeList",
  "apiVersion": "v1",
  "metadata": {
    "selfLink": "/api/v1/limitranges",
    "resourceVersion": "100"
  },
  "items": []
}

---

Answer 1

如果您希望 master 实际托管任何带有 --register-node=true 标志的 Pod 到在 master 上运行的 kubelet，您需要将 master 注册为节点。 CoreOs 教程没有将 master 注册为节点，因为那是理想的场景。

Answer 2

我相信你需要指定 --insecure-address=127.0.0.1 和 --insecure-port=8080 才能在 HTTP 上打开，默认是 https。