【问题标题】:Kubernetes metrics-server FailedDiscoveryCheckKubernetes 指标服务器 FailedDiscoveryCheck
【发布时间】:2020-06-17 06:11:33
【问题描述】:

希望能得到一点帮助,我的 Google-Fu 并没有让我更接近。我正在尝试为我的 fedora-coreos kubernetes 4 节点集群安装指标服务器,如下所示:

kubectl apply -f deploy/kubernetes/
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
serviceaccount/metrics-server created
deployment.apps/metrics-server created
service/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created

服务似乎永远不会启动

kubectl describe apiservice v1beta1.metrics.k8s.io
Name:         v1beta1.metrics.k8s.io
Namespace:
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"apiregistration.k8s.io/v1beta1","kind":"APIService","metadata":{"annotations":{},"name":"v1beta1.metrics.k8s.io"},"spec":{"...
API Version:  apiregistration.k8s.io/v1
Kind:         APIService
Metadata:
  Creation Timestamp:  2020-03-04T16:53:33Z
  Resource Version:    1611816
  Self Link:           /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
  UID:                 65d9a56a-c548-4d7e-a647-8ce7a865a266
Spec:
  Group:                     metrics.k8s.io
  Group Priority Minimum:    100
  Insecure Skip TLS Verify:  true
  Service:
    Name:            metrics-server
    Namespace:       kube-system
    Port:            443
  Version:           v1beta1
  Version Priority:  100
Status:
  Conditions:
    Last Transition Time:  2020-03-04T16:53:33Z
    Message:               failing or missing response from https://10.3.230.59:443/apis/metrics.k8s.io/v1beta1: bad status from https://10.3.230.59:443/apis/metrics.k8s.io/v1beta1: 403
    Reason:                FailedDiscoveryCheck
    Status:                False
    Type:                  Available
Events:                    <none>

我在谷歌上搜索了一下:

kubectl get deploy,svc -n kube-system |egrep metrics-server
deployment.apps/metrics-server   1/1     1            1           8m7s
service/metrics-server   ClusterIP   10.3.230.59   <none>        443/TCP         8m7s

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"
Error from server (ServiceUnavailable): the server is currently unable to handle the request

kubectl get all --all-namespaces | grep -i metrics-server
kube-system      pod/metrics-server-75b5d446cd-zj4jm                              1/1     Running   0          9m11s
kube-system   service/metrics-server   ClusterIP      10.3.230.59    <none>        443/TCP                                     9m11s
kube-system      deployment.apps/metrics-server   1/1     1            1           9m11s
kube-system      replicaset.apps/metrics-server-75b5d446cd   1         1         1       9m11s

kubectl logs -f metrics-server-75b5d446cd-zj4jm -n kube-system
I0304 16:53:36.475657       1 serving.go:312] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
W0304 16:53:38.229267       1 authentication.go:296] Cluster doesn't provide requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
I0304 16:53:38.267760       1 secure_serving.go:116] Serving securely on [::]:4443

kubectl get -n kube-system deployment metrics-server -o yaml | grep -i args -A 10
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"k8s-app":"metrics-server"},"name":"metrics-server","namespace":"kube-system"},"spec":{"selector":{"matchLabels":{"k8s-app":"metrics-server"}},"template":{"metadata":{"labels":{"k8s-app":"metrics-server"},"name":"metrics-server"},"spec":{"containers":[{"args":["--cert-dir=/tmp","--secure-port=4443","--kubelet-insecure-tls","--kubelet-preferred-address-types=InternalIP"],"image":"k8s.gcr.io/metrics-server-amd64:v0.3.6","imagePullPolicy":"IfNotPresent","name":"metrics-server","ports":[{"containerPort":4443,"name":"main-port","protocol":"TCP"}],"securityContext":{"readOnlyRootFilesystem":true,"runAsNonRoot":true,"runAsUser":1000},"volumeMounts":[{"mountPath":"/tmp","name":"tmp-dir"}]}],"nodeSelector":{"beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64"},"serviceAccountName":"metrics-server","volumes":[{"emptyDir":{},"name":"tmp-dir"}]}}}}
  creationTimestamp: "2020-03-04T16:53:33Z"
  generation: 1
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
  resourceVersion: "1611810"
  selfLink: /apis/apps/v1/namespaces/kube-system/deployments/metrics-server
  uid: 006e758e-bd33-47d7-8378-d3a8081ee8a8
spec:
--
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP
        image: k8s.gcr.io/metrics-server-amd64:v0.3.6
        imagePullPolicy: IfNotPresent
        name: metrics-server
        ports:
        - containerPort: 4443
          name: main-port

最后是我的部署配置:

 spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.6
        command:
          - /metrics-server
          - --kubelet-insecure-tls
          - --kubelet-preferred-address-types=InternalIP
        args:
          - --cert-dir=/tmp
          - --secure-port=4443
          - --kubelet-insecure-tls
          - --kubelet-preferred-address-types=InternalIP
        ports:
        - name: main-port
          containerPort: 4443
          protocol: TCP
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp
      hostNetwork: true
      nodeSelector:
        beta.kubernetes.io/os: linux
        kubernetes.io/arch: "amd64"

我不知道它可以让指标服务启动什么,只是让基本的 kubectl 顶级节点显示我得到的任何信息

Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)

我已经搜索了互联网并尝试添加 args: 和 command: 行但没有运气

command:
           - /metrics-server
           - --kubelet-insecure-tls
           - --kubelet-preferred-address-types=InternalIP
args:
          - --cert-dir=/tmp
          - --secure-port=4443
          - --kubelet-insecure-tls
          - --kubelet-preferred-address-types=InternalIP

谁能解释一下如何解决这个问题?谢谢

Pastebin 日志文件 Log File

【问题讨论】:

  • 我猜你正在使用kubeadm。你能提供 kubeadm、kubectl、docker 等的版本吗?还有你用的是什么CNI?我已经在这里遇到过类似的问题:stackoverflow.com/questions/60101398/… 您能否将hostNetwork: true 添加到您的部署中?你试过用weave netCNI吗?
  • 系统信息:操作系统镜像:Fedora CoreOS 31.20200210.3.0 操作系统:linux 架构:amd64 容器运行时版本:docker://18.9.8 Kubelet 版本:v1.17.3 Kube-Proxy 版本:v1。 17.3
  • CNI 呢?您是否尝试使用 hostNetwork: true 运行 metrics-server 部署?
  • 我还从我的部署文件中取消了 hostNetwork: true 的注释并且没有变化。
  • 您是否也取消了- --kubelet-preferred-address-types=InternalIP- --kubelet-insecure-tls 的注释?这 2 个标志和 hostNetwork: true 不能评论

标签: kubernetes metrics-server


【解决方案1】:

我已经复制了您的问题。我使用Calico 作为CNI

$ kubectl get nodes
NAME              STATUS   ROLES    AGE     VERSION
fedora-master     Ready    master   6m27s   v1.17.3
fedora-worker-1   Ready    <none>   4m48s   v1.17.3
fedora-worker-2   Ready    <none>   4m46s   v1.17.3

fedora-master:~/metrics-server$ kubectl describe apiservice v1beta1.metrics.k8s.io
Status:
  Conditions:
    Last Transition Time:  2020-03-12T16:04:59Z
    Message:               failing or missing response from https://10.99.122.196:443/apis/metrics.k8s.io/v
1beta1: Get https://10.99.122.196:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled while waiting
 for connection (Client.Timeout exceeded while awaiting headers)

fedora-master:~/metrics-server$ kubectl top pod
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)

当集群中只有一个节点时,metrics-server repo 中的默认设置可以正常工作。当您有超过 2 个节点时会出现问题。我使用了 1 个 master 和 2 个 worker 来复制。下面的示例部署正确(具有所有必需的参数)。之前,请删除您当前的metrics-server YAML (kubectl delete -f deploy/kubernetes) 并执行:

$ git clone https://github.com/kubernetes-sigs/metrics-server
$ cd metrics-server/deploy/kubernetes/
$ vi metrics-server-deployment.yaml

在 YAML 下方粘贴:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      hostNetwork: true
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.6
        imagePullPolicy: IfNotPresent
        args:
          - /metrics-server
          - --kubelet-preferred-address-types=InternalIP
          - --kubelet-insecure-tls
          - --cert-dir=/tmp
          - --secure-port=4443
        ports:
        - name: main-port
          containerPort: 4443
          protocol: TCP
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp
      nodeSelector:
        kubernetes.io/os: linux
        kubernetes.io/arch: "amd64"

使用:wq保存并退出

$ cd ~/metrics-server
$ kubectl apply -f deploy/kubernetes/
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
serviceaccount/metrics-server created
deployment.apps/metrics-server created
service/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created

等待一段时间,metrics-server 从节点收集一些指标。

$ kubectl describe apiservice v1beta1.metrics.k8s.io
Name:         v1beta1.metrics.k8s.io
Namespace:    
...
Metadata:
  Creation Timestamp:  2020-03-12T16:57:58Z
...
Spec:
  Group:                     metrics.k8s.io
  Group Priority Minimum:    100
  Insecure Skip TLS Verify:  true
  Service:
    Name:            metrics-server
    Namespace:       kube-system
    Port:            443
  Version:           v1beta1
  Version Priority:  100
Status:
  Conditions:
    Last Transition Time:  2020-03-12T16:58:01Z
    Message:               all checks passed
    Reason:                Passed
    Status:                True
    Type:                  Available
Events:                    <none>

几分钟后您可以使用top

$ kubectl top nodes
NAME              CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
fedora-master     188m         9%     1315Mi          17%       
fedora-worker-1   109m         5%     982Mi           13%       
fedora-worker-2   84m          4%     969Mi           13%   

如果您仍然会遇到一些问题,请将- --v=6 添加到部署并提供来自metrics-server pod 的日志。

containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.1
        args:
          - /metrics-server
          - --v=6
          - --kubelet-preferred-address-types=InternalIP
          - --kubelet-insecure-tls

【讨论】:

  • 哇非常感谢详细的解释,但我仍然收到错误。将 pastebin 链接添加到原始帖子。谢谢
  • 您是否完全删除了早期的指标服务器资源?你用的是什么CNI?您尝试将Weave 用作 CNI 吗?
  • 是的,我已经完全删除了早期的 metric-server 资源。我使用 Calico 作为 CNI。我正在考虑炸毁这个 k8s 集群并使用 k3s 集群,因为我也遇到了让 rook ceph 工作的问题。
  • 我想我可能已经找到了问题所在,您在构建集群时是否启用了聚合器?由于我使用 typhoon 来构建集群,我注意到默认情况下它是禁用的。
  • 对于任何阅读本文并使用 posiden.io 构建 Kubernetes 集群的人,如果您查看 variables.tf,您必须启用聚合层,您会看到 arg。谢谢,@PjoterS 非常感谢详细的解释和帮助!
猜你喜欢
  • 2021-06-25
  • 1970-01-01
  • 2021-02-22
  • 2019-03-23
  • 2019-08-27
  • 1970-01-01
  • 1970-01-01
  • 2021-05-18
  • 1970-01-01
相关资源
最近更新 更多