Kubernetes Metric Server 无法调用 Kubelet API答案

【问题标题】：Kubernetes Metric Server cannot call Kubelet APIKubernetes Metric Server 无法调用 Kubelet API
【发布时间】：2022-08-19 07:09:14
【问题描述】：

在尝试在 Kubernetes 上安全地安装 metrics-server 时，我遇到了问题。

似乎 metric-server pod 无法在其 10250 端口上成功向 Kubelet API 发出请求。

NAME             READY   UP-TO-DATE   AVAILABLE   AGE
metrics-server   0/1     1            0           16h

Metrics Server 部署永远不会准备好，它会重复相同的错误日志序列：

I0522 01:27:41.472946       1 serving.go:342] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I0522 01:27:41.798068       1 configmap_cafile_content.go:201] \"Starting controller\" name=\"client-ca::kube-system::extension-apiserver-authentication::client-ca-file\"
I0522 01:27:41.798092       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0522 01:27:41.798068       1 dynamic_cafile_content.go:156] \"Starting controller\" name=\"request-header::/front-ca/front-proxy-ca.crt\"
I0522 01:27:41.798107       1 dynamic_serving_content.go:131] \"Starting controller\" name=\"serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key\"
I0522 01:27:41.798240       1 secure_serving.go:266] Serving securely on [::]:4443
I0522 01:27:41.798265       1 tlsconfig.go:240] \"Starting DynamicServingCertificateController\"
W0522 01:27:41.798284       1 shared_informer.go:372] The sharedIndexInformer has started, run more than once is not allowed
I0522 01:27:41.898439       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
E0522 01:27:55.297497       1 scraper.go:140] \"Failed to scrape node\" err=\"Get \\\"https://192.168.1.106:10250/metrics/resource\\\": context deadline exceeded\" node=\"system76-pc\"
E0522 01:28:10.297872       1 scraper.go:140] \"Failed to scrape node\" err=\"Get \\\"https://192.168.1.106:10250/metrics/resource\\\": context deadline exceeded\" node=\"system76-pc\"
I0522 01:28:10.325613       1 server.go:187] \"Failed probe\" probe=\"metric-storage-ready\" err=\"no metrics to serve\"
I0522 01:28:20.325231       1 server.go:187] \"Failed probe\" probe=\"metric-storage-ready\" err=\"no metrics to serve\"
E0522 01:28:25.297750       1 scraper.go:140] \"Failed to scrape node\" err=\"Get \\\"https://192.168.1.106:10250/metrics/resource\\\": context deadline exceeded\" node=\"system76-pc\"

我正在运行使用 kubeadm 版本 1.23.4 部署的 Kubernetes，并且我正在尝试安全地使用 metrics-server。

我正在寻找可以帮助的建议：

如何准确诊断问题？
或者，首先检查哪种配置似乎最有成效？

任何有助于我的心智模型的东西，我需要明确配置哪些证书和密钥以及自动处理的内容。

到目前为止，我已经尝试验证我可以检索 API 指标：

kubectl get --raw /api/v1/nodes/system76-pc/proxy/stats/summary

{
  \"node\": {
    \"nodeName\": \"system76-pc\",
    \"systemContainers\": [
      {
        \"name\": \"kubelet\",
        \"startTime\": \"2022-05-20T01:51:28Z\",
        \"cpu\": {
          \"time\": \"2022-05-22T00:48:40Z\",
          \"usageNanoCores\": 59453039,
          \"usageCoreNanoSeconds\": 9768130002000
        },
        \"memory\": {
          \"time\": \"2022-05-22T00:48:40Z\",
          \"usageBytes\": 84910080,
          \"workingSetBytes\": 84434944,
          \"rssBytes\": 67149824,
          \"pageFaults\": 893055,
          \"majorPageFaults\": 290
        }
      },
      {
        \"name\": \"runtime\",
        \"startTime\": \"2022-05-20T00:33:24Z\",
        \"cpu\": {
          \"time\": \"2022-05-22T00:48:37Z\",
          \"usageNanoCores\": 24731571,
          \"usageCoreNanoSeconds\": 3955659226000
        },
        \"memory\": {
          \"time\": \"2022-05-22T00:48:37Z\",
          \"usageBytes\": 484306944,
          \"workingSetBytes\": 242638848,
          \"rssBytes\": 84647936,
          \"pageFaults\": 56994074,
          \"majorPageFaults\": 428
        }
      },
      {
        \"name\": \"pods\",
        \"startTime\": \"2022-05-20T01:51:28Z\",
        \"cpu\": {
          \"time\": \"2022-05-22T00:48:37Z\",
          \"usageNanoCores\": 292818104,
          \"usageCoreNanoSeconds\": 45976001446000
        },
        \"memory\": {
          \"time\": \"2022-05-22T00:48:37Z\",
          \"availableBytes\": 29648396288,
          \"usageBytes\": 6108573696,

kubectl get --raw /api/v1/nodes/system76-pc/proxy/metrics/resource

# HELP container_cpu_usage_seconds_total [ALPHA] Cumulative cpu time consumed by the container in core-seconds
# TYPE container_cpu_usage_seconds_total counter
container_cpu_usage_seconds_total{container=\"alertmanager\",namespace=\"flux-system\",pod=\"alertmanager-prometheus-stack-kube-prom-alertmanager-0\"} 108.399948 1653182143362
container_cpu_usage_seconds_total{container=\"calico-kube-controllers\",namespace=\"kube-system\",pod=\"calico-kube-controllers-56fcbf9d6b-n87ts\"} 206.442768 1653182144294
container_cpu_usage_seconds_total{container=\"calico-node\",namespace=\"kube-system\",pod=\"calico-node-p6pxk\"} 6147.643669 1653182155672
container_cpu_usage_seconds_total{container=\"cert-manager\",namespace=\"cert-manager\",pod=\"cert-manager-795d7f859d-8jp4f\"} 134.583294 1653182142601
container_cpu_usage_seconds_total{container=\"cert-manager\",namespace=\"cert-manager\",pod=\"cert-manager-cainjector-5fcddc948c-vw4zz\"} 394.286782 1653182151252
container_cpu_usage_seconds_total{container=\"cert-manager\",namespace=\"cert-manager\",pod=\"cert-manager-webhook-5b64f87794-pl7fb\"} 404.53758 1653182140528
container_cpu_usage_seconds_total{container=\"config-reloader\",namespace=\"flux-system\",pod=\"alertmanager-prometheus-stack-kube-prom-alertmanager-0\"} 6.01391 1653182139771
container_cpu_usage_seconds_total{container=\"config-reloader\",namespace=\"flux-system\",pod=\"prometheus-prometheus-stack-kube-prom-prometheus-0\"} 42.706567 1653182130750
container_cpu_usage_seconds_total{container=\"controller\",namespace=\"flux-system\",pod=\"sealed-secrets-controller-5884bbf4d6-mql9x\"} 43.814816 1653182144648
container_cpu_usage_seconds_total{container=\"controller\",namespace=\"ingress-nginx\",pod=\"ingress-nginx-controller-f9d6fc8d8-sgwst\"} 645.109711 1653182141169
container_cpu_usage_seconds_total{container=\"coredns\",namespace=\"kube-system\",pod=\"coredns-64897985d-crtd9\"} 380.682251 1653182141861
container_cpu_usage_seconds_total{container=\"coredns\",namespace=\"kube-system\",pod=\"coredns-64897985d-rpmxk\"} 365.519839 1653182140533
container_cpu_usage_seconds_total{container=\"dashboard-metrics-scraper\",namespace=\"kubernetes-dashboard\",pod=\"dashboard-metrics-scraper-577dc49767-cbq8r\"} 25.733362 1653182141877
container_cpu_usage_seconds_total{container=\"etcd\",namespace=\"kube-system\",pod=\"etcd-system76-pc\"} 4237.357682 1653182140459
container_cpu_usage_seconds_total{container=\"grafana\",namespace=\"flux-system\",pod=\"prometheus-stack-grafana-757f9b9fcc-9f58g\"} 345.034245 1653182154951
container_cpu_usage_seconds_total{container=\"grafana-sc-dashboard\",namespace=\"flux-system\",pod=\"prometheus-stack-grafana-757f9b9fcc-9f58g\"} 123.480584 1653182146757
container_cpu_usage_seconds_total{container=\"grafana-sc-datasources\",namespace=\"flux-system\",pod=\"prometheus-stack-grafana-757f9b9fcc-9f58g\"} 35.851112 1653182145702
container_cpu_usage_seconds_total{container=\"kube-apiserver\",namespace=\"kube-system\",pod=\"kube-apiserver-system76-pc\"} 14166.156638 1653182150749
container_cpu_usage_seconds_total{container=\"kube-controller-manager\",namespace=\"kube-system\",pod=\"kube-controller-manager-system76-pc\"} 4168.427981 1653182148868
container_cpu_usage_seconds_total{container=\"kube-prometheus-stack\",namespace=\"flux-system\",pod=\"prometheus-stack-kube-prom-operator-54d9f985c8-ml2qj\"} 28.79018 1653182155583
container_cpu_usage_seconds_total{container=\"kube-proxy\",namespace=\"kube-system\",pod=\"kube-proxy-gg2wd\"} 67.215459 1653182155156
container_cpu_usage_seconds_total{container=\"kube-scheduler\",namespace=\"kube-system\",pod=\"kube-scheduler-system76-pc\"} 579.321492 1653182147910
container_cpu_usage_seconds_total{container=\"kube-state-metrics\",namespace=\"flux-system\",pod=\"prometheus-stack-kube-state-metrics-56d4759d67-h6lfv\"} 158.343644 1653182153691
container_cpu_usage_seconds_total{container=\"kubernetes-dashboard\",namespace=\"kubernetes-dashboard\",pod=\"kubernetes-dashboard-69dc48777b-8cckh\"} 78.231809 1653182139263
container_cpu_usage_seconds_total{container=\"manager\",namespace=\"flux-system\",pod=\"helm-controller-dfb4b5478-7zgt6\"} 338.974637 1653182143679
container_cpu_usage_seconds_total{container=\"manager\",namespace=\"flux-system\",pod=\"image-automation-controller-77fd9657c6-lg44h\"} 280.841645 1653182154912
container_cpu_usage_seconds_total{container=\"manager\",namespace=\"flux-system\",pod=\"image-reflector-controller-86db8b6f78-5rz58\"} 2909.277578 1653182144081
container_cpu_usage_seconds_total{container=\"manager\",namespace=\"flux-system\",pod=\"kustomize-controller-cd544c8f8-hxvk6\"} 596.392781 1653182152714
container_cpu_usage_seconds_total{container=\"manager\",namespace=\"flux-system\",pod=\"notification-controller-d9cc9bf46-2jhbq\"} 244.387967 1653182142902
container_cpu_usage_seconds_total{container=\"manager\",namespace=\"flux-system\",pod=\"source-controller-84bfd77bf8-r827h\"} 541.650877 1653182148963
container_cpu_usage_seconds_total{container=\"metrics-server\",namespace=\"flux-system\",pod=\"metrics-server-55bc5f774-zznpb\"} 174.229886 1653182146946
container_cpu_usage_seconds_total{container=\"nfs-subdir-external-provisioner\",namespace=\"flux-system\",pod=\"nfs-subdir-external-provisioner-858745f657-zcr66\"} 244.061329 1653182139840
container_cpu_usage_seconds_total{container=\"node-exporter\",namespace=\"flux-system\",pod=\"prometheus-stack-prometheus-node-exporter-wj2fx\"} 29.852036 1653182148779
container_cpu_usage_seconds_total{container=\"prometheus\",namespace=\"flux-system\",pod=\"prometheus-prometheus-stack-kube-prom-prometheus-0\"} 7141.611234 1653182154042
# HELP container_memory_working_set_bytes [ALPHA] Current working set of the container in bytes
# TYPE container_memory_working_set_bytes gauge
container_memory_working_set_bytes{container=\"alertmanager\",namespace=\"flux-system\",pod=\"alertmanager-prometheus-stack-kube-prom-alertmanager-0\"} 2.152448e+07 1653182143362

度量服务器配置：

 spec:
      containers:
      - args:
        - --secure-port=4443
        - --cert-dir=/tmp
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        - --kubelet-preferred-address-types=Hostname
        - --requestheader-client-ca-file=/front-ca/front-proxy-ca.crt
        - --kubelet-certificate-authority=/ca/ca.crt
        image: k8s.gcr.io/metrics-server/metrics-server:v0.6.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /livez
            port: https
            scheme: HTTPS
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: metrics-server
        ports:
        - containerPort: 4443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: https
            scheme: HTTPS
          initialDelaySeconds: 20
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp
          name: tmp
        - mountPath: /front-ca
          name: front-proxy-ca-dir
        - mountPath: /ca
          name: ca-dir
      dnsPolicy: ClusterFirst
      priorityClassName: system-cluster-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: metrics-server
      serviceAccountName: metrics-server
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: tmp
      - configMap:
          defaultMode: 420
          name: front-proxy-ca
        name: front-proxy-ca-dir
      - configMap:
          defaultMode: 420
          name: kubelet-ca
        name: ca-dir

kube-apiserver 配置：

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.1.106:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.1.106
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: k8s.gcr.io/kube-apiserver:v1.23.4

请注意，10250 是 kubelet 端口。因此，指标服务器似乎很难从那里抓取指标。
检查防火墙的 4443 和 10250。
在你的 kube-apiserver 配置上设置一些 -advertise-address=192.168.1.106，我想知道：这是一个负载均衡器，还是你的实际控制平面地址？如果没有：您是否尝试过在您的指标服务器 pod 上设置 hostNetwork=true / SDN 配置中是否有可能阻止这些连接（网络策略？SDN 本身的问题？...）
为更详细的日志记录添加 metrics-server 参数有点帮助-v=9，请求工作但由于权限而无法获取指标信息。谷歌搜索一篇文章，提示 metrics-server 是一个 API 扩展，“apiserver 必须能够验证 apiserver 请求连接时度量服务器提供的证书”。尝试安装相同机密的建议：` - --tls-cert-file=/crt/apiserver.crt - --tls-private-key-file=/key/apiserver.key` 有效。 brightbox.com/blog/2020/09/15/secure-kubernetes-metrics
我应该结束这个问题还是你认为这对其他人有用？

标签： kubernetes pki

【解决方案1】：

在我的情况下，由于主节点上只有 1 个 ocpu，我在度量服务器上遇到了同样的问题。至少使用 2 个。

【讨论】：