无法在 Kubernetes 的 Grafana 仪表板中获取 Prometheus 数据

Question

我在 Kubernetes 中创建了一个 prometheus 和 grafana 设置，如下所述：https://github.com/ContainerSolutions/k8s-deployment-strategies。

安装普罗米修斯：

helm install \
    --namespace=monitoring \
    --name=prometheus \
    --set server.persistentVolume.enabled=false \
    --set alertmanager.persistentVolume.enabled=false \
    stable/prometheus

设置 grafana：

helm install \
   --namespace=monitoring \
   --name=grafana \
   --version=1.12.0 \
   --set=adminUser=admin \
   --set=adminPassword=admin \
   --set=service.type=NodePort \
   stable/grafana


kubectl get pods -n monitoring
NAME                                             READY   STATUS    RESTARTS   AGE
grafana-6d4f6ff6d5-vw8r2                         1/1     Running   0          17h
prometheus-alertmanager-6cb6bc6b7-76fs4          2/2     Running   0          17h
prometheus-kube-state-metrics-5ff476d674-c7mpt   1/1     Running   0          17h
prometheus-node-exporter-4zhmk                   1/1     Running   0          17h
prometheus-node-exporter-g7jqm                   1/1     Running   0          17h
prometheus-node-exporter-sdnwg                   1/1     Running   0          17h
prometheus-pushgateway-7967b4cf45-j24hx          1/1     Running   0          17h
prometheus-server-5dfc4f657d-sl7kv               2/2     Running   0          17h

我可以从 grafana 容器内 curl 到我的 prometheus 容器 http://prometheus-server（它回复“找到”）。

Grafana 配置：

Name: prometheus
Type: Prometheus
http://prometheus-server

我在名为 Prometheus 2.0 stats 的默认仪表板中看到指标。 我创建了一个自己的仪表板（也在 github 链接中描述）。

sum(rate(http_requests_total{app="my-app"}[5m])) by (version)

我已经部署了我的应用程序，它正在运行，我经常卷曲它，但在我的仪表板中什么也看不到。

kubectl get pods
NAME                                          READY   STATUS    RESTARTS   AGE
my-app-7bd4b55cbd-8zm8b                       1/1     Running   0          17h
my-app-7bd4b55cbd-nzs2p                       1/1     Running   0          17h
my-app-7bd4b55cbd-zts78                       1/1     Running   0          17h

卷曲

while sleep 0.1; do curl http://192.168.50.10:30513/; done
Host: my-app-7bd4b55cbd-8zm8b, Version: v2.0.0
Host: my-app-7bd4b55cbd-zts78, Version: v2.0.0
Host: my-app-7bd4b55cbd-nzs2p, Version: v2.0.0
Host: my-app-7bd4b55cbd-8zm8b, Version: v2.0.0
Host: my-app-7bd4b55cbd-zts78, Version: v2.0.0
Host: my-app-7bd4b55cbd-zts78, Version: v2.0.0
Host: my-app-7bd4b55cbd-8zm8b, Version: v2.0.0
Host: my-app-7bd4b55cbd-8zm8b, Version: v2.0.0

我该如何调试或者我做错了什么？

更新： 我的应用部署配置

kubectl describe deployment my-app
Name:               my-app
Namespace:          default
CreationTimestamp:  Tue, 02 Apr 2019 22:17:31 +0200
Labels:             app=my-app
Annotations:        deployment.kubernetes.io/revision: 2
                    kubectl.kubernetes.io/last-applied-configuration:
                      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"my-app"},"name":"my-app","namespace":"default"},...
Selector:           app=my-app
Replicas:           3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType:       Recreate
MinReadySeconds:    0
Pod Template:
  Labels:       app=my-app
                version=v2.0.0
  Annotations:  prometheus.io/port: 9101
                prometheus.io/scrape: true
  Containers:
   my-app:
    Image:       containersol/k8s-deployment-strategies
    Ports:       8080/TCP, 8086/TCP
    Host Ports:  0/TCP, 0/TCP
    Liveness:    http-get http://:probe/live delay=5s timeout=1s period=5s #success=1 #failure=3
    Readiness:   http-get http://:probe/ready delay=0s timeout=1s period=5s #success=1 #failure=3
    Environment:
      VERSION:  v2.0.0
    Mounts:     <none>
  Volumes:      <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   my-app-7bd4b55cbd (3/3 replicas created)
Events:          <none>

yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  replicas: 3
  strategy:
    type: Recreate

  # The selector field tell the deployment which pod to update with
  # the new version. This field is optional, but if you have labels
  # uniquely defined for the pod, in this case the "version" label,
  # then we need to redefine the matchLabels and eliminate the version
  # field from there.
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
        version: v2.0.0
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9101"
    spec:
      containers:
      - name: my-app
        image: containersol/k8s-deployment-strategies
        ports:
        - name: http
          containerPort: 8080
        - name: probe
          containerPort: 8086
        env:
        - name: VERSION
          value: v2.0.0
        livenessProbe:
          httpGet:
            path: /live
            port: probe
          initialDelaySeconds: 5
          periodSeconds: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: probe
          periodSeconds: 5

Answer 1

在您的部署中，您说要抓取端口 9101，但您没有在容器上发布此端口。

您的 prometheus 端点在端口 9101 还是 8080/8086 上？