【问题标题】:Prometheus on GCP Kubernetes cluster error "no such host"GCP Kubernetes 集群上的 Prometheus 错误“没有这样的主机”
【发布时间】:2021-12-31 06:52:54
【问题描述】:

我正在尝试在 GCP Kubernetes 集群上设置 Prometheus。我关注了this tutorial

我正在运行部署 pod。 现在我正在尝试通过端口转发访问 Prometheus 仪表板,如博客中所示。 但是页面无法访问。 我尝试调试并查看了部署pod的日志,发现那里有错误。

ts=2021-12-31T06:49:42.109Z caller=notifier.go:526 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on <IP>:53: no such host"

【问题讨论】:

  • 这用于警报管理器检查您是否首先警报管理器运行。
  • 怎么做?
  • 你能提供更多关于你的设置的细节吗?你已经尝试过什么?
  • 我在 GKE 上创建了一个 kubernetes 集群,并按照链接(博客)中的相同步骤进行操作
  • 所以 - 你被困在kubectl port-forward prometheus-monitoring-3331088907-hm5n1 8080:9090 -n monitoring(取自链接教程)命令?如果是这样,请提供您收到的确切命令和错误消息。

标签: kubernetes google-cloud-platform google-kubernetes-engine prometheus prometheus-alertmanager


【解决方案1】:

您遇到的错误可能是由于您的集群配置或常规设置造成的。

我已经逐步完成了您链接的教程并且一切正常 - 我可以连接到主页。我没有进一步说明,但我将描述以下所有步骤。

  1. 我使用 autopilot feature 创建了新的 GKE 集群。
  2. 然后我在Cloud Shell 中运行了闲置命令(但您可以使用安装了kubectl 的任何其他系统):
ACCOUNT=$(gcloud info --format='value(config.account)')
kubectl create clusterrolebinding owner-cluster-admin-binding \
    --clusterrole cluster-admin \
    --user $ACCOUNT
  1. 然后我克隆了 Git repo git clone https://github.com/bibinwilson/kubernetes-prometheus

  2. 创建命名空间:kubectl create namespace monitoring

  3. 创建了clusterRole.yaml 文件,内容如下:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: default
  namespace: monitoring
  1. 创建了一个角色kubectl create -f clusterRole.yaml

  2. 创建了一个包含闲置内容的文件congi-map.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-conf
  labels:
    name: prometheus-server-conf
  namespace: monitoring
data:
  prometheus.rules: |-
    groups:
    - name: devopscube demo alert
      rules:
      - alert: High Pod Memory
        expr: sum(container_memory_usage_bytes) > 1
        for: 1m
        labels:
          severity: slack
        annotations:
          summary: High Memory Usage
  prometheus.yml: |-
    global:
      scrape_interval: 5s
      evaluation_interval: 5s
    rule_files:
      - /etc/prometheus/prometheus.rules
    alerting:
      alertmanagers:
      - scheme: http
        static_configs:
        - targets:
          - "alertmanager.monitoring.svc:9093"

    scrape_configs:
      - job_name: 'node-exporter'
        kubernetes_sd_configs:
          - role: endpoints
        relabel_configs:
        - source_labels: [__meta_kubernetes_endpoints_name]
          regex: 'node-exporter'
          action: keep
      
      - job_name: 'kubernetes-apiservers'

        kubernetes_sd_configs:
        - role: endpoints
        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        relabel_configs:
        - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
          action: keep
          regex: default;kubernetes;https

      - job_name: 'kubernetes-nodes'

        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        kubernetes_sd_configs:
        - role: node

        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics     
      
      - job_name: 'kubernetes-pods'

        kubernetes_sd_configs:
        - role: pod

        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name
      
      - job_name: 'kube-state-metrics'
        static_configs:
          - targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']

      - job_name: 'kubernetes-cadvisor'

        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        kubernetes_sd_configs:
        - role: node

        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
      
      - job_name: 'kubernetes-service-endpoints'

        kubernetes_sd_configs:
        - role: endpoints

        relabel_configs:
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
          action: replace
          target_label: __scheme__
          regex: (https?)
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
          action: replace
          target_label: __address__
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_service_name]
          action: replace
          target_label: kubernetes_name
  1. 创建了一个配置映射kubectl create -f config-map.yaml
  2. 创建了prometheus-deployment.yaml 文件,内容如下:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-deployment
  namespace: monitoring
  labels:
    app: prometheus-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-server
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus
          args:
            - "--storage.tsdb.retention.time=12h"
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus/"
          ports:
            - containerPort: 9090
          resources:
            requests:
              cpu: 500m
              memory: 500M
            limits:
              cpu: 1
              memory: 1Gi
          volumeMounts:
            - name: prometheus-config-volume
              mountPath: /etc/prometheus/
            - name: prometheus-storage-volume
              mountPath: /prometheus/
      volumes:
        - name: prometheus-config-volume
          configMap:
            defaultMode: 420
            name: prometheus-server-conf
  
        - name: prometheus-storage-volume
          emptyDir: {}
  1. 已创建部署kubectl create -f prometheus-deployment.yaml
  2. 检查过:
wb@cloudshell:~$ kubectl get deployments --namespace=monitoring
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
prometheus-deployment   1/1     1            1           145m
  1. 获得 pod 名称:
wb@cloudshell:~$ kubectl get pods --namespace=monitoring
NAME                                     READY   STATUS    RESTARTS   AGE
prometheus-deployment-7f57368-bk4   1/1     Running   0          146m
  1. 转发端口kubectl port-forward prometheus-deployment-7f57368-bk4 8080:9090 -n monitoring

  2. 在第二个窗口中,我打开了另一个 shell 并运行:curl http://localhost:8080,结果是:

wb@cloudshell:~ (wb-project-1)$ curl http://localhost:8080
<a href="/graph">Found</a>.

这意味着该页面可以访问并且可以正常工作 - 您可以尝试继续按照我的步骤进行操作,如果出现问题,请提供您遇到问题的详细信息以及错误所在。

【讨论】:

    猜你喜欢
    • 1970-01-01
    • 2020-04-15
    • 2021-03-09
    • 1970-01-01
    • 2016-03-22
    • 1970-01-01
    • 2022-01-19
    • 1970-01-01
    • 2020-03-14
    相关资源
    最近更新 更多