使用普罗米修斯和警报管理器未在松弛中显示警报消息

Question

我正在尝试让 Prometheus 找到的 alert 使用 alertmanager 在 slack 中得到通知。

这是 alert.rules 文件，工作正常

groups:
- name: Instances
  rules:
  # Alert for any instance that is unreachable for >5 minutes.
  - alert: InstanceDown
    expr: up == 0
    for: 5m
    labels:
      severity: page
    # Prometheus templates apply here in the annotation and label fields of the alert.
    annotations:
      description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.'
      summary: 'Instance {{ $labels.instance }} down'

它正在成功显示一个实例。

但是我的 alertmanager.yml 中的问题是它没有将通知发送到 slack。我还成功设置了 slack webhook，甚至测试了钩子是否工作正常，同时使用 slack 提供的服务创建了一个钩子

alertmanager.yml

groups:
- name: Instances
  rules:
  # Alert for any instance that is unreachable for >5 minutes.
  - alert: InstanceDown
    expr: up == 0
    for: 5m
    labels:
      severity: page
    # Prometheus templates apply here in the annotation and label fields of the alert.
    annotations:
      description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.'
      summary: 'Instance {{ $labels.instance }} down'
[tgurung@ip131 prometheus_graphana_myversion]$ cat alertmanager/alertmanager.yml 


route: 
  receiver: 'slack-notifications'
  #group_by: [alertname, datacenter, app]

receivers:
- name: 'slack-notifications'
  slack_configs:
  - api_url: https://hooks.slack.com/services/T52GRFN3F/B93KTCUHH/JC
    channel: #general
    send_resolved: true

    # Alertmanager templates apply here.
    text: "<!channel> \nsummary: {{ .CommonAnnotations.summary }}\ndescription: {{ .CommonAnnotations.description }}"

在运行 docker-compose up 时，我得到了关注

prometheus_1     | level=error ts=2018-02-06T09:36:35.580565429Z caller=notifier.go:454 component=notifier alertmanager=http://x.x.x.x:9093/api/v1/alerts count=0 msg="Error sending alert" err="Post http://x.X.x.x:9093/api/v1/alerts: dial tcp x.x.x.x:9093: getsockopt: no route to host"

解决上述错误： 为了解决上述路由问题，我在全新的实例中运行了警报管理器，并且克服了该错误

转到错误消息中的 API 链接我可以看到这个

{"status":"success","data":[]}

这是 alert_manager 的，看起来效果很好。

alertmanager_1   | level=info ts=2018-02-06T09:36:37.66654544Z caller=main.go:141 msg="Starting Alertmanager" version="(version=0.13.0, branch=HEAD, revision=fb713f6d8239b57c646cae30f78e8b4b8861a1aa)"
alertmanager_1   | level=info ts=2018-02-06T09:36:37.66661402Z caller=main.go:142 build_context="(go=go1.9.2, user=root@d83981af1d3d, date=20180112-10:32:46)"
alertmanager_1   | level=info ts=2018-02-06T09:36:37.668103448Z caller=main.go:279 msg="Loading configuration file" file=/alertmanager/alertmanager.yml
alertmanager_1   | level=info ts=2018-02-06T09:36:37.673288146Z caller=main.go:354 msg=Listening address=:9093

这是 prometheus.yml 配置文件

global:
  scrape_interval: 5s
  external_labels:
    monitor: 'my-monitor'

#alerting rules file
rule_files:
  - '/alertmanager/alert.rules'


scrape_configs:
    - job_name: 'prometheus'
      static_configs: 
       - targets: ['localhost:9090']

    - job_name: 'node-exporter'
      static_configs:
        - targets: ['node-exporter:9100']


alerting:
  alertmanagers:
    - static_configs:
      - targets: ["54.36.X.X:9093"]  #this is the alertmanager service url 

这是我的 docker-compose.yml

version: '2'
volumes:
    grafana_data: {}

services:
    prometheus:
        image: prom/prometheus
        privileged: true
        volumes:
            - ./prometheus.yml:/etc/prometheus/prometheus.yml
            - ./alertmanager/alert.rules:/alertmanager/alert.rules
            - ./alertmanager/alertmanager.yml:/alertmanager/alertmanager.yml
        command:
            - '--config.file=/etc/prometheus/prometheus.yml'
        ports:
            - '9090:9090'
        links:
            - "alertmanager"


    node-exporter:
        image: prom/node-exporter
        ports:
            - '9100:9100'

    alertmanager:
        image: prom/alertmanager
        privileged: true
        volumes:
            - ./alertmanager/alertmanager.yml:/alertmanager/alertmanager.yml
        command:
            - '--config.file=/alertmanager/alertmanager.yml'
        ports:
            - '9093:9093'

alertmanager 状态链接未显示从 docker-composer 中的卷传递的配置。它显示默认配置

Answer 1

正如一些人已经指出的那样，alertmanager 配置只包括如何发送警报而不包括创建警报。那是普罗米修斯的工作。 看看这个repo，它用alertmanager 设置prometheus 很简单

https://github.com/stefanprodan/dockprom