允许 Prometheus alertmanager 接收器字段中的模板化字段答案

【问题标题】：Allowing templated fields in Prometheus alertmanager receiver field允许 Prometheus alertmanager 接收器字段中的模板化字段
【发布时间】：2018-06-26 13:38:49
【问题描述】：

我正在尝试实施 Brian Brazil 的建议：

https://www.robustperception.io/using-labels-to-direct-email-notifications/

我正在做的是将 OpenShift 命名空间注释添加到我的指标中，然后使用 Alertmanager 配置获取该标签。但它似乎不起作用并抱怨“to”字段为空。

time="2018-06-26T13:28:48Z" level=debug msg="Notify attempt 1 for "email" failed: parsing to addresses: mail: no address" source="notify.go:585" 
time="2018-06-26T13:28:48Z" level=error msg="Error on notify: Cancelling notify retry for "email" due to unrecoverable error: parsing to addresses: mail: no address" source="notify.go:283" 
time="2018-06-26T13:28:48Z" level=error msg="Notify for 6 alerts failed: Cancelling notify retry for "email" due to unrecoverable error: parsing to addresses: mail: no address" source="dispatch.go:262"

以下是我尝试过的一些配置：

  # default route if none match
  group_by: [annotation_contact_email]
  receiver: projectalerts

  group_by: []
  group_wait: 0s
  group_interval: 2s
  repeat_interval: 2s

receivers:
- name: alert-buffer-wh
  webhook_configs:
  - url: http://localhost:9099/topics/alerts 

- name: projectalerts
  email_configs:
  - to: '{{.GroupLabels.annotation_contact_email}}'

还有

  - name: projectalerts
      email_configs:
      - to: '{{.Labels.annotation_contact_email}}'

它适用于静态定义的电子邮件。

【问题讨论】：

annotation_contact_email 标签有什么值？
我正在使用这个查询获取它： sum by (annotation_contact_email) (floor(increase(kube_pod_container_status_restarts_total[2h])>2) * on(namespace) group_left(annotation_contact_email) kube_namespace_annotations{annotation_contact_email=~" .+"}) > 2 结果如下所示：{annotation_contact_email="email@address.com"}
@brian-brazil 见上文。

标签： prometheus prometheus-alertmanager

【解决方案1】：

我终于用这个来工作了：

global:

# The root route on which each incoming alert enters.
route:
  # default route if none match
  group_by: [annotation_contact_email]
  receiver: projectalerts

  # The labels by which incoming alerts are grouped together. For example,
  # multiple alerts coming in for cluster=A and alertname=LatencyHigh would
  # be batched into a single group.
  # TODO:
  #group_by: []
  group_wait: 0s
  group_interval: 5s
  repeat_interval: 3600s

  # All the above attributes are inherited by all child routes and can
  # overwritten on each.

receivers:
- name: alert-buffer-wh
  webhook_configs:
  - url: http://localhost:9099/topics/alerts

- name: projectalerts
  email_configs:
  - to: "{{ .GroupLabels.annotation_contact_email }}"

【讨论】：