将指标推送到 Prometheus Pushgateway 经常失败答案

【问题标题】：Pushing metrics to Prometheus Pushgateway fails frequently将指标推送到 Prometheus Pushgateway 经常失败
【发布时间】：2020-08-27 12:29:46
【问题描述】：

我正在使用Prometheus Client library 将指标推送到推送网关。在推送指标时，我经常遇到错误。我怎样才能找到这个问题的根本原因？

 push_to_gateway(
  File "/usr/local/lib/python3.8/dist-packages/prometheus_client/exposition.py", line 285, in push_to_gateway
    _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler)
  File "/usr/local/lib/python3.8/dist-packages/prometheus_client/exposition.py", line 358, in _use_gateway
    handler(
  File "/usr/local/lib/python3.8/dist-packages/prometheus_client/exposition.py", line 217, in handle
    resp = build_opener(HTTPHandler).open(request, timeout=timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 1369, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/lib/python3.8/urllib/request.py", line 1330, in do_open
    r = h.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1332, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without respons

【问题讨论】：

标签： urllib prometheus prometheus-pushgateway

【解决方案1】：

我们在内部使用 Kubernetes 来部署服务，在这里我尝试使用其入口推送指标来推送网关。将其更改为使用 Kubernetes 服务名称而不是入口可以显着减少这些错误，但如果服务被重新定位到其他集群，这不是一个可移植的解决方案。对我有用的解决方案是使用 python 装饰器函数重试

【讨论】：