让 Heapster 和 InfluxDB 进行通信

Question

我正在尝试在 Kubernetes 集群上设置 Heapster/InfluxDB/Grafana。看起来 Heapster 正在生成数据，但没有将其发送到 InfluxDB。当我查看 Heapster pod 中 eventer 容器的容器日志时，日志看起来像这样：

E1020 18:43:20.006608      52 influxdb.go:147] Failed to create infuxdb: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp: lookup monitoring-influxdb on 10.254.0.10:53: read udp 172.16.23.2:43533->10.254.0.10:53: i/o timeout
I1020 18:43:30.000165      52 manager.go:95] Exporting 88 events
I1020 18:44:00.000173      52 manager.go:95] Exporting 29 events
W1020 18:44:20.000451      52 manager.go:108] Failed to events data to sink: InfluxDB Sink
E1020 18:44:20.075039      52 influxdb.go:147] Failed to create infuxdb: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp: lookup monitoring-influxdb on 10.254.0.10:53: read udp 172.16.23.2:60660->10.254.0.10:53: i/o timeout
I1020 18:44:30.000211      52 manager.go:95] Exporting 26 events
I1020 18:45:00.000169      52 manager.go:95] Exporting 23 events
W1020 18:45:20.000568      52 manager.go:108] Failed to events data to sink: InfluxDB Sink
E1020 18:45:20.004656      52 influxdb.go:147] Failed to create infuxdb: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp: lookup monitoring-influxdb on 10.254.0.10:53: read udp 172.16.23.2:53401->10.254.0.10:53: i/o timeout
I1020 18:45:30.000568      52 manager.go:95] Exporting 30 events
I1020 18:46:00.000188      52 manager.go:95] Exporting 26 events
W1020 18:46:20.000504      52 manager.go:108] Failed to events data to sink: InfluxDB Sink
E1020 18:46:20.074697      52 influxdb.go:147] Failed to create infuxdb: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp: lookup monitoring-influxdb on 10.254.0.10:53: read udp 172.16.23.2:49101->10.254.0.10:53: i/o timeout
I1020 18:46:30.000267      52 manager.go:95] Exporting 39 events
I1020 18:46:35.052332      52 influxdb.go:199] Created database "k8s" on influxDB server at "monitoring-influxdb:8086"
I1020 18:47:00.000173      52 manager.go:95] Exporting 64 events
I1020 18:47:30.000212      52 manager.go:95] Exporting 52 events
I1020 18:48:00.000234      52 manager.go:95] Exporting 43 events
I1020 18:48:30.070486      52 manager.go:95] Exporting 52 events
I1020 18:49:00.000340      52 manager.go:95] Exporting 40 events
I1020 18:49:30.000170      52 manager.go:95] Exporting 40 events
I1020 18:50:00.000205      52 manager.go:95] Exporting 57 events
I1020 18:50:30.000211      52 manager.go:95] Exporting 94 events

之后，只有一系列事件被导出。我的第一个问题是 Heapster 是成功连接到 InfluxDB 服务还是只是将数据发送到http://monitoring-influxdb:8086 假设将收集一些东西？如果确实连接了，为什么我的 InfluxDB 中的 k8s 数据库是空的？

我使用 kubernetes/contrib/ansible 中的 ansible 脚本部署了我的集群，并且 yaml 文件是从 somwhere 下载的（我不知道在哪里）。如果你想看的话，我将它们推送到了一个 git 存储库 here。

Answer 1

确定它是否正在收集数据的最简单方法 (IMO) 是查看 influxdb 的日志。在我的环境中（从https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/influxdb 部署），pod 都在 kube-system 命名空间中：

kubectl get pods -n kube-system
NAME                                    READY     STATUS    RESTARTS   AGE
heapster-603813915-tggbr                1/1       Running   0          34d
kubernetes-dashboard-3313488171-mpn5b   1/1       Running   2          43d
monitoring-grafana-2175968514-zxgn1     1/1       Running   0          34d
monitoring-influxdb-1957622127-92r2w    1/1       Running   0          34d

还有日志：

• kubectl logs monitoring-influxdb-1957622127-92r2w -n kube-system

  [httpd] 10.244.1.18 - 根 [23/Oct/2017:21:13:05 +0000] "POST /write?consistency=&db=k8s&precision=&rp=default HTTP/1.1" 204 0 "-" " heapster/v1.4.0" f5e74e91-b836-11e7-84d0-000000000000 121606 [httpd] 10.244.1.18 - 根 [23/Oct/2017:21:14:05 +0000] "POST /write?consistency=&db=k8s&precision=&rp=default HTTP/1.1" 204 0 "-" "heapster/v1 .4.0" 19a8a32c-b837-11e7-84d1-000000000000 137213 [httpd] 10.244.1.18 - 根 [23/Oct/2017:21:15:05 +0000] "POST /write?consistency=&db=k8s&precision=&rp=default HTTP/1.1" 204 0 "-" "heapster/v1 .4.0" 3d6eb415-b837-11e7-84d2-000000000000 125065 [httpd] 10.244.1.18 - 根 [23/Oct/2017:21:16:05 +0000] "POST /write?consistency=&db=k8s&precision=&rp=default HTTP/1.1" 204 0 "-" "heapster/v1 .4.0" 6133e847-b837-11e7-84d3-000000000000 128300

204 响应非常清楚，在这种情况下它正在接受数据。如果您的 heapster 由于某种原因没有与 influxdb 通信，那么答案很可能也在 influxdb pod 的日志中。