如何使用 Kafka 导出器 Docker 镜像？答案

【问题标题】：How to use the Kafka exporter Docker image?如何使用 Kafka 导出器 Docker 镜像？
【发布时间】：2022-01-03 17:53:14
【问题描述】：

我正在尝试使用 Bitnami 打包的 Kafka Exporter，https://github.com/bitnami/bitnami-docker-kafka-exporter，以及用于 Kafka 的 Bitnami 映像，https://github.com/bitnami/bitnami-docker-kafka。我正在尝试运行以下docker-compose.yml：

version: '2'

networks:
  app-tier:
    driver: bridge

services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    environment:
      - 'ALLOW_ANONYMOUS_LOGIN=yes'
    networks:
      - app-tier
  kafka:
    image: 'bitnami/kafka:latest'
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    networks:
      - app-tier
  kafka-exporter:
    image: bitnami/kafka-exporter:latest
    ports:
      - "9308:9308"
    command:
      - --kafka.server=kafka:9092

但是，如果我使用 docker-compose up 运行它，我会收到以下错误：

bitnami-docker-kafka-kafka-exporter-1  | F0103 17:44:12.545739       1 kafka_exporter.go:865] Error Init Kafka Client: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

我尝试使用How to pass arguments to entrypoint in docker-compose.yml 的答案为kafka-exporter 服务指定command，假设entrypoint 以exec 形式定义-应该在调用Docker 时附加额外的标志出口商二进制文件。但是，似乎kafka:9092 的值不适合kafka.server 标志的值，或者标志没有被拾起，或者可能存在某种竞争条件，导出器失败并在之前退出卡夫卡已经启动并运行。关于如何让这个例子工作的任何想法？

【问题讨论】：

标签： docker apache-kafka docker-compose prometheus

【解决方案1】：

这似乎只是由 Kafka 导出器在启动之前尝试连接到 Kafka 的竞争条件引起的。如果我只是运行docker-compose up 并让Kafka Exporter 失败，然后单独运行danielqsh/kafka-exporter 容器，它可以工作：

> docker run -it -p 9308:9308 --network bitnami-docker-kafka_app-tier danielqsj/kafka-exporter
I0103 18:49:04.694898       1 kafka_exporter.go:774] Starting kafka_exporter (version=1.4.2, branch=HEAD, revision=15e4ad6a9ea8203135d4b974e825f22e31c750e5)
I0103 18:49:04.703058       1 kafka_exporter.go:934] Listening on HTTP :9308

在 http://localhost:9308 上我可以看到指标：

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 8.82e-05
go_gc_duration_seconds{quantile="0.25"} 8.82e-05
go_gc_duration_seconds{quantile="0.5"} 8.82e-05
go_gc_duration_seconds{quantile="0.75"} 8.82e-05
go_gc_duration_seconds{quantile="1"} 8.82e-05
go_gc_duration_seconds_sum 8.82e-05
go_gc_duration_seconds_count 1
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 20
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.17.3"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 3.546384e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 4.492048e+06
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.448119e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 6512
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 1.6141668631896834e-06
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 4.835688e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 3.546384e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 2.686976e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 5.07904e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 4687
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 1.736704e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 7.766016e+06
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.6412358823440428e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 11199
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 9600
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 71536
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 81920
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 5.283728e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.499625e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 622592
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 622592
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 1.6270344e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 8
# HELP kafka_brokers Number of Brokers in the Kafka Cluster.
# TYPE kafka_brokers gauge
kafka_brokers 1
# HELP kafka_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which kafka_exporter was built.
# TYPE kafka_exporter_build_info gauge
kafka_exporter_build_info{branch="HEAD",goversion="go1.17.3",revision="15e4ad6a9ea8203135d4b974e825f22e31c750e5",version="1.4.2"} 1
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.11
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 12
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.703936e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.64123574422e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.35379456e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 3
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

更新

执行此操作的更可靠方法是使用包装脚本来执行https://docs.docker.com/compose/startup-order/ 中所述的特定于应用程序的运行状况检查。目录结构如下，

.
├── docker-compose.yml
└── kafka-exporter
    ├── Dockerfile
    └── run.sh

以下Dockerfile，

FROM bitnami/kafka-exporter:latest
COPY run.sh /opt/bitnami/kafka-exporter/bin

ENTRYPOINT ["run.sh"]

还有以下run.sh，

#!/bin/sh
while ! bin/kafka_exporter; do
    echo "Waiting for the Kafka cluster to come up..."
    sleep 1
done

还有以下docker-compose.yml，

version: '2'

networks:
  app-tier:
    driver: bridge

services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    environment:
      - 'ALLOW_ANONYMOUS_LOGIN=yes'
    networks:
      - app-tier
  kafka:
    image: 'bitnami/kafka:latest'
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    networks:
      - app-tier
  kafka-exporter:
    build: kafka-exporter
    ports:
      - "9308:9308"
    networks:
      - app-tier
    entrypoint: ["run.sh"]
  myapp:
    image: 'bitnami/kafka:latest'
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    networks:
      - app-tier

在运行docker-compose build && docker-compose up 后，我可以从日志中看到，大约 2 秒后（第三次尝试）Kafka 导出器成功启动：

> docker logs bitnami-docker-kafka-kafka-exporter-1 -f
I0104 16:05:39.765921       8 kafka_exporter.go:769] Starting kafka_exporter (version=1.4.2, branch=non-git, revision=non-git)
F0104 16:05:40.525065       8 kafka_exporter.go:865] Error Init Kafka Client: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
Waiting for the Kafka cluster to come up...
I0104 16:05:41.538482      16 kafka_exporter.go:769] Starting kafka_exporter (version=1.4.2, branch=non-git, revision=non-git)
F0104 16:05:42.295872      16 kafka_exporter.go:865] Error Init Kafka Client: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
Waiting for the Kafka cluster to come up...
I0104 16:05:43.307293      24 kafka_exporter.go:769] Starting kafka_exporter (version=1.4.2, branch=non-git, revision=non-git)
I0104 16:05:43.686798      24 kafka_exporter.go:929] Listening on HTTP :9308

【讨论】：

您是否尝试将depends_on 部分添加到导出器？
@OneCricketeer 根据docs.docker.com/compose/compose-file/compose-file-v3/…，depends_on 不会等待容器“准备好”，只是它已经启动。我已使用特定于应用程序的运行状况检查更新了我的答案。
没错，不过，它仍然会有一个更具确定性的容器启动顺序

【解决方案2】：

运行 kafka-exporter 所需的所有依赖项有点困难。我自己做了几个简单的步骤如下。

第一步

docker pull danielqsj/kafka-exporter:latest

第 2 步

./bin/zookeeper-server-start.sh config/zookeeper.properties

第 3 步

./bin/kafka-server-start.sh config/server.properties

第四步

docker run -ti --rm -p 9308:9308 danielqsj/kafka-exporter --kafka.server=host.docker.internal:9092 --log.enable-sarama

如果你注意到上面的命令，我使用了“host.docker.internal”，这将有助于 kafka-exporter 监听我的机器 localhost。

【讨论】：