Prometheus和Grafana下监控kafka

Question

我希望用 Prometheus 和 Grafana 监控 Kafka。

我已经下载了kafka_2.11-0.10.0.0

 cd kafka_2.11-0.10.0.0

并下载：

wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.6/jmx_prometheus_javaagent-0.6.jar
wget https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/kafka-0-8-2.yml

使用以下方法启动 Zookeeper：

nohup bin/zookeeper-server-start.sh config/zookeeper.properties >> zookeeper.log &
KAFKA_OPTS="$KAFKA_OPTS -javaagent:$PWD/jmx_prometheus_javaagent-0.6.jar=7071:$PWD/kafka-0-8-2.yml"

使用 :

启动 kafka

nohup bin/kafka-server-start.sh config/server.properties >> kafka.log &

动物园管理员日志：

INFO Got user-level KeeperException when processing sessionid:0x15b18c79a630075 type:create cxid:0x81f216 zxid:0x2b87c2 txntype:-1 reqpath:n/a Error 
Path:/consumers/logstash/ids/logstash_wavescore-staging-1490769576466-92cd1041 Error:KeeperErrorCode = NodeExists for /consumers/logstash/ids/logstash_wavescore-staging-149076
9576466-92cd1041 (org.apache.zookeeper.server.PrepRequestProcessor)
INFO Got user-level KeeperException when processing sessionid:0x15b18c79a630075 type:create cxid:0x81f219 zxid:0x2b87c3 txntype:-1 reqpath:n/a Error 
Path:/consumers/logstash/ids/logstash_wavescore-staging-1490769576466-92cd1041 Error:KeeperErrorCode = NodeExists for /consumers/logstash/ids/logstash_wavescore-staging-149076
9576466-92cd1041 (org.apache.zookeeper.server.PrepRequestProcessor)
INFO Got user-level KeeperException when processing sessionid:0x15b18c79a630075 type:create cxid:0x81f21c zxid:0x2b87c4 txntype:-1 reqpath:n/a Error 
Path:/consumers/logstash/ids/logstash_wavescore-staging-1490769576466-92cd1041 Error:KeeperErrorCode = NodeExists for /consumers/logstash/ids/logstash_wavescore-staging-149076
9576466-92cd1041 (org.apache.zookeeper.server.PrepRequestProcessor)

kafka 的日志：

INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,36] in 1 milliseconds. (kafka.coordinator.GroupMetadataM
anager)
INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,39] (kafka.coordinator.GroupMetadataManager)
INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,39] in 0 milliseconds. (kafka.coordinator.GroupMetadataM
anager)
INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,42] (kafka.coordinator.GroupMetadataManager)
INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,42] in 1 milliseconds. (kafka.coordinator.GroupMetadataM
anager)
INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,45] (kafka.coordinator.GroupMetadataManager)
INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,45] in 0 milliseconds. (kafka.coordinator.GroupMetadataM
anager)
INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,48] (kafka.coordinator.GroupMetadataManager)
INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,48] in 1 milliseconds. (kafka.coordinator.GroupMetadataM
anager)
INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator.GroupMetadataManager

当我检查服务器中的netstat 时，没有打开端口7071 并且curl localhost:7071 was resulted curl: (7) couldn't connect to host .

参考链接： https://www.robustperception.io/monitoring-kafka-with-prometheus/ 在此链接中，他们使用的是 kafka 版本：kafka_2.11-0.10.1.0.tgz

我已经下载了：kafka_2.11-0.10.0.0

Answer 1

这可能是错误的变量。我已经设法使用 jolokia 代理和 metricbeat 监控我的 kafka 代理，但它应该是相同的。这是我用来启动代理的脚本：

export KAFKA_JMX_OPTS=-javaagent:/opt/kafka/jolokia-jvm-1.3.7-agent.jar=port=8778,host=localhost
./bin/kafka-server-start.sh -daemon config/server_cluster.properties

希望对你有帮助。

注意：确保代理 jar 的所有者与您用于启动代理的用户相同。