kafka + 描述 ConsumerGroupCommand 的视图答案

【问题标题】：kafka + describe views for ConsumerGroupCommandkafka + 描述 ConsumerGroupCommand 的视图
【发布时间】：2022-01-17 03:48:03
【问题描述】：

我们检查了 kafka 消费者群体，用 kafka-consumer-groups.sh --group gonb_cars --describe --bootstrap-server kafka1:6667 描述

我们有 3 个 Kafka 集群，Kafka 版本 - 1.X

当 Kafka 客户端安装在机器上时 - 192.9.200.17

输出中有趣的是 CLIENT-ID - consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f 服务于所有主题分区，但我不确定

我期望得到的是更多的消费者群体，如consumer-1 , consumer-2 , consumer-3 , 等

但我分享以下详细信息是为了了解我描述的配置是正常还是错误？

kafka-consumer-groups.sh  --group gonb_cars --describe --bootstrap-server kafka1:6667

GROUP           TOPIC                    PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
gonb_cars       cars_data.analist 97         284837505       284851115       13610           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 75         284834918       284848533       13615           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 55         284836247       284849847       13600           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 71         284837402       284851021       13619           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 193        284834839       284848444       13605           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 185        284834685       284848173       13488           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 19         284837151       284850648       13497           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 42         284835514       284849130       13616           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 59         284834551       284848159       13608           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 135        284836249       284849732       13483           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 117        284836386       284849734       13348           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 26         284837584       284850604       13020           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 50         140315368       140328987       13619           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 20         284837585       284851179       13594           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 131        140315302       140328818       13516           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 163        284836297       284849352       13055           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 155        284835748       284849365       13617           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 147        140315792       140329251       13459           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 36         284835913       284849373       13460           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 199        140315144       140328749       13605           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 175        284837539       284851148       13609           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 14         284836853       284850469       13616           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 115        284835166       284848811       13645           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 113        140315598       140329112       13514           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 84         140315725       140329237       13512           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 29         284834374       284847530       13156           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 33         284835149       284848741       13592           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 100        284835550       284849150       13600           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 88         284606827       284619888       13061           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 12         284836044       284849656       13612           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 121        284835889       284849491       13602           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 129        284835787       284849073       13286           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 189        284837129       284850596       13467           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 8          284837481       284850618       13137           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 104        140315623       140328901       13278           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 45         284837201       284850805       13604           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 81         284834957       284848440       13483           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 169        284836263       284849417       13154           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 132        140316122       140329240       13118           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 145        284835688       284849193       13505           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 181        284835218       284848860       13642           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 151        284835087       284848590       13503           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 93         284543058       284556571       13513           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 11         284834392       284847881       13489           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 159        284835770       284849285       13515           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 179        284837419       284850682       13263           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 62         284836901       284850398       13497           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 109        284835710       284849311       13601           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 171        284835340       284848886       13546           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 49         284835619       284849265       13646           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 78         284837225       284850826       13601           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 66         284837838       284850975       13137           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 69         284835066       284848551       13485           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 53         140315424       140329016       13592           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 133        140315359       140328959       13600           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 178        284835829       284849335       13506           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 79         284835854       284849452       13598           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 38         140315261       140328876       13615           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 124        284837984       284851165       13181           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 162        284835399       284849005       13606           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 82         284835704       284849350       13646           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars       cars_data.analist 149        284835242       284848840       13598           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 102        140315734       140329214       13480           

.
.
.

gonb_cars      cars_data.analist 122        284837124       284850732       13608           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 190        284834967       284848468       13501           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 40         284836001       284849172       13171           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 130        140316138       140329176       13038           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 9          284835448       284849067       13619           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 112        284834924       284848526       13602           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 142        140315258       140328869       13611           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 0          140316130       140329235       13105           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 37         140315957       140329125       13168           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 136        284835454       284848628       13174           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 16         140315438       140329085       13647           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 166        140315556       140329157       13601           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 4          140315503       140329002       13499           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 139        284835672       284849275       13603           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 3          140315708       140329199       13491           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 172        140315560       140329155       13595           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 21         284837356       284850976       13620           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 85         284835288       284848787       13499           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 95         284835237       284848252       13015           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 76         284835035       284848642       13607           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 70         140315972       140329153       13181           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 148        284835516       284849163       13647           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 54         284836267       284849889       13622           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 186        284837590       284851070       13480           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 118        140316126       140329147       13021           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 194        284836523       284850118       13595           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 126        284835543       284849162       13619           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 103        284837853       284851019       13166           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 156        284836413       284849881       13468           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
gonb_cars      cars_data.analist 101        284835379       284848892       13513           consumer-1-6d373e02-52bd-4da2-a3d2-abc93b02b48f /192.9.200.17    consumer-1
[root@kafka1 ~]#

参考 - https://dbmstutorials.com/kafka/kafka-consumer-groups.html

【问题讨论】：

也许它在您的终端上被切断了？ 200多个分区很多。或者消费者群体重新平衡，只有一个活跃的消费者。源代码绝对没有限制github.com/apache/kafka/blob/3.0/core/src/main/scala/kafka/…
另外，您的最后一个问题是 Kafka 版本 2.7。这里你说的是 1.X...所以你确实有一个以上的集群？
是的，它们是差异集群
关于您的第一条笔记 - 并非所有行都与同一消费者组相同！
1) 那么，您是否尝试过使用相同的group.id 和不同的client.id 运行多个消费者，或者甚至可能不止一个主题？ 2) 显然，所有行都有相同的消费者组，因为这是您输入的命令中描述的组

标签： apache-kafka kafka-consumer-api

【解决方案1】：

命令kafka-consumer-groups.sh --group gonb_cars --describe --bootstrap-server kafka1:6667 将仅用于描述一个消费者组，即gonb_cars

该命令似乎是正确的，并且会显示正确的结果，这个特定的消费者组中只有一个消费者，即消费者 1。

根据问题，您正在寻找多个消费者群体，可以通过-—list 参数看到。

【讨论】：

所以您同意我们“只有一个消费者，即消费者 1”。，是逻辑配置吗？，或者我们需要使用几个消费者组
@jessica 您需要提供有关您的消费者客户的更多详细信息。当然，运行一个消费者是完全有效的。
@jessica，它会起作用吗？确实。理想吗？不。消费者组是同一应用程序的多个实例如何在它们之间共享 kafka 数据，使用相同的消费者组 ID。在您的情况下，如果 consumer1 停止，则管道将停止。消费者组中的消费者可以从 0 缩放到 no。主题中的分区数。因此，如果一个消费者死亡，kafka 会重新平衡并开始将数据发送给同一组的其他成员。如果您希望将相同的数据发送到两组不同的应用程序，请使用不同的消费者组。
例如，100 个分区的首选消费组是什么
@jessica Diff 消费者组用于不同的应用程序集需要相同的数据时。理想情况下，kafka 确保将相同的数据副本发送到正在阅读主题的每个消费者组。在每个这样的组中，任何一个消费者都可以阅读它。想象一个用例，其中 diff 需要相同的主题。服务，然后可以随时动态创建消费者组并读取主题（这称为发布订阅）。现在在每个这样的组中可以有多个消费者，其数量将取决于您的用例和设计，例如高可用性和消费者效率。