KAFKA - 如何重新平衡可用经纪人的分区

【问题标题】：KAFKA - how to re-balance the partitions to available brokersKAFKA - 如何重新平衡可用经纪人的分区
【发布时间】：2018-09-29 19:27:52
【问题描述】：

我们有 3 台 kafka 机器和 3 个 zookeeper 的服务器

kafka1 - 1001 ( broker ID )
kafka2 - 1002 ( broker ID )
kafka3 - 1003 ( broker ID )

我们有问题即将重新平衡可用代理的分区，我们可以看到（向下）一些分区复制不足，isr 中只有两个代理。取而代之的是 3

在 ISR 中重新平衡 kafka 主题分区的最佳方法是什么

第二个 - 我们可以看到领导者 1002 丢失了，对此有什么解决方案？

remark - 我们有 23 个主题（下面的列表是部分的）

[kafka@kafka01 bin]$ ./kafka-topics.sh -describe --zookeeper master:2181
Topic:lop_gt        PartitionCount:50       ReplicationFactor:3     Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
        Topic: lop_gt       Partition: 0    Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 1    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 2    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 3    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 4    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 5    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 6    Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 7    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 8    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 9    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 10   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 11   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 12   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 13   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 14   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 15   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 16   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 17   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 18   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 19   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 20   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 21   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 22   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 23   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 24   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 25   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 26   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 27   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 28   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 29   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 30   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 31   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 32   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 33   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 34   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 35   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 36   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 37   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 38   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 39   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 40   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 41   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 42   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 43   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 44   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 45   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 46   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 47   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 48   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 49   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
Topic:_schemas  PartitionCount:1        ReplicationFactor:3     Configs:cleanup.policy=compact
        Topic: _schemas Partition: 0    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
Topic:ambari_kafka_service_check        PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: ambari_kafka_service_check       Partition: 0    Leader: 1002    Replicas: 1002  Isr: 1002
Topic:jr_dfse    PartitionCount:10       ReplicationFactor:3     Configs:
        Topic: jr_dfse   Partition: 0    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: jr_dfse   Partition: 1    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: jr_dfse   Partition: 2    Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: jr_dfse   Partition: 3    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: jr_dfse   Partition: 4    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: jr_dfse   Partition: 5    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: jr_dfse   Partition: 6    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: jr_dfse   Partition: 7    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: jr_dfse   Partition: 8    Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: jr_dfse   Partition: 9    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
Topic:frte_nnc   PartitionCount:6        ReplicationFactor:3     Configs:
        Topic: frte_nnc  Partition: 0    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: frte_nnc  Partition: 1    Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: frte_nnc  Partition: 2    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: frte_nnc  Partition: 3    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: frte_nnc  Partition: 4    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: frte_nnc  Partition: 5    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
Topic:erw_plk    PartitionCount:100      ReplicationFactor:3     Configs:
        Topic: erw_plk   Partition: 0    Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 1    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 2    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 3    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 4    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 5    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 6    Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 7    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 8    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 9    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 10   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 11   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 12   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 13   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 14   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 15   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 16   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 17   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 18   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 19   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 20   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 21   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 22   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 23   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 24   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 25   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 26   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 27   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 28   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 29   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 30   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 31   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 32   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 33   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 34   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 35   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 36   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 37   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 38   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 39   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 40   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 41   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 42   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 43   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 44   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 45   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 46   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 47   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 48   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 49   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 50   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 51   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 52   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 53   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 54   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 55   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 56   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 57   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 58   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 59   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 60   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 61   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 62   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 63   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 64   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 65   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 66   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 67   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 68   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 69   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 70   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 71   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 72   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 73   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 74   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 75   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 76   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 77   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 78   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 79   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 80   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 81   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 82   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 83   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 84   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 85   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 86   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 87   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 88   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 89   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 90   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 91   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 92   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 93   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 94   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 95   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 96   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 97   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 98   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 99   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
Topic:loe_sd        PartitionCount:6        ReplicationFactor:3     Configs:
        Topic: loe_sd       Partition: 0    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: loe_sd       Partition: 1    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: loe_sd       Partition: 2    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: loe_sd       Partition: 3    Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: loe_sd       Partition: 4    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: loe_sd       Partition: 5    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002

【问题讨论】：

  • 你试过查看 1002 的日志吗？
  • 仍然没有，但是您对在日志中搜索什么有什么建议，您的意思是我们需要关注网络问题还是其他问题？
  • 我在日志中没有看到异常的东西，请您指教这个问题的原因是什么，所以我会集中搜索
  • 嗯，您使用 Ambari 设置集群，并且服务器应该都在同一个 LAN 中，所以如果您确实遇到了网络问题，那么您准备服务器的方式就有问题了.一个更好的解释可能是某些分区位于故障硬盘上无论如何，显然您的其中一台服务器不健康，来自 Kafka 进程的日志或其他一些 JMX 指标应该表明它有问题（提示：安装并在 HDP 中设置 Grafana 以监控 Kafka）
  • 我们还检查了网络和挂载的磁盘，它们都很好，所以我们需要搜索其他东西，看 jakub 回答他说“更多时间重新同步数据”做你知道什么意思吗？

标签： apache-kafka kafka-topic


【解决方案1】：

如何影响分区领导有两种选择。将配置选项auto.leader.rebalance.enable 设置为true（这应该是默认设置），Kafka 将自动尝试将每个分区的领导权移动到您首选的代理。首选代理是副本列表中的第一个。它作为定期检查执行，因此可能不会立即发生。

或者 - 如果关闭自动重新平衡 - 您可以使用 bin/kafka-preferred-replica-election.sh 工具手动重新分配副本。如需更多信息，请参阅Kafka docs。

但是，在您的情况下，您的代理 1002 似乎是在重新启动后不久，或者可能需要更多时间来重新同步数据，因为代理 1002 还不是所有分区的 ISR。如果这是永久状态，并且您 100% 确定您的代理有足够的时间同步所有分区等，那么 1002 可能还有其他一些问题。但是如果没有来自代理等的日志，这很难说。

【讨论】：

  • 是 auto.leader.rebalance.enable 的默认值是 true ，那么我们需要验证 broker 1002 是否不在 leader 中？ ，我需要检查一下，是否也可能是因为网络问题？
  • 第二如何验证broker有足够的时间同步所有分区？
  • 嗯，您可以随时检查日志是否有错误。至于同步分区的时间，你需要知道分区的大小是多少。您可以通过简单地检查提交日志的目录大小来使用某些文件管理器或命令行来检查它。或者您可以使用实用程序kafka-log-dirs.sh 来检查主题的分区和滞后。
猜你喜欢
  • 2013-04-17
  • 1970-01-01
  • 2021-01-26
  • 2017-03-25
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2018-02-19
  • 2017-07-05
相关资源
最近更新 更多
热门标签
Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode