kafka + Apache Kafka 配置的最佳实践以避免同步问题

Question

我们有 3 台 Kafka 机器，版本 - 0.1

我们看到一些主题为 - bio_test_covid9_verification 不平衡且 ISR 不同步如下

 Topic: bio_test_covid9_verifcation  Partition: 0    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003
        Topic: bio_test_covid9_verifcation  Partition: 1    Leader: 1003    Replicas: 1001,1002,1003        Isr: 1003
        Topic: bio_test_covid9_verifcation  Partition: 2    Leader: 1002    Replicas: 1002,1003,1001        Isr: 1003,1002,1001
        Topic: bio_test_covid9_verifcation  Partition: 3    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003
        Topic: bio_test_covid9_verifcation  Partition: 4    Leader: 1003    Replicas: 1001,1003,1002        Isr: 1003
        Topic: bio_test_covid9_verifcation  Partition: 5    Leader: 1002    Replicas: 1002,1001,1003        Isr: 1003,1002,1001
        Topic: bio_test_covid9_verifcation  Partition: 6    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003
        Topic: bio_test_covid9_verifcation  Partition: 7    Leader: 1003    Replicas: 1001,1002,1003        Isr: 1003
        Topic: bio_test_covid9_verifcation  Partition: 8    Leader: 1002    Replicas: 1002,1003,1001        Isr: 1003,1002,1001
        Topic: bio_test_covid9_verifcation  Partition: 9    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003
        Topic: bio_test_covid9_verifcation  Partition: 10   Leader: 1003    Replicas: 1001,1003,1002        Isr: 1003
        Topic: bio_test_covid9_verifcation  Partition: 11   Leader: 1002    Replicas: 1002,1001,1003        Isr: 1003,1002,1001
        

预期应该是这样的

 Topic: bio_test_covid9_verifcation  Partition: 0    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003
        Topic: bio_test_covid9_verifcation  Partition: 1    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1002,1001
        Topic: bio_test_covid9_verifcation  Partition: 2    Leader: 1002    Replicas: 1002,1003,1001        Isr: 1003,1002,1001
        Topic: bio_test_covid9_verifcation  Partition: 3    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1002,1001
        Topic: bio_test_covid9_verifcation  Partition: 4    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1002,1001
        Topic: bio_test_covid9_verifcation  Partition: 5    Leader: 1002    Replicas: 1002,1001,1003        Isr: 1003,1002,1001
        Topic: bio_test_covid9_verifcation  Partition: 6    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1002,1001
        Topic: bio_test_covid9_verifcation  Partition: 7    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1002,1001
        Topic: bio_test_covid9_verifcation  Partition: 8    Leader: 1002    Replicas: 1002,1003,1001        Isr: 1003,1002,1001
        Topic: bio_test_covid9_verifcation  Partition: 9    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1002,1001
        Topic: bio_test_covid9_verifcation  Partition: 10   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1002,1001
        Topic: bio_test_covid9_verifcation  Partition: 11   Leader: 1002    Replicas: 1002,1001,1003        Isr: 1003,1002,1001
        
    

因此，总共 63 个主题中只有 2 个主题处于不良状态，即 ISR 不同步且领导者不平衡

所以为了调查这里发生的事情，我们首先验证一些 Kafka 参数，以便了解同步问题是否与不正确的 Kafka 参数有关 我这里给出了一些配置的参数

在我们的 Kafka 配置中，例如

    `min.insync.replicas=1`  ( while replication.factor is 3 )
    

据我所知，min.insync.replicas 是代理上的一个配置，表示代理允许的最小同步副本数 acks=全部

所以我想问一下min.insync.replicas=1 的这个设置是否是问题的一部分？如果我们设置min.insync.replicas=3，那么它将避免同步ISR问题

你怎么看？

Answer 1

“同步问题”（例如关闭副本）并不是代理配置错误的症状，而是表明您有其他网络或硬件故障导致代理变得不健康

您绝对应该维护min.insync.replicas=2 以确保失去一个经纪人不会丢失数据。设置为 3 以及相同的复制因子并不是真正的最佳实践

这就是in Cloudera documentation。