Confluent kafka Python 客户端 Avro producer.producer() 执行没有错误，但主题中没有数据答案

【问题标题】：Confluent kafka Python client Avro producer.producer() executes without error but no data in topicConfluent kafka Python 客户端 Avro producer.producer() 执行没有错误，但主题中没有数据
【发布时间】：2020-04-03 17:18:47
【问题描述】：

我的生产者没有抛出任何错误，但数据没有发送到目标主题。你能推荐任何技术来调试这种情况吗？

我在同步循环中调用了 Confluent Python Avro Producer，以将数据发送到如下主题：

self.producer.produce(topic=test2, value=msg_dict)

在此调用之后，我有一段类似这样的代码来刷新队列：

num_messages_in_queue = self.producer.flush(timeout = 2.0)
print(f"flushed {num_messages_in_queue} messages from producer queue in iteration {num_iterations} ")

这个执行没有任何错误。但是在此代码执行后也没有触发回调。我的制作人发起如下：

 def __init__(self,broker_url=None,topic=None,schema_registry_url=None,schema_path=None):
  try:
    with open(schema_path, 'r') as content_file:
      schema = avro.loads(content_file.read())
  except Exception as e:
    print(f"Error when trying to read avro schema file : {schema_path}")

  self.conf = {
    'bootstrap.servers': broker_url,
    'on_delivery': self.delivery_report,
    'schema.registry.url': schema_registry_url,
    'acks': -1, #This guarantees that the record will not be lost as long as at least one in-sync replica remains alive. 
    'enable.idempotence': False, #
    "error_cb":self.error_cb
  }
  self.topic = topic
  self.schema_path = schema_path
  self.producer = AvroProducer(self.conf,default_key_schema=schema, default_value_schema=schema)

我的回调方法如下：

def delivery_report(self, err, msg):
    print(f"began delivery_report")
    if err is None:
        print(f"delivery_report --> Delivered msg.value = {msg.value()} to topic= {msg.topic()} offset = {msg.offset} without err.")
    else:
        print(f"conf_worker AvroProducer failed to deliver message {msg.value()} to topic {self.topic}. got error= {err}")

这段代码执行后，我在模式注册表容器上查看我的主题，如下所示：

docker exec schema_registry_container kafka-avro-console-consumer --bootstrap-server kafka:29092 --topic test2 --from-beginning

我看到了这个输出：

[2020-04-03 15:48:38,064] 信息已注册 kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$) [2020-04-03 15:48:38,742] 信息 ConsumerConfig 值： auto.commit.interval.ms = 5000 auto.offset.reset = 最早 bootstrap.servers = [kafka:29092] check.crcs = true client.dns.lookup = 默认客户.id = 连接数.max.idle.ms = 540000 default.api.timeout.ms = 60000 enable.auto.commit = false exclude.internal.topics = true fetch.max.bytes = 52428800 fetch.max.wait.ms = 500 fetch.min.bytes = 1 group.id = 控制台消费者 49056 heartbeat.interval.ms = 3000 拦截器.classes = [] internal.leave.group.on.close = true 隔离级别 = read_uncommitted key.deserializer = 类 >> org.apache.kafka.common.serialization.ByteArrayDeserializer max.partition.fetch.bytes = 1048576 max.poll.interval.ms = 300000 max.poll.records = 500 元数据.max.age.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = 信息指标.sample.window.ms = 30000 partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor] 接收缓冲区字节 = 65536 重新连接.backoff.max.ms = 1000 重新连接.backoff.ms = 50 request.timeout.ms = 30000 重试.backoff.ms = 100 sasl.client.callback.handler.class= null sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class= null sasl.login.class= null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.mechanism = GSSAPI security.protocol = PLAINTEXT 发送缓冲区字节 = 131072 session.timeout.ms = 10000 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2，TLSv1.1，TLSv1] ssl.endpoint.identification.algorithm = https ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLS ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS value.deserializer = 类 >>org.apache.kafka.common.serialization.ByteArrayDeserializer (org.apache.kafka.clients.consumer.ConsumerConfig) [2020-04-03 15:48:38,887] INFO Kafka 版本：2.1.0-cp1 (org.apache.kafka.common.utils.AppInfoParser) [2020-04-03 15:48:38,887] INFO Kafka commitId：bda8715f42a1a3db (org.apache.kafka.common.utils.AppInfoParser) [2020-04-03 15:48:39,221] 信息集群 ID：KHKziPBvRKiozobbwvP1Fw (org.apache.kafka.clients.Metadata) [2020-04-03 15:48:39,224] INFO [Consumer clientId=consumer-1, groupId=console-consumer-49056] 发现组协调员 kafka:29092 (id: 2147483646 rack: null) (org.apache.kafka. clients.consumer.internals.AbstractCoordinator) [2020-04-03 15:48:39,231] 信息 [Consumer clientId=consumer-1, groupId=console-consumer-49056] 撤销以前分配的分区 [] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [2020-04-03 15:48:39,231] INFO [Consumer clientId=consumer-1, groupId=console-consumer-49056]（重新）加入组>（org.apache.kafka.clients.consumer.internals.AbstractCoordinator ) [2020-04-03 15:48:42,264] INFO [Consumer clientId=consumer-1, groupId=console-consumer-49056] 已成功加入第 1 代组 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [2020-04-03 15:48:42,267] INFO [Consumer clientId=consumer-1, groupId=console-consumer-49056] 设置新分配的分区 [test2-0] >(org.apache.kafka.clients.consumer. internals.ConsumerCoordinator) [2020-04-03 15:48:42,293] INFO [Consumer clientId=consumer-1, groupId=console-consumer-49056] 将分区 test2-0 的偏移量重置为偏移量 0。 >(org.apache.kafka.clients.消费者.internals.Fetcher)

【问题讨论】：

标签： kafka-producer-api confluent-schema-registry confluent-platform

【解决方案1】：

所以答案是如此琐碎以至于令人尴尬！但它确实指出了这样一个事实，即在多层基础架构中，错误地设置单个值可能会导致无声故障，这可能非常繁琐。

所以问题来自我在 docker-compose.yml 文件中的参数设置不正确，其中未设置 broker_url 的 env 变量。应用程序代码需要这个变量来引用 kafka 代理。但是，没有为这个丢失的参数抛出异常，它默默地失败了。

【讨论】：