在 docker compose 中将 kafka 连接器与 kafka 代理连接时，代理可能不可用答案

【问题标题】：Broker may not be available in connecting kafka connector with kafka broker in docker compose在 docker compose 中将 kafka 连接器与 kafka 代理连接时，代理可能不可用
【发布时间】：2020-10-19 12:21:49
【问题描述】：

我正在使用docker容器来构建zookeeper、kafka和connect。

version: '2.1'
services:


  zookeeper:
    image: debezium/zookeeper
    ports:
     - 2181:2181
     - 2888:2888
     - 3888:3888
  kafka:
    image: wurstmeister/kafka
    container_name: kafka-multibinder-1
    ports:
      - "9092:9092"
      - "9094:9094"

    environment:
      - KAFKA_ADVERTISED_HOST_NAME=127.0.0.1
      - KAFKA_ADVERTISED_PORT=9092
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181

      - KAFKA_ADVERTISED_LISTENERS=INSIDE://:9094,OUTSIDE://localhost:9092
      - KAFKA_LISTENERS=INSIDE://:9094,OUTSIDE://:9092
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      - KAFKA_INTER_BROKER_LISTENER_NAME=INSIDE

    depends_on:
      - zookeeper

  kafka-connect:
    image: debezium/connect
    hostname: kafka-connect
    ports:
     - 8083:8083
    depends_on:
     - kafka
    environment:
     BOOTSTRAP_SERVERS: kafka:9092
     GROUP_ID: 1
     CONFIG_STORAGE_TOPIC: my_connect_configs
     OFFSET_STORAGE_TOPIC: my_connect_offsets

但是日志温暖了我，'无法建立。经纪人可能不可用'

kafka-connect_1  | 2020-10-18 04:11:44,671 INFO   ||  Kafka version: 2.5.0   [org.apache.kafka.common.utils.AppInfoParser]
kafka-connect_1  | 2020-10-18 04:11:44,672 INFO   ||  Kafka commitId: 66563e712b0b9f84   [org.apache.kafka.common.utils.AppInfoParser]
kafka-connect_1  | 2020-10-18 04:11:44,674 INFO   ||  Kafka startTimeMs: 1602994304669   [org.apache.kafka.common.utils.AppInfoParser]
kafka-connect_1  | 2020-10-18 04:11:44,806 WARN   ||  [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available.   [org.apache.kafka.clients.NetworkClient]
kafka-connect_1  | 2020-10-18 04:11:44,918 WARN   ||  [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available.   [org.apache.kafka.clients.NetworkClient]
kafka-connect_1  | 2020-10-18 04:11:45,024 WARN   ||  [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available.   [org.apache.kafka.clients.NetworkClient]
kafka-connect_1  | 2020-10-18 04:12:44,713 INFO   ||  [AdminClient clientId=adminclient-1] Metadata update failed   [org.apache.kafka.clients.admin.internals.AdminMetadataManager]
kafka-connect_1  | org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1602994364719) timed out at 9223372036854775807 after 1 attempt(s)
kafka-connect_1  | Caused by: org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited.
kafka-connect_1  | 2020-10-18 04:12:44,722 ERROR  ||  Stopping due to error   [org.apache.kafka.connect.cli.ConnectDistributed]
kafka-connect_1  | org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
kafka-connect_1  |  at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
kafka-connect_1  |  at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:45)
kafka-connect_1  |  at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:95)
kafka-connect_1  |  at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
kafka-connect_1  | Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1602994364707) timed out at 1602994364708 after 1 attempt(s)
kafka-connect_1  |  at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
kafka-connect_1  |  at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
kafka-connect_1  |  at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
kafka-connect_1  |  at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
kafka-connect_1  |  at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:58)
kafka-connect_1  |  ... 3 more
kafka-connect_1  | Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1602994364707) timed out at 1602994364708 after 1 attempt(s)
kafka-connect_1  | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.

【问题讨论】：

主机是否提供代理？

标签： apache-kafka docker-compose apache-kafka-connect

【解决方案1】：

在您的 Kafka 配置中，端口 9094 用于代理间通信。但是在 Kafka 连接中，您尝试通过端口 9092 连接到代理。

当您尝试连接到端口 9092 时，localhost:9092 将作为元数据发送到 Kafka 连接以进行进一步通信，但 localhost:9092 在 docker 网络中不可用。使用下面的 Kafka 连接配置进行连接 -

kafka-connect:
    image: debezium/connect
    hostname: kafka-connect
    ports:
     - 8083:8083
    depends_on:
     - kafka
    environment:
     BOOTSTRAP_SERVERS: kafka:9094
     GROUP_ID: 1
     CONFIG_STORAGE_TOPIC: my_connect_configs
     OFFSET_STORAGE_TOPIC: my_connect_offsets

Advertised listeners 的概念有点棘手。如果您想了解有关听众和广告听众的更多信息，请参阅此link。

【讨论】：