无法使用 docker-file 与 docker 中的 kafka 连接

Question

version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:5.5.1
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    volumes:
      - ./.docker-volumes/zookeeper/_data:/var/lib/zookeeper/data
      - ./.docker-volumes/zookeeper/_log:/var/lib/zookeeper/log

  kafka:
    image: confluentinc/cp-kafka:5.5.1
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://0.0.0.0:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    volumes:
      - ./.docker-volumes/kafka/_data:/var/lib/kafka/data

以上是 docker-compose 文件，用于在 docker 中启动 zookeeper 和 kafka 实例。

当我运行试图与 kafka 建立连接的服务时。我正在低于错误。我在互联网上找到了它，但没有得到太多。

服务层错误

.12:43:18.730 [kafka-producer-network-thread | producer-2] WARN  o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-2] Connection to node -1 (localhost/127.0.0.1:9092) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. 
.12:43:19.295 [kafka-producer-network-thread | producer-1] WARN  o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:9092) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. 
.12:43:19.481 [kafka-admin-client-thread | vendor_srs_consumer_srs_restaurant_finance_pan_validate_client_id_0] WARN  o.apache.kafka.clients.NetworkClient - [AdminClient clientId=vendor_srs_consumer_srs_restaurant_finance_pan_validate_client_id_0] Connection to node -1 (localhost/127.0.0.1:9092) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. 

当 kafka-producer 尝试与 kafka 代理连接时，Docker 出错。

[2021-07-12 07:13:19,492] WARN [SocketServer brokerId=1] Unexpected error from /172.20.0.1; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 369296129 larger than 104857600)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:105)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:447)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:397)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:678)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:580)
at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
at kafka.network.Processor.poll(SocketServer.scala:893)
at kafka.network.Processor.run(SocketServer.scala:792)
at java.lang.Thread.run(Thread.java:748)

不确定为什么生产者无法与代理连接。 提前致谢。

Answer 1

最初它们位于默认桥接网络上，为了在容器之间进行通信，您必须使用容器 IP。

您可以尝试在同一网络上部署这两个容器。

在 docker-compose 中添加网络定义

networks:
  default:
    external:
      name: zookeeper-kafka
  zookeeper-kafka:
    driver: bridge

在服务定义中，您可以添加网络名称和端口。

例如，对于zookeeper服务

zookeeper:
    networks:
    - zookeeper-kafka
    image: confluentinc/cp-zookeeper:5.5.1
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    volumes:
      - ./.docker-volumes/zookeeper/_data:/var/lib/zookeeper/data
      - ./.docker-volumes/zookeeper/_log:/var/lib/zookeeper/log

然后你可以通过zookeeper-kafka:2181（即network-name:port）访问zookeeper服务，同时通过kafka连接。

更新kafka服务定义示例

kafka:
    networks:
    - zookeeper-kafka
    image: confluentinc/cp-kafka:5.5.1
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-kafka:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://zookeeper-kafka:29092,PLAINTEXT_HOST://0.0.0.0:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    volumes:
      - ./.docker-volumes/kafka/_data:/var/lib/kafka/data

希望对你有所帮助

Answer 2

PLAINTEXT_HOST://0.0.0.0:9092 不是有效的广告侦听器；这需要是一个可解析的地址。

在你的错误中，Connection to node -1 (localhost/127.0.0.1:9092) terminated，不清楚这段代码在哪里运行，但如果在 Docker 之外的同一主机上，那么你应该宣传localhost，而不是0.0.0.0

您的另一个错误，InvalidReceiveException: Invalid receive (size = 369296129 larger than 104857600) 表示您已连接到其他服务器并得到客户端未预期的响应。