kafka：无法在 docker 内向 kafka 生成消息

Question

docker-compose.yml (https://github.com/wurstmeister/kafka-docker)

version: "2.1"
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_CREATE_TOPICS: "test:3:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

尝试在https://kafka.apache.org/quickstart 之后生成消息时出错：

~/kafka_2.11-1.0.0$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>gh 
>[2018-01-19 17:28:15,385] ERROR Error when sending message to topic test with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for test-0: 1566 ms has passed since batch creation plus linger time

列出主题：

~/kafka_2.11-1.0.0$ bin/kafka-topics.sh --list --zookeeper localhost:2181
__consumer_offsets
test

为什么？谢谢

更新

如何设置KAFKA_ADVERTISED_HOST_NAME 或网络以使我的python/java 程序或kafka-console-producer.sh（在docker 容器外部）通过localhost:9092 向kafka 生成消息？

更新

似乎以下 docker-compose.yml 工作正常

version: "2"

services:
  zookeeper:
    image: "wurstmeister/zookeeper:latest"
    network_mode: "host"
    ports:
      - 2181:2181
  kafkaserver:
    image: "wurstmeister/kafka:latest"
    network_mode: "host"
    ports:
      - 9092:9092
    environment:
      KAFKA_CREATE_TOPICS: "test:3:1"
      KAFKA_ZOOKEEPER_CONNECT: localhost:2181

Answer 1

我遇到了同样的问题。 kafka-docker README 中建议的语法与提供的 docker-compose.yml 不匹配，后者无法按原样工作。我终于找到了这篇文章，并且 BEA 更新的 docker-compose.yml 文件的变体对我有用。谢谢！

这里是详细信息。

我正在按照https://bertrandszoghy.wordpress.com/2018/05/03/building-the-hyperledger-fabric-vm-and-docker-images-version-1-1-from-scratch/ 所述设置的 Ubuntu 16.04 虚拟映像上运行 wurstmeister/kafka-docker

我的 docker-compose.yml 文件：

version: '2'
services:
  zookeeper:
    image: "wurstmeister/zookeeper:latest"
    network_mode: "host"
    ports:
      - "2181:2181"
  kafka:
    image: "wurstmeister/kafka:latest"
    network_mode: "host"
    ports:
      - 9092:9092
    environment:
      KAFKA_LISTENERS: PLAINTEXT://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://172.17.0.1:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "BertTopic:3:1"

在我安装 NodeJs 的同一台虚拟机上：

curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash –
sudo apt-get install -y nodejs
cd
mkdir nodecode
cd nodecode
sudo npm install -g node-pre-gyp
sudo npm install kafka-node

然后我运行以下程序来生成几条消息：

var kafka = require('kafka-node'),
    Producer = kafka.Producer,
    KeyedMessage = kafka.KeyedMessage,
    client = new kafka.Client(),
    producer = new Producer(client),
    km = new KeyedMessage('key', 'message'),
    payloads = [
        { topic: 'BertTopic', messages: 'first test message', partition: 0 },
        { topic: 'BertTopic', messages: 'second test message', partition: 0 }
    ];
producer.on('ready', function () {
    producer.send(payloads, function (err, data) {
        console.log(data);
        process.exit(0);
    });
});

producer.on('error', function (err) {
console.log('ERROR: ' + err.toString());
});

返回：

{ BertTopic: { '0': 0 } }

我运行了第二个 NodeJs 程序来使用（最后）消息：

var options = {
    fromOffset: 'latest'
};

var kafka = require('kafka-node'),
    Consumer = kafka.Consumer,
    client = new kafka.Client(),
    consumer = new Consumer(
        client,
        [
            { topic: 'BertTopic', partition: 0 }
        ],
        [
                {
                        autoCommit: false
                },
                options =
                {
                        fromOffset: 'latest'
                }
        ]
    );

返回：

{ topic: 'BertTopic',
  value: 'first test message',
  offset: 0,
  partition: 0,
  highWaterOffset: 2,
  key: null }
{ topic: 'BertTopic',
  value: 'second test message',
  offset: 1,
  partition: 0,
  highWaterOffset: 2,
  key: null }

我还有第三个 NodeJs 程序来显示我的博文https://bertrandszoghy.wordpress.com/2017/06/27/nodejs-querying-messages-in-apache-kafka/中列出的主题中的所有历史消息

希望这可以帮助某人。