Docker compose 创建 kafka 主题

Question

问题：无法从 docker-compose 创建主题。在运行被测系统之前，我需要创建 kafka 主题。计划将其用作管道的一部分，因此不可以使用 UI。

注意：kafka 需要大约 15 秒才能准备好，所以我需要在添加主题之前休眠 15 秒。

可能的解决方案：

1. 创建一个 shell.sh 文件，命令等待 15 秒，然后添加一堆主题
2. 为它创建一个 dockerfile
3. 在启动被测系统之前将该 docker 映像包含在 docker-compose.yml 中

当前流量：

1. 创建 zookeeper - 确定
2. 创建 kafka1 - 确定
3. rest-proxy - 好的
4. 创建主题
5. 创建 SUT - 确定

当前 docker-compose.yml：

version: '2'
services:
zookeeper:
image: docker.io/confluentinc/cp-zookeeper:5.4.1
hostname: zookeeper
container_name: zookeeper
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000

Kafka1:
image: docker.io/confluentinc/cp-enterprise-kafka:5.4.1
hostname: Kafka1
container_name: Kafka1
depends_on:
- zookeeper
ports:
- "9092:9092"
- "9101:9101"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_ADVERTISED_HOST_NAME: Kafka1
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://Kafka1:29092,PLAINTEXT_HOST://localhost:9092
KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_JMX_PORT: 9101
KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL: http://schema-registry:8081
CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: Kafka1:29092
CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: zookeeper:2181
CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
CONFLUENT_METRICS_ENABLE: 'true'
CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'

rest-proxy:
image: docker.io/confluentinc/cp-kafka-rest:5.4.1
depends_on:
- zookeeper
- Kafka1
ports:
- 8082:8082
hostname: rest-proxy
container_name: rest-proxy
environment:
KAFKA_REST_HOST_NAME: rest-proxy
KAFKA_REST_BOOTSTRAP_SERVERS: 'Kafka1:29092'
KAFKA_REST_LISTENERS: "http://0.0.0.0:8082"
KAFKA_REST_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081'


topics:
image: topics:latest
hostname: topics
container_name: topics
depends_on:
- zookeeper
- Kafka1
- rest-proxy


sut:
image: sut:latest
hostname: sut
container_name: sut
depends_on:
- zookeeper
- Kafka1
- rest-proxy
ports:
- 5000:80

主题容器的当前 Dockerfile：

FROM ubuntu:14.04

ADD topics.sh /usr/local/bin/topics.sh

RUN chmod +x /usr/local/bin/topics.sh

CMD /usr/local/bin/topics.sh

当前topics.sh文件：

#!/bin/sh
echo "Start: Sleep 15 seconds"
sleep 30;
wait;
echo "Begin creating topics"
docker exec Kafka1 kafka-topics --create --if-not-exists --zookeeper zookeeper:2181 --partitions 1 --replication-factor 1 --topic MY_AWESOME_TOPIC_ONE
docker exec Kafka1 kafka-topics --create --if-not-exists --zookeeper zookeeper:2181 --partitions 1 --replication-factor 1 --topic MY_AWESOME_TOPIC_TWO
echo "Done creating topics"

电流输出：

/usr/local/bin/topics.sh: 1: /usr/local/bin/topics.sh: #!/bin/sh: not found
Start: Sleep 15 seconds
Begin creating topics
/usr/local/bin/topics.sh: 8: /usr/local/bin/topics.sh: docker: not found
/usr/local/bin/topics.sh: 9: /usr/local/bin/topics.sh: docker: not found
/usr/local/bin/topics.sh: 10: /usr/local/bin/topics.sh: docker: not found
/usr/local/bin/topics.sh: 11: /usr/local/bin/topics.sh: docker: not found
/usr/local/bin/topics.sh: 12: /usr/local/bin/topics.sh: docker: not found
/usr/local/bin/topics.sh: 13: /usr/local/bin/topics.sh: docker: not found
/usr/local/bin/topics.sh: 14: /usr/local/bin/topics.sh: docker: not found
/usr/local/bin/topics.sh: 15: /usr/local/bin/topics.sh: docker: not found
Done creating topics

未创建主题。我被困住了。请帮忙。

Answer 1

ubuntu 容器没有安装 docker。

它也没有kafka-topics 命令，因此您应该重新使用您已经拉出的cp-enterprise-kafka 图像并将 ENTRYPOINT 或 CMD 更改为您的脚本，但运行 kafka-topics 命令直接

或者用 wurstmeister/kafka 替换你的 Kafka 容器，并添加一个环境变量来创建主题

Answer 2

变体 1：运行 topic.sh（只是 kafka-topics --create 在另一个 docker 容器中）

很抱歉没有提供完整的例子，但让我分享一下这个想法：

1. 创建一个单独的 docker 容器“kafka-setup”，这是获取 kafka 命令行工具所必需的。在那个替换启动命令以执行一些（足够好）等待操作并运行通过卷注入的/kafka/topic_creator.sh（带有zookeeper和kafka的主机：端口参数）。执行该脚本后，将创建一个文件 K_OUTPUT_FILE 并通过卷公开（调用 docker-compose up 之前的先决条件，该文件需要删除）。

来自 docker-compose.yml 的片段（./kafka 文件夹包含主题创建者脚本，./output 获取主题创建文件 'kafka-done.txt'）：

  # This "container" is a helper to pre-create topics
  kafka-setup:
    image: confluentinc/cp-kafka:5.4.3
    depends_on:
      - kafka
    volumes:
      - ./kafka:/kafka
      - ./output:/output
    command: "bash -c 'chmod +x /kafka/topic_creator.sh && \
                       /kafka/topic_creator.sh /kafka/topics.txt $$K_ZK $$K_KAFKA && \
                       touch \"$${K_OUTPUT_FILE}\" && chmod a+rw  \"$${K_OUTPUT_FILE}\"'"
    environment:
      K_ZK: localhost:22181
      K_KAFKA: localhost:19092
      K_OUTPUT_FILE: "/output/kafka-done.txt"

      # dummy values
      KAFKA_BROKER_ID: ignored
      KAFKA_ZOOKEEPER_CONNECT: ignored
    network_mode: host

1. 以正确的顺序运行所有内容

bash 脚本 sn-p:

K_SETUP_OUTPUT="./output"
mkdir -p "$K_SETUP_OUTPUT"
rm -f "$K_SETUP_OUTPUT/kafka-done.txt"

# Start stuff
docker-compose up -d --force-recreate --build --remove-orphans

wait_for_file "$K_SETUP_OUTPUT/kafka-done.txt"
sleep 5
# do your stuff here (e.g.  read -r -p "Press any key to continue..." key )
do_something

带有wait_for_file函数

function wait_for_file {
    local name=$1
    echo "File waiting ${name}."
    seconds=0
    while [[ "$seconds" -lt "$timeout"  && ! -f "$name" ]];
    do
      echo -n .
      seconds=$((seconds+1))
      sleep 1
    done

    if [ "$seconds" -lt "$timeout" ]; then
      echo "${name} created (${seconds}s)!"
    else
      echo "  ERROR: not found ${name}" >&2
      exit 1
    fi
}

如何按顺序工作：

1. 运行测试脚本
2. 创建文件夹并删除./output/kafka-done.txt
3. 执行 docker-compose up
4. 在 kafka-setup 中首先等待 zookeeper 和 kafka 端口的可用性
5. 使用 zookeeper 和 kafka 端口的参数运行 ./kafka/topic_creator.sh
6. 创建./output/kafka-done.txt
7. wait_for_file ./output/kafka-done.txt 成功
8. do_something .. 那是你的测试或其他什么

变体 2：只允许 docker 在没有 root 的情况下运行

见Manage Docker as a non-root user

Answer 3

请在kafka服务的compose文件中包含kafka容器卷中的docker路径添加-v $(which docker):/usr/bin/docker

Answer 4

此解决方案允许使用从 docker-compse.yml 创建主题

• 参考你各自kafka镜像服务的DockerFile

• 记下相应 dockerFile 镜像中的最后一条命令（镜像/镜像层的 DockerHub 存储库）

• 就我的图像 blockconfluentinc/cp-kafka:latest 而言，启动 kafka 服务的最后一个命令是“/etc/confluent/docker/run”

• 因此在您的 docker-compose.yml 中包含以下命令

  command: sh -c "((sleep 15 && kafka-topics --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 3 --topic topicName)&) && /etc/confluent/docker/run "

这将启动kafka服务，延迟15秒然后创建一个主题。

请注意，我们假设 kafka 服务需要 15 秒才能完全运行

    kafka:
        image: confluentinc/cp-kafka:latest
        depends_on:
            - zookeeper
        ports:
            - "29092:29092"
        environment:
            KAFKA_BROKER_ID: 1
            KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
            KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
            KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
            KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
            KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
        command: sh -c "((sleep 15 && kafka-topics --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 3 --topic quick-starter)&) && /etc/confluent/docker/run ">

Answer 5

最简单的方法是在 docker-compose 文件中启动一个单独的容器（在下面的示例中称为 init-kafka）以启动各种 kafka-topics --create ... 命令，同时首先通过简单地运行使其等待 Kafka 可以访问kafka-topics --list ....

像这样：

version: '2.1'
services:

  zookeeper:
    image: confluentinc/cp-zookeeper:6.1.1
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181

  # reachable on 9092 from the host and on 29092 from inside docker compose
  kafka:
    image: confluentinc/cp-kafka:6.1.1
    depends_on:
      - zookeeper
    ports:
      - '9092:9092'
    expose:
      - '29092'
    environment:
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: '1'
      KAFKA_MIN_INSYNC_REPLICAS: '1'

  init-kafka:
    image: confluentinc/cp-kafka:6.1.1
    depends_on:
      - kafka
    entrypoint: [ '/bin/sh', '-c' ]
    command: |
      "
      # blocks until kafka is reachable
      kafka-topics --bootstrap-server kafka:29092 --list

      echo -e 'Creating kafka topics'
      kafka-topics --bootstrap-server kafka:29092 --create --if-not-exists --topic my-topic-1 --replication-factor 1 --partitions 1
      kafka-topics --bootstrap-server kafka:29092 --create --if-not-exists --topic my-topic-2 --replication-factor 1 --partitions 1

      echo -e 'Successfully created the following topics:'
      kafka-topics --bootstrap-server kafka:29092 --list
      "

运行它时，init-kafka 容器应该记录如下内容：

docker logs docker_init-kafka_1


[2021-10-12 02:00:28,728] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.24.0.3:29092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2021-10-12 02:00:28,832] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.24.0.3:29092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2021-10-12 02:00:29,033] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.24.0.3:29092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2021-10-12 02:00:29,335] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.24.0.3:29092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)


Creating kafka topics
Created topic my-topic-1.
Created topic my-topic-2.
Successfully created the following topics:
my-topic-1
my-topic-2