Docker compose 不会将数据保存在卷中

Question

我正在尝试使用 docker-compose 运行 Kafka。我得到了这个 yml 文件：

version: '3'


services:
  zookeeper:
    image: ${REPOSITORY}/cp-zookeeper:${TAG}
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    volumes:
      - ./zoo:/var/lib/zookeeper

  broker:
    image: ${REPOSITORY}/cp-kafka:${TAG}
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "29092:29092"
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
    volumes:
      - ./broker:/var/lib/kafka

我在带有docker-compose.yml文件的目录中运行了一条命令：

docker-compose up -d

之后文件夹./broker 和./zoo 出现在我的目录中。它们的内部结构类似于容器内部（./zoo/data、./broker/data）。但是目录中没有文件。

我试过了

docker-compose exec broker ls /var/lib/kafka/data

我看到了有关默认主题的文件夹和文件

Answer 1

这归结为volumes（如Dockerfile 中声明的）与作为Docker Compose 一部分的您想要堆积的体积之间的交互。

如果您检查每个容器的 Dockerfile，您会看到它声明了卷，您也可以通过检查它看到。以下是使用您的配置时的样子：

➜ docker inspect zookeeper|jq '.[].Mounts[] | .Type ,.Destination'
"volume"
"/etc/zookeeper/secrets"
"bind"
"/var/lib/zookeeper"
"volume"
"/var/lib/zookeeper/log"
"volume"
"/var/lib/zookeeper/data"

您会注意到针对 ZK 的特定数据路径有两个卷（在映像本身中声明，即来自 Dockerfile）

• /var/lib/zookeeper/log
• /var/lib/zookeeper/data

另外，还有来自 Docker Compose 的绑定挂载：

• /var/lib/zookeeper/

这些冲突解释了您所看到的问题。

代理也存在类似的模式。

---

所以简而言之，您需要在映像中为每个特定卷挂载一个本地主机目录：

---
version: '3'


services:
  zookeeper:
    image: confluentinc/cp-zookeeper:5.4.1
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    volumes: 
      - ./zoo/data:/var/lib/zookeeper/data
      - ./zoo/log:/var/lib/zookeeper/log

  broker:
    image: confluentinc/cp-kafka:5.4.1
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
    volumes: 
      - ./broker/data:/var/lib/kafka/data

完成后，我们可以看到容器路径中没有冲突：

➜ docker inspect zookeeper|jq '.[].Mounts '
[
  {
    "Type": "bind",
    "Source": "/private/tmp/zoo/log",
    "Destination": "/var/lib/zookeeper/log",
    "Mode": "rw",
    "RW": true,
    "Propagation": "rprivate"
  },
  {
    "Type": "bind",
    "Source": "/private/tmp/zoo/data",
    "Destination": "/var/lib/zookeeper/data",
    "Mode": "rw",
    "RW": true,
    "Propagation": "rprivate"
  },
  {
    "Type": "volume",
    "Name": "6cbb584e0d9aa2f119869b264544f587909d9f417fc553a7bb2954dd28ecb8ea",
    "Source": "/var/lib/docker/volumes/6cbb584e0d9aa2f119869b264544f587909d9f417fc553a7bb2954dd28ecb8ea/_data",
    "Destination": "/etc/zookeeper/secrets",
    "Driver": "local",
    "Mode": "",
    "RW": true,
    "Propagation": ""
  }
]

以及来自容器的数据：

➜ docker exec zookeeper ls -l /var/lib/zookeeper/data /var/lib/zookeeper/log
/var/lib/zookeeper/data:
total 0
drwxr-xr-x 3 root root 96 Apr  3 08:59 version-2

/var/lib/zookeeper/log:
total 0
drwxr-xr-x 3 root root 96 Apr  3 08:59 version-2

➜ docker exec broker ls -l /var/lib/kafka/data
total 16
drwxr-xr-x 6 root root 192 Apr  3 08:59 __confluent.support.metrics-0
-rw-r--r-- 1 root root   0 Apr  3 08:59 cleaner-offset-checkpoint
-rw-r--r-- 1 root root   4 Apr  3 09:01 log-start-offset-checkpoint
-rw-r--r-- 1 root root  88 Apr  3 08:59 meta.properties
-rw-r--r-- 1 root root  36 Apr  3 09:01 recovery-point-offset-checkpoint
-rw-r--r-- 1 root root  36 Apr  3 09:02 replication-offset-checkpoint
-rw-r--r-- 1 root root   0 Apr  3 08:30 wibble

存储在本地主机上：

➜ ls -l broker/data zoo/data zoo/log
broker/data:
total 32
drwxr-xr-x  6 rmoff  wheel  192  3 Apr 09:59 __confluent.support.metrics-0
-rw-r--r--  1 rmoff  wheel    0  3 Apr 09:59 cleaner-offset-checkpoint
-rw-r--r--  1 rmoff  wheel    4  3 Apr 10:00 log-start-offset-checkpoint
-rw-r--r--  1 rmoff  wheel   88  3 Apr 09:59 meta.properties
-rw-r--r--  1 rmoff  wheel   36  3 Apr 10:00 recovery-point-offset-checkpoint
-rw-r--r--  1 rmoff  wheel   36  3 Apr 10:01 replication-offset-checkpoint
-rw-r--r--  1 rmoff  wheel    0  3 Apr 09:30 wibble

zoo/data:
total 0
drwxr-xr-x  3 rmoff  wheel  96  3 Apr 09:59 version-2

zoo/log:
total 0
drwxr-xr-x  3 rmoff  wheel  96  3 Apr 09:59 version-2

另见Data Volumes for Kafka and ZooKeeper

Answer 2

添加到罗宾的答案：

如果您不想直接挂载目录，而是使用 docker 的命名卷，您仍然可以这样做。 你可以参考我用来让它工作的 docker-compose 文件。

链接 - https://github.com/girishyt/kafka-docker-compose/blob/main/docker-compose-simple.yml