Docker Stack 中错误的 HDFS 配置容量

Question

我正在使用一个 Docker 堆栈，该堆栈在同一台机器上实现了一个 Hadoop Namenode、两个 Datanode、两个节点管理器、一个资源管理器、一个历史服务器和其他技术。 p>

我遇到了与 HDFS UI 中显示的 HDFS 配置容量相关的问题。

我正在使用一台容量为 256GB 的机器，并且我正在使用上面提到的两个 datanodes 实现。 HDFS 不是在两个节点之间分配总容量，而是通过为每个数据节点分配 226.87GB 来复制整个机器的容量。 As you can see here.

关于如何让 HDFS 显示正确的容量有什么想法吗？

这是实现上述 hadoop 技术的 docker compose 部分。

    services:
  # Hadoop master
  namenode:
    image: bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8
    container_name: namenode
    ports:
      - 9870:9870
      - 8020:8020
    volumes:
      - ./namenode/home/${ADMIN_NAME:?err}:/home/${ADMIN_NAME:?err}
      - ./namenode/hadoop-data:/hadoop-data
      - ./namenode/entrypoint.sh:/entrypoint.sh
      - hadoop-namenode:/hadoop/dfs/name
    env_file:
      - ./hadoop.env
      - .env
    networks:
      - hadoop

  resourcemanager:
    restart: always
    image: bde2020/hadoop-resourcemanager:2.0.0-hadoop3.2.1-java8
    container_name: resourcemanager
    ports:
      - 8088:8088
    environment:
      SERVICE_PRECONDITION: "namenode:9870 datanode1:9864"
    env_file:
      - ./hadoop.env
    networks:
      - hadoop

  # Hadoop slave 1
  datanode1:
    image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8
    container_name: datanode1
    volumes:
      - hadoop-datanode-1:/hadoop/dfs/data
    environment:
      SERVICE_PRECONDITION: "namenode:9870"
    env_file:
      - ./hadoop.env
    networks:
      - hadoop

  nodemanager1:
    image: bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8
    container_name: nodemanager1
    volumes:
      - ./nodemanagers/entrypoint.sh:/entrypoint.sh
    environment:
      SERVICE_PRECONDITION: "namenode:9870 datanode1:9864 resourcemanager:8088"
    env_file:
      - ./hadoop.env
      - .env
    networks:
      - hadoop

  # Hadoop slave 2
  datanode2:
    image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8
    container_name: datanode2
    volumes:
      - hadoop-datanode-2:/hadoop/dfs/data
    environment:
      SERVICE_PRECONDITION: "namenode:9870"
    env_file:
      - ./hadoop.env
    networks:
      - hadoop

  nodemanager2:
    image: bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8
    container_name: nodemanager2
    volumes:
      - ./nodemanagers/entrypoint.sh:/entrypoint.sh
    environment:
      SERVICE_PRECONDITION: "namenode:9870 datanode2:9864 resourcemanager:8088"
    env_file:
      - ./hadoop.env
      - .env
    networks:
      - hadoop

  historyserver:
    image: bde2020/hadoop-historyserver:2.0.0-hadoop3.2.1-java8
    container_name: historyserver
    ports:
      - 8188:8188
    environment:
      SERVICE_PRECONDITION: "namenode:9870 datanode1:9864 datanode2:9864 resourcemanager:8088"
    volumes:
      - hadoop-historyserver:/hadoop/yarn/timeline
    env_file:
      - ./hadoop.env
    networks:
      - hadoop

Answer 1

您需要创建具有适合您机器的已定义大小的 docker 卷，然后要求每个 DN 使用该卷。然后当 DN 检查其卷的大小时，它应该返回卷的大小而不是整个机器的容量并将其用于容量。