火花读取文件扩展名

Question

我编写了以下 shell 脚本来配置纱线调度程序，但这不能正常工作 - 当我将此脚本设置为输入参数时，Dataproc 集群的创建失败。

您知道如何解决这个问题吗？

下面是脚本：

#!/usr/bin/env bash

echo "<allocations>" >> /etc/hadoop/conf/fair-scheduler.xml
echo "  <userMaxAppsDefault>999</userMaxAppsDefault>" >> /etc/hadoop/conf/fair-scheduler.xml
echo "  <queueMaxAppsDefault>999</queueMaxAppsDefault>" >> /etc/hadoop/conf/fair-scheduler.xml
echo "</allocations>" >> /etc/hadoop/conf/fair-scheduler.xml

sed -i '$ d' /etc/hadoop/conf/yarn-site.xml

echo "  <property>" >> /etc/hadoop/conf/yarn-site.xml
echo "    <name>yarn.scheduler.fair.allocation.file</name>" >> /etc/hadoop/conf/yarn-site.xml
echo "    <value>/etc/hadoop/conf/fair-scheduler.xml</value>" >> /etc/hadoop/conf/yarn-site.xml
echo "  </property>" >> /etc/hadoop/conf/yarn-site.xml
echo "</configuration>" >> /etc/hadoop/conf/yarn-site.xml

systemctl restart hadoop-yarn-resourcemanager.service

Answer 1

您需要使用 Dataproc initialization action 在 Dataproc 上配置 YARN Fair Scheduler。

您可以查看此答案以了解如何完成此操作的示例：https://stackoverflow.com/a/49693693/3227693