YARN MapReduce 没有足够的 RAM答案

【问题标题】：YARN MapReduce has not enough RAMYARN MapReduce 没有足够的 RAM
【发布时间】：2016-02-09 17:01:59
【问题描述】：

我正在尝试在 Yarn MapReduce 中启动我的应用程序。我在 4 个主机（1 个资源管理器和 4 个节点管理器）中设置了 MapReduce。每台主机有 2 个内核和 4GB 的 RAM。

当我运行我的应用程序时，它由于缺少 RAM [1] 而中止。我应该如何设置 Yarn MapReduce 以便作业不会耗尽 RAM？

[1] 我遇到的错误。

2016-02-09 16:01:41,607 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 289.6 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:44,612 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 289.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:47,616 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 289.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:50,621 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 289.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:53,626 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 291.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:56,631 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 295.1 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:59,643 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 295.3 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
 (...)
2016-02-09 16:02:01,173 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001/container_1455032058147_0001_01_000001/syslog
2016-02-09 16:02:01,174 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001/container_1455032058147_0001_01_000001/stdout
2016-02-09 16:02:01,174 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001/container_1455032058147_0001_01_000001/stderr
2016-02-09 16:02:01,250 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001
2016-02-09 16:02:02,644 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1455032058147_0001_01_000001
2016-02-09 16:03:10,146 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-02-09 16:03:11,147 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-02-09 16:03:12,148 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-02-09 16:03:13,149 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

【问题讨论】：

标签： hadoop mapreduce hadoop-yarn

【解决方案1】：

您需要在yarn-site.xml 中配置分配给YARN 及其容器的内存量：

<property>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>4096</value>
</property>
<property>
  <name>yarn.scheduler.minimum-allocation-mb</name>
  <value>1024</value>
</property>

检查此示例yarn-site.xml 文件。在运行 MapReduce 作业时，您还必须在 mapred-site.xmlfile 中设置类似的属性。这篇文章可能会对你有所帮助 - link。

【讨论】：