Yarn资源配置需要平衡服务器cpu,内存,磁盘资源。Hortonworks推荐两个Container使用1个cpu,1块磁盘能达到很好平衡。

   Hortonworks提供了yarn内存参数设置脚本,执行yarn-utils.sh(下载地址https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.1/bk_installing_manually_book/content/rpm-chap1-9.html)脚本生成配置参数。

 示例:python yarn-utils.py -c 40 -m 224 -d 12 -k False

 参数:

Option

Description

-c CORES

The number of cores on each host.

-m MEMORY

The amount of memory on each host in GB.

-d DISKS

The number of disks on each host.

-k HBASE

"True" if HBase is installed, "False" if not.

 

内存配置:

    系统及hbase预留内存参考:

Total Memory per Node

Recommended Reserved System Memory

Recommended Reserved HBase Memory

4 GB

1 GB

1 GB

8 GB

2 GB

1 GB

16 GB

2 GB

2 GB

24 GB

4 GB

4 GB

48 GB

6 GB

8 GB

64 GB

8 GB

8 GB

72 GB

8 GB

8 GB

96 GB

12 GB

16 GB

128 GB

24 GB

24 GB

256 GB

32 GB

32 GB

512 GB

64 GB

64 GB

 

计算每台服务器最多有多少个containers

containers = min (2*CORES, 1.8*DISKS, (Total available RAM) / MIN_CONTAINER_SIZE)

CORES:cpu

DISKS:磁盘数

Total available RAM:剩余内存数(减去系统及其它已使用的内存)

MIN_CONTAINER_SIZE:container最小的容量大小

 

MIN_CONTAINER_SIZE参考值:

Total RAM per Node

Recommended Minimum Container Size

Less than 4 GB

256 MB

Between 4 GB and 8 GB

512 MB

Between 8 GB and 24 GB

1024 MB

Above 24 GB

2048 MB

 

The final calculation is to determine the amount of RAM per container:

RAM-per-container = max(MIN_CONTAINER_SIZE, (Total Available RAM) / containers))

Configuration File

Configuration Setting

Value Calculation

yarn-site.xml

yarn.nodemanager.resource.memory-mb

= containers * RAM-per-container

yarn-site.xml

yarn.scheduler.minimum-allocation-mb

= RAM-per-container

yarn-site.xml

yarn.scheduler.maximum-allocation-mb

= containers * RAM-per-container

mapred-site.xml

mapreduce.map.memory.mb

= RAM-per-container

mapred-site.xml        

mapreduce.reduce.memory.mb

= 2 * RAM-per-container

mapred-site.xml

mapreduce.map.java.opts

= 0.8 * RAM-per-container

mapred-site.xml

mapreduce.reduce.java.opts

= 0.8 * 2 * RAM-per-container

yarn-site.xml (check)

yarn.app.mapreduce.am.resource.mb

= 2 * RAM-per-container

yarn-site.xml (check)

yarn.app.mapreduce.am.command-opts

= 0.8 * 2 * RAM-per-container

 

以上是Hortonworks的计算内存公式。具体可以参考官网:https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.1/bk_installing_manually_book/content/rpm-chap1-11.html

Cloudera CDH也提供了yarn资源配置的Excel表格,根据表格填写对应的数值计算并校验数值是否合理。下载地址:http://tiny.cloudera.com/yarn-tuning-guide

YARN资源配置

实际配置的时候参考Hortonworks配置值对Excel中的值进行调整及实际生产任务大小情况调整。这样下来基本可以获得比较理想的效果

分类:

技术点:

相关文章: