四节点集群上的 Hadoop 复制因子为 1

Question

我在四个节点上安装了 hadoop。一个节点用于 Namenode 和辅助 NameNode。其他三个是数据节点。我运行了一个复制因子为 3 的 sqoop 作业。sqoop 作业成功，数据在所有三个数据节点上。使用 6 个映射器完成这项工作大约需要 1.5 小时。我以 1 的复制因子运行了相同的作业。这项作业也很成功，它运行了大约 1 个小时，有 12 个映射器。
我的问题是：

1. when i ran the job for second time with replication factor of 1 where is the data stored? (Is the data split and stored in all the three datanodes? (or) The data is stored on the machine from which i ran the job? )

2. I have 6 core processors on each datanode with 64 GB of ram. Which are the properties should i set to obtain optimum values for the sqoop job?

这些是第一份工作的日志：

15/06/30 00:21:28 INFO mapreduce.Job: Counters: 30 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=749046 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=864 HDFS: Number of bytes written=253986997858 HDFS: Number of read operations=24 HDFS: Number of large read operations=0 HDFS: Number of write operations=12 Job Counters Launched map tasks=6 Other local map tasks=6 Total time spent by all maps in occupied slots (ms)=20582400 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=20582400 Total vcore-seconds taken by all map tasks=20582400 Total megabyte-seconds taken by all map tasks=73767321600 Map-Reduce Framework Map input records=162991238 Map output records=162991238 Input split bytes=864 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=187671 CPU time spent (ms)=21216950 Physical memory (bytes) snapshot=5210345472 Virtual memory (bytes) snapshot=57549950976 Total committed heap usage (bytes)=6410469376 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=253986997858 15/06/30 00:21:28 INFO mapreduce.ImportJobBase: Transferred 236.5438 GB in 5,524.6156 seconds (43.8439 MB/sec) 15/06/30 00:21:28 INFO mapreduce.ImportJobBase: Retrieved 162991238 records.

这些是第二个作业的日志：

15/06/30 10:21:02 INFO mapreduce.Job: Counters: 30 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=1498130 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=1744 HDFS: Number of bytes written=253986997858 HDFS: Number of read operations=48 HDFS: Number of large read operations=0 HDFS: Number of write operations=24 Job Counters Launched map tasks=12 Other local map tasks=12 Total time spent by all maps in occupied slots (ms)=22551454 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=22551454 Total vcore-seconds taken by all map tasks=22551454 Total megabyte-seconds taken by all map tasks=80824411136 Map-Reduce Framework Map input records=162991238 Map output records=162991238 Input split bytes=1744 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=186898 CPU time spent (ms)=21910100 Physical memory (bytes) snapshot=9802846208 Virtual memory (bytes) snapshot=115099107328 Total committed heap usage (bytes)=12298747904 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=253986997858 15/06/30 10:21:02 INFO mapreduce.ImportJobBase: Transferred 236.5438 GB in 3,647.7444 seconds (66.4029 MB/sec) 15/06/30 10:21:02 INFO mapreduce.ImportJobBase: Retrieved 162991238 records.

Answer 1

这是我对您的两个问题的回答。 1. 当你以复制因子 1 运行时。你在 HDFS 中的数据块副本是 1，但数据将分布在所有三个节点上。数据块会自动分布在集群中，这就是原因。

1. 根据集群中可用的核心/插槽指定工作中的映射器数量，这将是最佳的。这里您有 6 个核心机器，我假设映射器的核心分配是 4，reducer 是 2。所以你有 4* 3*2（每个核心可以运行 2 个映射器）=24 映射器对于这项工作来说是最佳的。默认情况下

希望这能澄清你的疑问。