hadoop 任务没有以伪分布式模式进行

Question

我尝试使用 Hadoop，然后安装并可以作为独立模式使用。 但是当我用作伪分布式模式时，出现了以下消息并且没有继续该过程。

17/10/24 02:04:15 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032

17/10/24 02:04:16 INFO input.FileInputFormat: Total input files to process : 10

17/10/24 02:04:16 INFO mapreduce.JobSubmitter: number of splits:10

17/10/24 02:04:17 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1508778206216_0001

17/10/24 02:04:17 INFO impl.YarnClientImpl: Submitted application application_1508778206216_0001

17/10/24 02:04:17 INFO mapreduce.Job: The url to track the job: http://MacBook.local:8088/proxy/application_1508778206216_0001/

17/10/24 02:04:17 INFO mapreduce.Job: Running job: job_1508778206216_0001

我检查了 localhost:50070 并且有一个工作的 Datanode。 我展示了我的设置过程。

①安装Hadoop

brew install hadoop

②hadoop配置 ○libexec/etc/hadoop/core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

○libexec/etc/hadoop/hdfs-site.xml

<configuration>
<property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

○libexec/etc/hadoop/yarn-site.xml

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
</configuration>
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>localhost</value>
</property>

---

○libexec/etc/hadoop/mapred-site.xml

<configuration>
 <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
 <property>
    <name>mapred.job.tracker</name>
    <value>localhost:54311</value>
  </property>
</configuration>

③启动hadoop

sbin/start-all.sh

与 jps

• 资源管理器
• 节点管理器
• SecondaryNameNode
• 名称节点
• 数据节点

工作。

④Hadoop的运行

hadoop jar libexec/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar pi 10 100000

然后

Number of Maps  = 10
Samples per Map = 100000
17/10/24 02:04:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
17/10/24 02:04:15 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
17/10/24 02:04:16 INFO input.FileInputFormat: Total input files to process : 10
17/10/24 02:04:16 INFO mapreduce.JobSubmitter: number of splits:10
17/10/24 02:04:17 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1508778206216_0001
17/10/24 02:04:17 INFO impl.YarnClientImpl: Submitted application application_1508778206216_0001
17/10/24 02:04:17 INFO mapreduce.Job: The url to track the job: http://MacBook.local:8088/proxy/application_1508778206216_0001/
17/10/24 02:04:17 INFO mapreduce.Job: Running job: job_1508778206216_0001

该过程从未进行。 请告诉我它不起作用的原因。我访问“http://macbook.local:8088/proxy/application_1508759907777_0001/”查看Jobtracker，但返回错误代码“ERR_EMPTY_RESPONSE”。

Answer 1

首先，请注意 Hadoop 2.x 使用 YARN 资源管理器和节点管理器代替作业跟踪器和任务跟踪器。

相反，您可以尝试将以下属性添加到yarn-site.xml：

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>localhost</value>
</property>

我不确定 Hadoop 2.x 中的 jobtracker 属性会发生什么，但它可能会产生干扰。删除它并显式设置 resourcemanager.hostname 可能会解决。

您可以通过在浏览器中打开localhost:8032 来测试它是否可用。有关设置伪分布式集群的更多信息，请访问Apache Hadoop docs。