我在测试hadoop多节点集群时遇到了一些问题答案

【问题标题】：I have some problems when I test hadoop multi node cluster我在测试hadoop多节点集群时遇到了一些问题
【发布时间】：2014-04-25 13:31:55
【问题描述】：

你好我正在使用带有 2 个节点的 hadoop 0.23.9 当我尝试单个节点时它工作正常但是当我尝试测试字数统计作业时，我得到以下结果，但我不知道错误在哪里

我在两台电脑上都使用 ubuntu 12.04 LTS

        mapred-site.xml


        <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
        </property>

        </configuration>

         yarn-site.xml


<configuration>
<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce.shuffle</value>
  </property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>

<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>


 <property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
  </property>

<property>
 <name>mapreduce.job.tracker</name>
  <value>master</value>
 </property>
<configuration>

<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce.shuffle</value>
  </property>




<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>

<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>

<property>
<name>yarn.nodemanager.address</name>
<value>0.0.0.0:8042</value>
</property>

</configuration>

这里是ubuntu的hosts文件

    127.0.0.1       localhost
    127.0.1.1       fatima1-HP-ProBook-4530s

192.168.1.10    master
192.168.1.11    slave

#the following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters









        14/04/25 15:41:39 INFO input.FileInputFormat: Total input paths to process : 7
        14/04/25 15:41:39 WARN snappy.LoadSnappy: Snappy native library not loaded
        14/04/25 15:41:39 INFO mapreduce.JobSubmitter: number of splits:7
        14/04/25 15:41:39 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
        14/04/25 15:41:39 WARN conf.Configuration: mapred.output.value.class is deprecated.    Instead, use mapreduce.job.output.value.class
        14/04/25 15:41:39 WARN conf.Configuration: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
        14/04/25 15:41:39 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
        14/04/25 15:41:39 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
        14/04/25 15:41:39 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
        14/04/25 15:41:39 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
        14/04/25 15:41:39 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
        14/04/25 15:41:39 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
        14/04/25 15:41:39 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
        14/04/25 15:41:39 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
        14/04/25 15:41:39 INFO mapred.ResourceMgrDelegate: Submitted application application_1398429615167_0001 to ResourceManager at master/192.168.1.10:8032
        14/04/25 15:41:39 INFO mapreduce.Job: The url to track the job: `http://master:8088/proxy/application_1398429615167_0001/`
        14/04/25 15:41:39 INFO mapreduce.Job: Running job: job_1398429615167_0001
        14/04/25 15:41:46 INFO mapreduce.Job: Job job_1398429615167_0001 running in uber mode : false
        14/04/25 15:41:46 INFO mapreduce.Job:  map 0% reduce 0%
        14/04/25 15:41:47 INFO mapreduce.Job: Task Id : attempt_1398429615167_0001_m_000006_0, Status : FAILED
    Container launch failed for container_1398429615167_0001_01_000008 : java.lang.reflect.UndeclaredThrowableException
        at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:111)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:157)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:396)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)
    Caused by: com.google.protobuf.ServiceException: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  `http://wiki.apache.org/hadoop/UnknownHost`
        at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:144)
        at com.sun.proxy.$Proxy28.startContainer(Unknown Source)
        at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:104)
    ... 5 more
Caused by: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  `http://wiki.apache.org/hadoop/UnknownHost`
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:713)
    at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:238)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1194)
    at org.apache.hadoop.ipc.Client.call(Client.java:1074)
    at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:141)
    ... 7 more
Caused by: java.net.UnknownHostException
    ... 11 more

14/04/25 15:41:49 INFO mapreduce.Job: Task Id : attempt_1398429615167_0001_m_000006_1, Status : FAILED
Container launch failed for container_1398429615167_0001_01_000009 : java.lang.reflect.UndeclaredThrowableException
    at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:111)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:157)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:396)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: com.google.protobuf.ServiceException: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  `http://wiki.apache.org/hadoop/UnknownHost`
    at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:144)
    at com.sun.proxy.$Proxy28.startContainer(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:104)
    ... 5 more
Caused by: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  http://wiki.apache.org/hadoop/UnknownHost
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:713)
    at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:238)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1194)
    at org.apache.hadoop.ipc.Client.call(Client.java:1074)
    at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:141)
    ... 7 more
Caused by: java.net.UnknownHostException
    ... 11 more

14/04/25 15:41:51 INFO mapreduce.Job: Task Id : attempt_1398429615167_0001_m_000006_2, Status : FAILED
Container launch failed for container_1398429615167_0001_01_000010 : java.lang.reflect.UndeclaredThrowableException
    at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:111)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:157)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:396)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: com.google.protobuf.ServiceException: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  `http://wiki.apache.org/hadoop/UnknownHost`
    at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:144)
    at com.sun.proxy.$Proxy28.startContainer(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:104)
    ... 5 more
Caused by: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  `http://wiki.apache.org/hadoop/UnknownHost`
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:713)
    at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:238)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1194)
    at org.apache.hadoop.ipc.Client.call(Client.java:1074)
    at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:141)
    ... 7 more
Caused by: java.net.UnknownHostException
    ... 11 more

14/04/25 15:41:53 INFO mapreduce.Job:  map 14% reduce 0%
14/04/25 15:41:53 INFO mapreduce.Job: Job job_1398429615167_0001 failed with state FAILED due to: Task failed task_1398429615167_0001_m_000006
Job failed as tasks failed. failedMaps:1 failedReduces:0

14/04/25 15:41:53 INFO mapreduce.Job: Counters: 5
    Job Counters 
        Launched map tasks=6
        Other local map tasks=3
        Data-local map tasks=7
        Total time spent by all maps in occupied slots (ms)=0
        Total time spent by all reduces in occupied slots (ms)=0

【问题讨论】：

您的主机名设置不正确。更新/编辑您的帖子并添加有关您的操作系统和节点配置的详细信息。答案在您上面的日志中，但很大程度上取决于您正在运行的操作系统和您的节点配置。
发布 /etc/hosts 文件，并将 xxx$ 主机名的输出粘贴到本地主机上。它可以看到您的目的地 fatima-HP-ProBook-4520s 但无法解析您的本地主机
我在上面发布了 /etc/hosts 文件，但我不明白你的另一个问题
在 localhost 的终端中输入命令 xxx$ hostname 并粘贴输出。同时发布 xxx$ jps 的输出
主机名的输出：fatima1-HP-ProBook-4530s 和 jps：4167 Jps 2793 NameNode 3609 NodeManager 3394 ResourceManager 3014 DataNode 3274 SecondaryNameNode

标签： apache hadoop mapreduce hadoop-yarn

【解决方案1】：

您的主机名在 /etc/hosts 中设置不正确。您应该完全删除以下行。

127.0.1.1       fatima1-HP-ProBook-4530s

然后在 localhost 条目之后添加您的实际 IP 地址。您的 /etc/hosts 文件应如下所示。注意：我假设这是主机，我可能错了，在这种情况下用实际的 IP 地址替换它。还要检查您的目标机器 /etc/hosts 文件。

    127.0.0.1       localhost
    192.168.1.10    fatima1-HP-ProBook-4530s

192.168.1.10    master
192.168.1.11    slave

the following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

完成此操作后，您需要重新启动计算机并重新启动 hadoop

编辑：您的 core-site.xml 必须具有正确的主/从条目，如下所示。您可能在 core-site.xml 中设置了其他属性，这很好。但在分布式模式下，您需要将 fs.default.name 指定为 master 或 slave。

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://master:9000</value>
  </property>
</configuration>

【讨论】：

我尝试了上面的配置但我得到了相同的结果
我上面给你的内容对你的 /etc/hosts 是正确的。您是否生成了 ssh 主/从密钥？有关 core-site.xml 建议，请参见上文
您好抱歉迟到了，我尝试了上面的所有方法，结果相同。
如果我没有域名会出问题吗？？
您可以尝试通过将 .localdomain 添加到您的 conf 文件（Core、yarn 和 mapred）-sites.xml 中的主机名来添加完整的主机名。

【解决方案2】：

在尝试了该问题的所有可能解决方案后，解决方案是：使作为主机“主机”和从机“从机”的 PC 的主机名。然后在“/etc/hosts”文件中输入PC的IP地址。

非常感谢apesa先生的回答。

【讨论】：