简单 Hadoop Map Reduce 中的错误答案

【问题标题】：Error in simple Hadoop Map Reduce简单 Hadoop Map Reduce 中的错误
【发布时间】：2013-06-10 01:37:38
【问题描述】：

我尝试运行 hadoop map 减少字数问题。据我所知，我正确设置了 HDFS 和所有内容。

当我执行它时，

我在尝试运行一个简单的字数统计问题时遇到此错误。我是 Hadoop 新手。任何有关如何解决此问题的帮助将不胜感激。

13/06/13 20:21:17 INFO input.FileInputFormat: Total input paths to process : 4
13/06/13 20:21:17 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/06/13 20:21:17 WARN snappy.LoadSnappy: Snappy native library not loaded
13/06/13 20:21:17 INFO mapred.JobClient: Running job: job_201306131940_0007
13/06/13 20:21:18 INFO mapred.JobClient:  map 0% reduce 0%
13/06/13 20:21:21 INFO mapred.JobClient: Task Id : attempt_201306131940_0007_m_000005_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Creation of symlink from /home/vraju/Desktop/hadoop/libexec/../logs/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000005_0 to /app/hadoop/tmp/mapred/local/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000005_0 failed.
at org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:110)
at org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)

13/06/13 20:21:21 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000005_0&filter=stdout
13/06/13 20:21:21 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000005_0&filter=stderr
13/06/13 20:21:24 INFO mapred.JobClient: Task Id : attempt_201306131940_0007_m_000005_1, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Creation of symlink from /home/vraju/Desktop/hadoop/libexec/../logs/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000005_1 to /app/hadoop/tmp/mapred/local/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000005_1 failed.
at org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:110)
at org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)

13/06/13 20:21:25 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000005_1&filter=stdout
13/06/13 20:21:25 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000005_1&filter=stderr
13/06/13 20:21:28 INFO mapred.JobClient: Task Id : attempt_201306131940_0007_m_000005_2, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Creation of symlink from /home/vraju/Desktop/hadoop/libexec/../logs/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000005_2 to /app/hadoop/tmp/mapred/local/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000005_2 failed.
at org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:110)
at org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)

13/06/13 20:21:28 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000005_2&filter=stdout
13/06/13 20:21:28 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000005_2&filter=stderr
13/06/13 20:21:34 INFO mapred.JobClient: Task Id : attempt_201306131940_0007_m_000004_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Creation of symlink from /home/vraju/Desktop/hadoop/libexec/../logs/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000004_0 to /app/hadoop/tmp/mapred/local/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000004_0 failed.
at org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:110)
at org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)

13/06/13 20:21:34 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000004_0&filter=stdout
13/06/13 20:21:34 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000004_0&filter=stderr
13/06/13 20:21:37 INFO mapred.JobClient: Task Id : attempt_201306131940_0007_m_000004_1, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Creation of symlink from /home/vraju/Desktop/hadoop/libexec/../logs/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000004_1 to /app/hadoop/tmp/mapred/local/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000004_1 failed.
at org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:110)
at org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)

13/06/13 20:21:37 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000004_1&filter=stdout
13/06/13 20:21:37 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000004_1&filter=stderr
13/06/13 20:21:40 INFO mapred.JobClient: Task Id : attempt_201306131940_0007_m_000004_2, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Creation of symlink from /home/vraju/Desktop/hadoop/libexec/../logs/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000004_2 to /app/hadoop/tmp/mapred/local/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000004_2 failed.
at org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:110)
at org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
13/06/13 20:21:40 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000004_2&filter=stdout
13/06/13 20:21:40 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000004_2&filter=stderr
13/06/13 20:21:43 INFO mapred.JobClient: Job complete: job_201306131940_0007
13/06/13 20:21:43 INFO mapred.JobClient: Counters: 4
13/06/13 20:21:43 INFO mapred.JobClient:   Job Counters 
13/06/13 20:21:43 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=24100
13/06/13 20:21:43 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/06/13 20:21:43 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/06/13 20:21:43 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0

【问题讨论】：

尽管出现错误，但似乎工作正在完成：13/06/13 20:21:43 INFO mapred.JobClient: 工作完成：job_201306131940_0007。你能检查输出目录并看看你是否有那里的输出？
你在本地机器上运行吗？
但是map reduce不执行。所以工作没有完成。

标签： ubuntu hadoop mapreduce word-count

【解决方案1】：

我在尝试做同样的事情时遇到了同样的错误 - 对简单文本执行字数统计。就我而言，我解决了更改 Hadoop 所依赖的 JDK 版本的问题。我使用的是 Hadoop 1.2.1，我在网上发现它的首选 Java 版本是 1.6

【讨论】：

【解决方案2】：

我遇到了这个问题，但这是一个包名问题。

我是用maven编译的。在 pom.xml 中，我们有 <groupId>GROUPNAME</groupId>

我所有的包都必须在 GROUPNAME 中，例如 GROUPNAME.newpackage.ClassName

然后，它起作用了。

【讨论】：

【解决方案3】：

我遇到了同样的问题，我花了一整天的时间解决，直到我在网上找到一个解决方案，VirtualBox的共享目录不支持创建符号链接的操作。尝试将 hadoop 目录移动到 /usr/local 是否可以解决您的问题。

【讨论】：

【解决方案4】：

如你所愿

java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)

似乎分配给任务跟踪器的内存超过了节点的实际内存。

你可以得到更多here

【讨论】：