如何让 hadoop wordcount 示例工作？

Question

大家好，今天我按照本教程设置了一个 hadoop 多节点集群 http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/ 当我准备运行示例程序时，它给了我这个

 11/07/31 13:44:30 INFO input.FileInputFormat: Total input paths to process : 4
11/07/31 13:44:31 INFO mapred.JobClient: Running job: job_201107311331_0001
11/07/31 13:44:32 INFO mapred.JobClient:  map 0% reduce 0%
11/07/31 13:44:55 INFO mapred.JobClient: Task Id : attempt_201107311331_0001_m_000002_0, Status : FAILED
java.io.FileNotFoundException: File does not exist: /user/hduser/gutenberg/gutenberg
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1590)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1581)
        at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:423)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
        at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:67)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:521)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:762)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.mapred.Child.main(Child.java:253)

11/07/31 13:44:56 INFO mapred.JobClient:  map 25% reduce 0%
11/07/31 13:44:58 INFO mapred.JobClient:  map 50% reduce 0%
11/07/31 13:45:10 INFO mapred.JobClient:  map 75% reduce 8%
11/07/31 13:45:22 INFO mapred.JobClient: Task Id : attempt_201107311331_0001_m_000002_1, Status : FAILED
java.io.FileNotFoundException: File does not exist: /user/hduser/gutenberg/gutenberg
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1590)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1581)
        at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:423)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
        at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:67)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:521)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:762)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.mapred.Child.main(Child.java:253)

11/07/31 13:45:22 WARN mapred.JobClient: Error reading task outputACRS-HCN1
11/07/31 13:45:22 WARN mapred.JobClient: Error reading task outputACRS-HCN1
11/07/31 13:45:29 INFO mapred.JobClient: Task Id : attempt_201107311331_0001_m_000002_2, Status : FAILED
java.io.FileNotFoundException: File does not exist: /user/hduser/gutenberg/gutenberg
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1590)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1581)
        at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:423)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
        at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:67)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:521)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:762)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.mapred.Child.main(Child.java:253)

11/07/31 13:45:29 WARN mapred.JobClient: Error reading task outputACRS-HCN1
11/07/31 13:45:29 WARN mapred.JobClient: Error reading task outputACRS-HCN1
11/07/31 13:45:44 INFO mapred.JobClient: Job complete: job_201107311331_0001
11/07/31 13:45:44 INFO mapred.JobClient: Counters: 19
11/07/31 13:45:44 INFO mapred.JobClient:   Job Counters
11/07/31 13:45:44 INFO mapred.JobClient:     Launched reduce tasks=1
11/07/31 13:45:44 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=94857
11/07/31 13:45:44 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
11/07/31 13:45:44 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
11/07/31 13:45:44 INFO mapred.JobClient:     Launched map tasks=7
11/07/31 13:45:44 INFO mapred.JobClient:     Data-local map tasks=1
11/07/31 13:45:44 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=45085
11/07/31 13:45:44 INFO mapred.JobClient:     Failed map tasks=1
11/07/31 13:45:44 INFO mapred.JobClient:   FileSystemCounters
11/07/31 13:45:44 INFO mapred.JobClient:     HDFS_BYTES_READ=662018
11/07/31 13:45:44 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=330575
11/07/31 13:45:44 INFO mapred.JobClient:   File Input Format Counters
11/07/31 13:45:44 INFO mapred.JobClient:     Bytes Read=661664
11/07/31 13:45:44 INFO mapred.JobClient:   Map-Reduce Framework
11/07/31 13:45:44 INFO mapred.JobClient:     Map output materialized bytes=267146
11/07/31 13:45:44 INFO mapred.JobClient:     Combine output records=18043
11/07/31 13:45:44 INFO mapred.JobClient:     Map input records=12761
11/07/31 13:45:44 INFO mapred.JobClient:     Spilled Records=18043
11/07/31 13:45:44 INFO mapred.JobClient:     Map output bytes=1086767
11/07/31 13:45:44 INFO mapred.JobClient:     Combine input records=109848
11/07/31 13:45:44 INFO mapred.JobClient:     Map output records=109848
11/07/31 13:45:44 INFO mapred.JobClient:     SPLIT_RAW_BYTES=354

有人能告诉我什么是错的

java.io.FileNotFoundException: File does not exist: /user/hduser/gutenberg/gutenberg

Answer 1

好的，我发现问题出在哪里了，我必须添加一个* 到

bin/hadoop jar hadoop*examples*.jar wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output

教程中应该是这样的

bin/hadoop jar hadoop*examples*.jar wordcount /user/hduser/gutenberg/* /user/hduse/gutenberg-output3