hadoop map reduce中的文件系统访问异常答案

【问题标题】：File system access exception in hadoop map reducehadoop map reduce中的文件系统访问异常
【发布时间】：2015-08-23 07:11:25
【问题描述】：

我已经在 Ubuntu 14.04 上配置了 Hadoop 2.6.0。我最初正在运行 wordcount map reduce 程序以了解 map reduce 作业。我在访问文件系统时遇到了一些问题。我在/opt/hadoop2.6.0 中有 Hadoop 主目录。

驱动程序

    Configuration conf = new Configuration();
    Job job = Job.getInstance(conf, "word count");
    job.setJarByClass(WordCount.class);
    job.setMapperClass(TokenizerMapper.class);
    job.setCombinerClass(IntSumReducer.class);
    job.setReducerClass(IntSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    // configuration should contain reference to your namenode
     FileSystem hdfs =FileSystem.get(new Configuration()); 
    Path workingDir=hdfs.getWorkingDirectory();

    Path newFolderPath= new Path("/output");

    newFolderPath=Path.mergePaths(workingDir, newFolderPath);

    if(hdfs.exists(newFolderPath))

    {

        hdfs.delete(newFolderPath, true); //Delete existing Directory

    }
    hdfs.mkdirs(newFolderPath); 
    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job,newFolderPath );
    System.exit(job.waitForCompletion(true) ? 0 : 1); //line no. 76
    // job.submit();

核心站点.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name> 
        <value>/app/hadoop/tmp</value> 
    </property>
</configuration>

hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/opt/hadoop-2.6.0/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/opt/hadoop-2.6.0/dfs/data</value>
    </property>
    <property>
        <name>dfs.permissions.enabled</name>
        <value>false</value>
    </property>
    <property>
        <name>dfs.http.address</name>
        <value>localhost:50070</value>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
</configuration>

yarn-site.xml

<configuration>

  <!-- Site specific YARN configuration properties --> <property>
        <name>yarn.nodemanager.aux-services</name>  
        <value>mapreduce_shuffle</value> </property> <property>      
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>   
          <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property>
 <property>
         <name>yarn.application.classpath</name>
         <value>
              %HADOOP_HOME%\etc\hadoop,
              %HADOOP_HOME%\share\hadoop\common\*,
              %HADOOP_HOME%\share\hadoop\common\lib\*,
              %HADOOP_HOME%\share\hadoop\hdfs\*,
              %HADOOP_HOME%\share\hadoop\hdfs\lib\*,
              %HADOOP_HOME%\share\hadoop\mapreduce\*,
              %HADOOP_HOME%\share\hadoop\mapreduce\lib\*,
              %HADOOP_HOME%\share\hadoop\yarn\*,
              %HADOOP_HOME%\share\hadoop\yarn\lib\*
         </value> 
    </property>

运行map reduce jar：

 hadoop jar /home/ifs-admin/wordcount.jar  WordCount /user/ifs/input

执行异常：

    15/08/23 12:12:25 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
    15/08/23 12:12:25 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
    Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory
    hdfs://localhost:9000/user/ifs-admin/output already exists
    at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
    at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:562)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
    at WordCount.main(WordCount.java:76)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

如果我删除输出目录，它会显示以下错误：

   Exception in thread "main" ENOENT: No such file or directory
    at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
    at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
    at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:652)
    at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:490)
    at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:599)
    at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:182)
    at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:390)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:483)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
    at WordCount.main(WordCount.java:68)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

如何解决这个问题？

【问题讨论】：

标签： hadoop mapreduce hdfs

【解决方案1】：

试试这个代码注意-不要创建目录，hadoop会自动创建

FileSystem hdfs = FileSystem.get(new URI("hdfs://localhost:9000"),
            conf); 
    Path workingDir=hdfs.getWorkingDirectory();

    Path newFolderPath= new Path("/output");
    newFolderPath=Path.mergePaths(workingDir, newFolderPath);
    if(hdfs.exists(newFolderPath))

    {
        hdfs.delete(newFolderPath); //Delete existing Directory

    }
    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job,newFolderPath );
    System.exit(job.waitForCompletion(true) ? 0 : 1); //line no. 76
    // job.submit();

【讨论】：

嗨 Krish，这给了我一个例外：线程“主”org.apache.hadoop.mapred.FileAlreadyExistsException 中的异常：输出目录 hdfs://localhost:9000/user/ifs-admin/output已经存在。有配置问题吗？
在 windows 中它运行良好。 Ubuntu中是否有任何权限问题？我试过 chown -R 并将目录更改为 /usr/local/hadoop。请帮忙
你有没有从程序中删除这段代码---hdfs.mkdirs(newFolderPath);
给定的代码在 linux 上运行良好。尝试其他目录
hadoop 无法创建目录。它显示错误，线程“主”ENOENT中的异常：没有这样的文件或目录。如果，我使用 -mkdir -p 创建目录。它显示异常，线程“main”中的异常 org.apache.hadoop.mapred.FileAlreadyExistsException：输出目录 hdfs://localhost:9000/user/ifs-admin/output 已经存在。我被击倒了。任何帮助，我可以知道你的单节点设置吗？

【解决方案2】：

当我们在 /opt 文件夹下设置 temp.dir 或名称节点目录或数据节点目录时，Hadoop 无法在 hdfs 中创建目录。访问： https://unix.stackexchange.com/questions/11544/what-is-the-difference-between-opt-and-usr-local.

我已将 core-site.xml 中的 hadoop.tmp.dir 更改为 /usr/local/hadoop/dfs/data。

【讨论】：