【问题标题】:Running Hadoop Wordcount Job Error运行 Hadoop Wordcount 作业错误
【发布时间】:2015-07-24 03:21:59
【问题描述】:

我是使用 Hadoop 的新手,我目前正尝试在我的 ubuntu 桌面上的单节点集群上运行 wordcount。我正在尝试按照本指南运行 hadoop 作业:

http://javabeginnerstutorial.com/hadoop/your-first-hadoop-map-reduce-job/

目前我被困在第 3 步,当我尝试 mvn clean install 时,这是我收到的错误。我正在使用与步骤 3 中链接的完全相同的 pom.xml 文件。我是否应该编辑 pom.xml 文件中的某些内容以成功编译?或者链接可能不起作用?任何帮助都会很棒。谢谢。

hduser@venus:~/development/Hadoop_projects/word_count$ mvn clean install
[INFO] Scanning for projects...
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for com.impetus.code:hadoop-examples:jar:1.0
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing. @ line 38, column 12
[WARNING] 
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING] 
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING] 
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building hadoop-examples 1.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hadoop-examples ---
[INFO] 
[INFO] --- maven-resources-plugin:2.3:resources (default-resources) @ hadoop-examples ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /home/hduser/development/Hadoop_projects/word_count/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.0.2:compile (default-compile) @ hadoop-examples ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.3:testResources (default-testResources) @ hadoop-examples ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /home/hduser/development/Hadoop_projects/word_count/src/test/resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.0.2:testCompile (default-testCompile) @ hadoop-examples ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.10:test (default-test) @ hadoop-examples ---
[INFO] No tests to run.
[INFO] Surefire report directory: /home/hduser/development/Hadoop_projects/word_count/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------

Results :

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.874s
[INFO] Finished at: Wed Jul 15 00:34:04 PDT 2015
[INFO] Final Memory: 7M/240M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.10:test (default-test) on project hadoop-examples: Error creating properties files for forking; nested exception is java.io.IOException: No such file or directory -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

【问题讨论】:

  • 我不知道 pom.xml 文件应该做什么,在 POM 上的任何输入都会有所帮助。
  • pom.xml 告诉 maven 做什么:下载文件、构建工件(jars / classes)等。似乎很明显,当尝试使用 surefire 插件做某事时,maven 失败了。究竟发生了什么需要查看 pom.xml 的内容。
  • 我仍然无法编译。我在 pom 中没有看到任何关于 surfire 插件的内容。我正在使用这个 pom.xml:github.com/xamry/hadoop-examples/blob/master/pom.xml

标签: hadoop mapreduce pom.xml


【解决方案1】:

尝试从此http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-mapreduce-client-core 获取 pom 和 maven 依赖项 在添加这个 pom 文件之前尝试检查 hadoop 版本。

【讨论】:

    【解决方案2】:

    使用 -DskipTests 运行 maven 命令

    【讨论】:

    • 我仍然遇到构建失败,但似乎解决了 surefire-plugin 问题。我现在收到的错误是:[ERROR] Failed to execute goal org.apache.maven.plugins:maven-jar-plugin:2.2:jar (default-jar) on project WordCount: Error assemble JAR: Failed to create directory: / home/hduser/development/Hadoop_projects/word_count/target/maven-archiver -> [帮助1]
    猜你喜欢
    • 2016-09-29
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2012-05-05
    • 2023-04-04
    • 2017-06-01
    • 1970-01-01
    • 2023-03-08
    相关资源
    最近更新 更多