示例 map-reduce oozie 程序在 CDH 4.5 上不起作用

Question

我从一段时间以来一直在我的 Mac 上使用 Hadoop (CDH 4.5)，并定期运行 map reduce 作业。我最近按照以下说明安装了 oozie（同样是 CDH4.5）： http://archive.cloudera.com/cdh4/cdh/4/oozie-3.3.2-cdh4.5.0/DG_QuickStart.html，并尝试运行提供的示例程序。但是，它总是失败并出现以下错误。看起来工作流程根本没有运行。 Job 信息中的 Console URL 字段也是空的。

有人可以帮忙吗？

Oozie Job 日志的相关sn-p如下。

2014-06-10 17:27:18,414  INFO ActionStartXCommand:539 - USER[userXXX] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-140610172702069-oozie-usrX-W] ACTION[0000000-140610172702069-oozie-usrX-W@:start:] Start action [0000000-140610172702069-oozie-usrX-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2014-06-10 17:27:18,417  WARN ActionStartXCommand:542 - USER[userXXX] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-140610172702069-oozie-usrX-W] ACTION[0000000-140610172702069-oozie-usrX-W@:start:] [***0000000-140610172702069-oozie-usrX-W@:start:***]Action status=DONE
2014-06-10 17:27:18,417  WARN ActionStartXCommand:542 - USER[userXXX] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-140610172702069-oozie-usrX-W] ACTION[0000000-140610172702069-oozie-usrX-W@:start:] [***0000000-140610172702069-oozie-usrX-W@:start:***]Action updated in DB!
2014-06-10 17:27:18,576  INFO ActionStartXCommand:539 - USER[userXXX] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-140610172702069-oozie-usrX-W] ACTION[0000000-140610172702069-oozie-usrX-W@mr-node] Start action [0000000-140610172702069-oozie-usrX-W@mr-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2014-06-10 17:27:19,188  WARN MapReduceActionExecutor:542 - USER[userXXX] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-140610172702069-oozie-usrX-W] ACTION[0000000-140610172702069-oozie-usrX-W@mr-node] credentials is null for the action
2014-06-10 17:27:19,423  WARN ActionStartXCommand:542 - USER[userXXX] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-140610172702069-oozie-usrX-W] ACTION[0000000-140610172702069-oozie-usrX-W@mr-node] Error starting action [mr-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Unknown rpc kind RPC_WRITABLE]
org.apache.oozie.action.ActionExecutorException: JA009: Unknown rpc kind RPC_WRITABLE
at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:418)
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:392)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:773)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:927)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59)
at org.apache.oozie.command.XCommand.call(XCommand.java:277)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): Unknown rpc kind RPC_WRITABLE
at org.apache.hadoop.ipc.Client.call(Client.java:1238)
at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225)
at org.apache.hadoop.mapred.$Proxy30.getDelegationToken(Unknown Source)
at org.apache.hadoop.mapred.JobClient.getDelegationToken(JobClient.java:2125)
at org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:372)
at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:970)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:723)
... 10 more

2014-06-10 17:27:19,426  INFO ActionStartXCommand:539 - USER[userXXX] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-140610172702069-oozie-usrX-W] ACTION[0000000-140610172702069-oozie-usrX-W@mr-node] Next Retry, Attempt Number [1] in [60,000] milliseconds
2014-06-10 17:28:19,468  INFO ActionStartXCommand:539 - USER[userXXX] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-140610172702069-oozie-usrX-W] ACTION[0000000-140610172702069-oozie-usrX-W@mr-node] Start action [0000000-140610172702069-oozie-usrX-W@mr-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]

Answer 1

您是否使用 MapReduce 或 YARN 设置了 hadoop？

Configuring which Hadoop Version to Use
The Oozie client does not interact directly with Hadoop MapReduce, and so it does not require any MapReduce configuration.

The Oozie server can work with either MRv1 or YARN. It cannot work with both simultaneously.

The MapReduce version Oozie server works with is determined by the CATALINA_BASE variable in the /etc/oozie/conf/oozie-env.sh file. By default, CATALINA_BASE is set to /usr/lib/oozie/oozie-server-0.20. This setting configures the Oozie server to work with MRv1.

To configure the Oozie server to work with YARN instead, set CATALINA_BASE to /usr/lib/oozie/oozie-server.

由于您有 cdh4 发行版，请点击此链接 http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_17.html 安装 oozie

如果你有 YARN，在你的 job.properties 中更改 jobtracker 的端口

nameNode=hdfs://localhost:8020 
jobTracker=localhost:8021

使用资源管理器的端口：8032，而不是端口 8021 (jobTracker)

jobTracker=localhost:8032

https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/MlsqeUdMpgA