Hadoop：重定向到作业历史服务器时出现 NullPointerException

Question

我有一个 Hadoop 集群 (HDP 2.1)。一切都工作了很长时间，但突然作业开始返回以下经常性错误：

16/10/13 16:21:11 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
16/10/13 16:21:12 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
16/10/13 16:21:12 INFO impl.TimelineClientImpl: Timeline service address: http://dev-fiwr-bignode-12.hi.inet:8188/ws/v1/timeline/
16/10/13 16:21:13 INFO client.RMProxy: Connecting to ResourceManager at dev-fiwr-bignode-12.hi.inet/10.95.76.79:8050
16/10/13 16:21:13 INFO input.FileInputFormat: Total input paths to process : 2
16/10/13 16:21:13 INFO mapreduce.JobSubmitter: number of splits:2
16/10/13 16:21:13 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
16/10/13 16:21:14 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1476366871137_0003
16/10/13 16:21:14 INFO impl.YarnClientImpl: Submitted application application_1476366871137_0003
16/10/13 16:21:14 INFO mapreduce.Job: The url to track the job: http://dev-fiwr-bignode-12.hi.inet:8088/proxy/application_1476366871137_0003/
16/10/13 16:21:14 INFO mapreduce.Job: Running job: job_1476366871137_0003
16/10/13 16:21:19 INFO mapreduce.Job: Job job_1476366871137_0003 running in uber mode : false
16/10/13 16:21:19 INFO mapreduce.Job:  map 0% reduce 0%
16/10/13 16:21:23 INFO mapreduce.Job:  map 50% reduce 0%
16/10/13 16:21:24 INFO mapreduce.Job:  map 100% reduce 0%
16/10/13 16:21:28 INFO mapreduce.Job:  map 100% reduce 100%\
6/10/13 16:21:29 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
16/10/13 16:21:29 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
16/10/13 16:21:29 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
Exception in thread \"main\" java.io.IOException:
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskAttemptCompletionEvents(HistoryClientService.java:277)
org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBServiceImpl.java:173)
org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:283)
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAs(Subject.java:415)
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:334)
org.apache.hadoop.mapred.ClientServiceDelegate.getTaskCompletionEvents(ClientServiceDelegate.java:386)
org.apache.hadoop.mapred.YARNRunner.getTaskCompletionEvents(YARNRunner.java:539)
org.apache.hadoop.mapreduce.Job$5.run(Job.java:668)
org.apache.hadoop.mapreduce.Job$5.run(Job.java:665)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAs(Subject.java:415)
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
org.apache.hadoop.mapreduce.Job.getTaskCompletionEvents(Job.java:665)
org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1366)
org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1306)
dijkstra.adjacencylist.AdjacencyListDriver.jobRun(AdjacencyListDriver.java:53)
dijkstra.adjacencylist.AdjacencyListDriver.run(AdjacencyListDriver.java:31)
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
dijkstra.launch.LaunchClass.launchAdjMatrix(LaunchClass.java:226)
dijkstra.launch.LaunchClass.main(LaunchClass.java:199)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by:
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskAttemptCompletionEvents(HistoryClientService.java:277)
...

谷歌一下，我看到了这些问题：

• https://issues.apache.org/jira/browse/MAPREDUCE-5703
• https://issues.apache.org/jira/browse/MAPREDUCE-5547

它们似乎是相关的。然而，为什么集群到现在还可以正常运行呢？配置没有任何变化，集群没有处于安全模式，HDFS 空间使用率在 0.03% 左右……有什么线索吗？如果这与上述问题有关，有什么解决方法吗？

非常感谢，我会继续关注您的答案或其他信息要求。

Answer 1

根据堆栈跟踪判断，您的问题类似于 5703，并且如该错误中所述：

“GetTaskAttemptCompletionEventsResponse() 方法通过调用 verifyAndGetJob() 获取了一个 Job，但它从未检查过 job 是否为空，这是导致此问题的根本原因。”

使用作业 ID 进行作业查找，但未找到作业。

在该错误中，它列出了一种情况，其中向作业历史服务器 (JHS) 查询有关已完成作业但 JHS 未能接收到该作业的信息。

似乎存在关于作业终止和作业历史上传的未解决问题，当作业历史上传失败时允许发生此异常。在错误中，问题是通过在历史记录上传完成之前重新启动写入历史记录的节点触发的，或者由该节点没有可写入历史记录的好节点触发。

不幸的是，在您的情况下，这里没有任何其他内容可以帮助确定导致历史记录上传失败的原因，但这似乎是问题的根本原因。您的作业历史服务器没有成功完成作业的记录。