使用 `where 1=1` 时找不到自定义 Hive InputFormat答案

【问题标题】：Unable to find custom Hive InputFormat when using `where 1=1`使用 `where 1=1` 时找不到自定义 Hive InputFormat
【发布时间】：2014-11-09 14:20:35
【问题描述】：

我正在使用 Hive，但在使用自定义 InputFormat 执行查询时遇到异常。

当我使用查询 select * from micmiu_blog; 时，Hive 可以正常工作，但如果我使用 select * from micmiu_blog where 1=1;，框架似乎找不到我的自定义 InputFormat 类。

我已将 JAR 文件放入“hive/lib”、“hadoop/lib”，并将“hadoop/lib”放入 CLASSPATH。这是日志：

hive> select * from micmiu_blog where 1=1;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1415530028127_0004, Tracking URL = http:/ /hadoop01-master:8088/proxy/application_1415530028127_0004/
Kill Command = /home/hduser/hadoop-2.2.0/bin/hadoop job  -kill job_1415530028127_0004
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2014-11-09 19:53:32,996 Stage-1 map = 0%,  reduce = 0%
2014-11-09 19:53:52,010 Stage-1 map = 100%,  reduce = 0%
Ended Job = job_1415530028127_0004 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1415530028127_0004_m_000000 (and more) from job job_1415530028127_0004

Task with the most failures(4): 
-----
Task ID:
  task_1415530028127_0004_m_000000

URL:
  http://hadoop01-master:8088/taskdetails.jsp?jobid=job_1415530028127_0004&tipid=task_1415530028127_0004_m_000000
-----
Diagnostic Messages for this Task:
Error: java.io.IOException: cannot find class hiveinput.MyDemoInputFormat
        at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:564)
        at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:167)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:408)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)


FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched: 
Job 0: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

【问题讨论】：

标签： mapreduce hive hadoop2

【解决方案1】：

我刚才遇到了这个问题。您应该使用 hive cli 将 JAR 文件添加到 CLASSPATH。你可以做到：

hive&gt; add jar /usr/lib/xxx.jar;

【讨论】：

谢谢你的回答，我会试试你说的！最后我通过将JAR文件放到“$HADOOP_HOME/share/hadoop/yarn”来解决这个问题，但是很麻烦。