Hive 动态分区无法将分区写入最终子目录答案

【问题标题】：Hive Dyanamic Partitioning failing to write partitions to final subdirectoriesHive 动态分区无法将分区写入最终子目录
【发布时间】：2020-03-11 04:38:12
【问题描述】：

我还是个蜂巢新手

我在下表中加载了数据

CREATE TABLE table1 (
  col1 string,
  col2 string,
  col3 float,
  col4 float)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';

我想通过 col1 字符串对其进行分区

CREATE TABLE table1_partition (
  col2 string,
  col3 float,
  col4 float)
PARTITIONED BY (col1 string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';

运行以下命令时出现错误

INSERT INTO TABLE table1_partition PARTITION (col1)
SELECT col2, col3, col4, 
col1 FROM table1;

错误

Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Job running in-process (local Hadoop)
2020-03-11 05:23:22,584 Stage-1 map = 0%,  reduce = 0%
2020-03-11 05:24:22,944 Stage-1 map = 0%,  reduce = 0%
2020-03-11 05:25:23,122 Stage-1 map = 0%,  reduce = 0%
Ended Job = job_local441210382_0001 with errors
Error during job, obtaining debugging information...
Job Tracking URL: http://localhost:8080/
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched: 
Stage-Stage-1:  HDFS Read: 1858506752 HDFS Write: 1477771205 FAIL
Total MapReduce CPU Time Spent: 0 msec

我查看了日志文件，似乎应该写入的数据节点被删除了：

2020-03-11 05:47:01,916 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool BP-433273321-127.0.1.1-1583889697525 Total blocks: 14, missing metadata files:0, missing block files:0, missing blocks in memory:0, mismatched blocks:0

我还根据日志查看了 /tmp/hadoop-myuser/dfs/data/current/.../finalised/，subdir0 - subdir5 有文件，其中一些是空的。

请帮助我理解这个问题。

【问题讨论】：

尝试查找失败的容器日志，它会提供更多信息。

标签： hadoop hive hiveql database-partitioning

【解决方案1】：

将hive.exec.dynamic.partition.mode 设置为nostrict，例如

set hive.exec.dynamic.partition.mode=nostrict

在执行查询之前。默认情况下，hive 在严格模式下工作。来自 Hive 的手册

在严格模式下，用户必须至少指定一个静态分区，以防用户不小心覆盖了所有分区，在非严格模式下，所有分区都允许是动态的

【讨论】：