【问题标题】:Hive windowing in sub-query and outer query on DATEDATE 子查询和外部查询中的 Hive 窗口化
【发布时间】:2019-04-18 21:04:14
【问题描述】:
CREATE TABLE big_hive_table(
`partner` string,
start_date date,
end_date date,
`category` string,
`category2` string);

insert into big_hive_table values ('S1','2018-01-01','2018-03-31','c1','M');
insert into big_hive_table values ('S1','2017-12-01','2018-01-31','c1','M');
insert into big_hive_table values ('S1','2017-01-01','2017-11-30','c1','M');
insert into big_hive_table values ('S1','2018-02-01','2018-04-30','c1','M');
insert into big_hive_table values ('S1','2018-02-01','2018-04-30','c1','L');
insert into big_hive_table values ('S2','2018-02-01','2018-04-30','c1','S');
insert into big_hive_table values ('S3','2018-02-01','2018-04-30','c2','S');
insert into big_hive_table values ('S3','2018-01-01','2018-03-31','c2','S');
insert into big_hive_table values ('S3','2017-12-01','2018-01-31','c2','S');

问题:如果存在重叠时间段,则获取组(合作伙伴、categorycategory2)的最早开始日期和最晚结束日期

expected result: 

S1  01/12/2017  30/04/2018  c1  M
S1  01/01/2017  30/11/2017  c1  M
S1  01/02/2018  30/04/2018  c1  L
S2  01/02/2018  30/04/2018  c1  S
S3  01/12/2017  30/04/2018  c2  S

我的查询

SELECT DISTINCT partner, 
                category, 
                category2, 
                First_value(start_date) OVER (partition BY partner, category, category2 ORDER BY start_date) period_start,
                last_value(end_date) OVER (partition BY partner, category, category2 ORDER BY start_date rows BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED following) period_end
from (select pps.*, sum(start_new_period) over (partition BY partner, category, category2)
    FROM ( select partner, 
            start_date,
            end_date, 
            category, 
            category2, 
            lag(end_date) over (partition by partner, category, category2 order by start_date) previous_period_end
            , case 
                when start_date > lag(end_date) over (partition by partner, category, category2 order by start_date) 
                then 1 
                else 0 
                end start_new_period
            from big_hive_table
            where start_date is not null and end_date is not null) pps
)

目前,当我运行 2 个内部查询(来自 select pps.*)或整个查询时,出现以下错误:

Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.parse.SemanticException:Failed to breakup Windowing invocations into Groups. At least 1 group must only depend on input columns. Also check for circular dependencies.
Underlying error: Primitve type DATE not supported in Value Boundary expression

谁能建议我缺少什么。感谢您的帮助。

【问题讨论】:

  • 抱歉,我错过了最外层查询的别名。如果您正在测试,请添加别名。

标签: hive boundary


【解决方案1】:

只需在您的 first_value 窗口函数中添加 rows BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED following 并尝试再次运行。

更改您的查询

来自

First_value(start_date) OVER (partition BY partner, category, category2 
ORDER BY start_date) period_start

收件人

First_value(start_date) OVER (partition BY partner, category, category2 ORDER BY
 start_date rows BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED following) period_start

Jira 关于原始类型支持并在 Hive.2.1.0 中得到修复

【讨论】:

  • 感谢@Shu,但没有帮助。我相信这个问题与 HIVE-13973 无关,因为我可以在没有问题的情况下运行最里面的查询,并在日期上加窗。
  • @DE2018,如果我们使用 lead,lag 函数,那么我们不需要在子句之间指定行但是如果我们使用 first_value,last_valuedate datatype(Hive < 2.0) 上,我们需要在子句之间指定行。
  • @DE2018,有关此的更多详细信息:stackoverflow.com/questions/52979610/…
猜你喜欢
  • 1970-01-01
  • 1970-01-01
  • 2018-04-20
  • 1970-01-01
  • 1970-01-01
  • 2016-05-19
  • 1970-01-01
  • 2012-11-09
  • 1970-01-01
相关资源
最近更新 更多