窗口函数错误至少1组必须只依赖于输入列

Question

我有一个带有窗口函数的公用表表达式并不断收到错误消息：

编译语句时出错：FAILED: SemanticException Failed to 将窗口调用分解为组。至少 1 组必须仅 取决于输入列。还要检查循环依赖。 潜在错误：org.apache.hadoop.hive.ql.parse.SemanticException： 第 82:6 行 CTE 定义中的列引用“gcr_amt”无效 pro_orders [选择 o.shopper_id 作为 pro_shopper_id, date_format(o.order_date, 'YYYYMM') as ym_order, sum(o.gcr_amt) as total_gcr, sum(o.product_pnl_new_renewal_name = 'New 购买'然后 o.gcr_amt end) as new_gcr, sum(o.gcr_amt) over （按 o.shopper_id 行在前 12 行和后 1 行之间进行分区） 作为 12months_direct_gcr 来自 dp_enterprise.uds_order o 内部连接 combine_shopper_level_data cs.pro_shopper_id = o.shopper_id 和 cs.year_month = date_format(o.order_date, 'YYYYMM') 其中 o.exclude_reason_desc 是 o.shopper_id, o.order_date 的 Null 组] 在第 83:5 行用作 po

我的 cte 是这样的：

pro_orders as (
  select  o.shopper_id as pro_shopper_id,
          date_format(o.order_date, 'YYYYMM') as ym_order,
          sum(o.gcr_amt) as total_gcr,
          sum(case when o.product_pnl_new_renewal_name = 'New Purchase' then o.gcr_amt end) as new_gcr,
          sum(o.gcr_amt) over (partition by o.shopper_id, cs.year_month order by cs.year_month desc rows between 12 preceding and 0 following) as 12months_direct_gcr
  from dp_enterprise.uds_order o
  right join combined_shopper_level_data cs on cs.pro_shopper_id = o.shopper_id and cs.year_month = date_format(o.order_date, 'YYYYMM')
  group by o.shopper_id, o.order_date
),

我不经常使用窗口函数，也许我的语法不正确。在英语中，我想要做的是获得度量“gcr”的 12 个月总数。

因此，在 year_month 201901 中，shopper_id 为 123abc 的一行，我想将前 11 个月加上 gcr 的当前行月份相加，总共 12 个月。不确定我的窗口功能是否正确设置？

引用的 year_month 的格式是 YYYYMM，例如201901.

我的窗口函数是否根据我的目标设置正确？

我该如何克服这个错误信息？

编辑： 仍然收到带有以下 CTE 的错误消息：

pro_orders as (
  select  o.shopper_id as pro_shopper_id,
          cs.year_month,
          sum(case when date_format(o.order_date, 'YYYYMM') = cs.year_month then o.gcr_amt else 0 end) as total_gcr,
          sum(case when date_format(o.order_date, 'YYYYMM') = cs.year_month and o.product_pnl_new_renewal_name = 'New Purchase' then o.gcr_amt else 0 end) as new_gcr,
          sum(sum(o.gcr_amt)) over  (partition by o.shopper_id 
                                order by cs.year_month desc 
                                rows between 12 preceding and 0 following) 
                                as 12months_direct_gcr
  from combined_shopper_level_data cs
  left join dp_enterprise.uds_order o on o.shopper_id = cs.pro_shopper_id
  where o.exclude_reason_desc is Null
  group by o.shopper_id, cs.year_month
),

导致类似的错误消息：

编译语句时出错：FAILED: SemanticException Failed to 将窗口调用分解为组。至少 1 组必须仅 取决于输入列。还要检查循环依赖。 潜在错误：org.apache.hadoop.hive.ql.parse.SemanticException： 第 83:10 行 CTE 定义中的列引用“gcr_amt”无效 pro_orders [选择 o.shopper_id 作为 pro_shopper_id，cs.year_month， sum(date_format(o.order_date, 'YYYYMM') = cs.year_month 然后的情况 o.gcr_amt else 0 end) as total_gcr, sum(case when date_format(o.order_date, 'YYYYMM') = cs.year_month 和 o.product_pnl_new_renewal_name = '新购买' 然后 o.gcr_amt else 0 end) as new_gcr, sum(sum(o.gcr_amt)) over (partition by o.shopper_id 按 cs.year_month desc 行排序，前 12 行和后 0 行） as 12months_direct_gcr from combined_shopper_level_data cs left join dp_enterprise.uds_order o on o.shopper_id = cs.pro_shopper_id where o.exclude_reason_desc 是 o.shopper_id, cs.year_month 的 Null 组] 在第 87:5 行用作 po

Answer 1

你有一个聚合查询，所以窗口函数看起来有点滑稽。基本思路是这样的：

sum(sum(o.gcr_amt)) over (partition by o.shopper_id, cs.year_month
                          order by cs.year_month desc
                          rows between 12 preceding and 0 following
                         ) as 12months_direct_gcr

这仍然行不通。首先，您在order by 和partition by 中有值。二、不在group by中。

假设每个月都有一个值，那么你可以使用：

sum(sum(o.gcr_amt)) over (partition by o.shopper_id
                          order by cs.year_month desc
                          rows between 12 preceding and 0 following
                         ) as 12months_direct_gcr

并在group by 中使用cs.year_month（这可能需要调整查询的其他部分。

为了可读性，我还建议您使用left join 而不是right join。对我（和大多数人）来说，在认知上说“保留我刚刚阅读的第一个表中的所有行”而不是“保留我将要阅读的某个表中的所有行在@的末尾987654330@子句”。

编辑：

我认为完整的查询是：

with pro_orders as (
      select o.shopper_id as pro_shopper_id,
             cs.year_month,
             sum(coalesce(o.gcr_amt, 0)) as total_gcr,
             sum(case when o.product_pnl_new_renewal_name = 'New Purchase' then o.gcr_amt else 0 end) as new_gcr,
             sum(sum(o.gcr_amt)) over (partition by o.shopper_id 
                                       order by cs.year_month desc 
                                       rows between 12 preceding and 0 following
                                      ) as 12months_direct_gcr
      from combined_shopper_level_data cs left join
           dp_enterprise.uds_order o
           on o.shopper_id = cs.pro_shopper_id and
              date_format(o.order_date, 'YYYYMM') = cs.year_month and
              o.exclude_reason_desc is Null
      group by o.shopper_id, cs.year_month
     ),

Hive 中可能存在对在聚合查询中使用窗口函数的限制（这让我有点吃惊，因为这些是分开处理的）。我找不到对此的具体参考。如果是这样，只需使用子查询：

with pro_orders as (
      select pro_shopper_id, year_month, total_gcr, new_gcr
             sum(sum(total_gcr_amt)) over (partition by pro_shopper_id
                                           order by year_month desc 
                                           rows between 12 preceding and 0 following
                                          ) as 12months_direct_gcr
      from (select o.shopper_id as pro_shopper_id,
                   cs.year_month,
                   sum(coalesce(o.gcr_amt, 0)) as total_gcr,
                   sum(case when o.product_pnl_new_renewal_name = 'New Purchase' then o.gcr_amt else 0 end) as new_gcr,
          from combined_shopper_level_data cs left join
               dp_enterprise.uds_order o
               on o.shopper_id = cs.pro_shopper_id and
                  date_format(o.order_date, 'YYYYMM') = cs.year_month and
                  o.exclude_reason_desc is Null
          group by o.shopper_id, cs.year_month
         ) ps
     ),