Presto / AWS Athena 查询，历史化表（聚合中的最后一个值）答案

【问题标题】：Presto / AWS Athena query, historicized table (last value in aggregation)Presto / AWS Athena 查询，历史化表（聚合中的最后一个值）
【发布时间】：2023-03-03 11:30:01
【问题描述】：

我有一个分为静态部分和历史部分的表格。我必须创建一个按一系列维度（包括年份和月份）分组的查询，并进行一些聚合。我需要投影的值之一是匹配给定年/月对的历史表的最后一个元组的值。

历史表有validity_date_start和validity_date_end，如果是最新的则为NULL。

这是我到目前为止所做的查询（使用临时表以便于复制）：

SELECT
  time.year,
  time.month,
  t1.name,
  FIRST_VALUE(t2.value1) OVER(ORDER BY t2.validity_date_start DESC) AS value, -- take the last valid t2 part for the month
  (CASE WHEN t1.id = 1 AND time.date >= timestamp '2020-07-01 00:00:00' THEN 27
     ELSE CASE WHEN t1.id = 1 AND time.date >= timestamp '2020-03-01 00:00:00' THEN 1
       ELSE CASE WHEN t1.id = 2 AND time.date >= timestamp '2020-05-01 00:00:00' THEN 42 END
     END
   END) AS expected_value
FROM
  (SELECT year(ts.date) year, month(ts.date) month, ts.date FROM (
    (VALUES (SEQUENCE(date '2020-01-01', current_date, INTERVAL '1' MONTH))) AS ts(ts_array)
    CROSS JOIN UNNEST(ts_array) AS ts(date)
  ) GROUP BY ts.date) time

  CROSS JOIN (VALUES (1, 'Hal'), (2, 'John'), (3, 'Jack')) AS t1 (id, name)
  
  LEFT JOIN (VALUES
    (1, 1, timestamp '2020-01-03 10:22:33', timestamp '2020-07-03 23:59:59'),
    (1, 27, timestamp '2020-07-04 00:00:00', NULL),
    (2, 42, timestamp '2020-05-29 10:22:31', NULL)
  ) AS t2 (id, value1, validity_date_start, validity_date_end)
    ON t1.id = t2.id
    AND t2.validity_date_start <= (CAST(time.date as timestamp) + interval '1' month - interval '1' second)
    AND (t2.validity_date_end IS NULL OR t2.validity_date_end >= (CAST(time.date as timestamp) + interval '1' month - interval '1' second)) -- last_day_of_month (Athena doesn't have the fn)
  
  GROUP BY time.date, time.year, time.month, t1.id, t1.name, t2.value1, t2.validity_date_start
  ORDER BY time.year, time.month, t1.id

value 和 expected_value 应该匹配，但它们不匹配（值始终为空）。我显然误解了FIRST_VALUE(...) OVER(...) 的工作原理。

你能帮帮我吗？

非常感谢！

【问题讨论】：

如果我运行查询，value 字段始终为 27。如果您可以将预期结果发布为表格，将会很有帮助。
您好@PhilippJohannis 感谢您的关注。预期结果显示在“expected_value”字段中。

标签： aggregation presto amazon-athena

【解决方案1】：

我最终发现我在这里做错了什么。

documents 中写着：

分区规范，将输入行分成不同的分区。这类似于 GROUP BY 子句如何将行分隔为聚合函数的不同组

这让我想到，如果我已经有一个GROUP BY 声明，那么这是没用的。它不是：通常如果你想获得给定组的数据，你也必须在PARTITION BY 语句中指定它（或者更好的是你在SELECT 部分中投影的尺寸）。

SELECT
  time.year,
  time.month,
  t1.name,
  FIRST_VALUE(t2.value1) OVER(PARTITION BY (time.year, time.month, t1.name) ORDER BY t2.validity_date_start DESC) AS value, -- take the last valid t2 part for the month
  (CASE WHEN time.date >= timestamp '2020-07-01 00:00:00' AND t1.id = 1 THEN 27
     ELSE CASE WHEN time.date >= timestamp '2020-05-01 00:00:00' AND t1.id = 2 THEN 42
       ELSE CASE WHEN time.date >= timestamp '2020-03-01 00:00:00' AND t1.id = 1 THEN 1 END
     END
   END) AS expected_value
FROM
  (SELECT year(ts.date) year, month(ts.date) month, ts.date FROM (
    (VALUES (SEQUENCE(date '2020-01-01', current_date, INTERVAL '1' MONTH))) AS ts(ts_array)
    CROSS JOIN UNNEST(ts_array) AS ts(date)
  ) GROUP BY ts.date) time

  CROSS JOIN (VALUES (1, 'Hal'), (2, 'John'), (3, 'Jack')) AS t1 (id, name)
  
  LEFT JOIN (VALUES
    (1, 1, timestamp '2020-03-01 10:22:33', timestamp '2020-07-03 23:59:59'),
    (1, 27, timestamp '2020-07-04 00:00:00', NULL),
    (2, 42, timestamp '2020-05-29 10:22:31', NULL)
  ) AS t2 (id, value1, validity_date_start, validity_date_end)
    ON t1.id = t2.id
    AND t2.validity_date_start <= (CAST(time.date as timestamp) + interval '1' month - interval '1' second)
    AND (t2.validity_date_end IS NULL OR t2.validity_date_end >= (CAST(time.date as timestamp) + interval '1' month - interval '1' second)) -- last_day_of_month (Athena doesn't have the fn)
  
  GROUP BY time.date, time.year, time.month, t1.id, t1.name, t2.value1, t2.validity_date_start
  ORDER BY time.year, time.month, t1.id

【讨论】：