使用 SQL / BigQuery 在移动窗口框架中查找第二个最大值

Question

我需要计算过去 120 分钟帧的移动窗口中的第二个最大值，就像我计算过去 120 分钟在 device_id 上分区的移动帧中的最大值一样。

MAX(Temperature) OVER(PARTITION BY device_type ORDER BY event_ts_seconds RANGE BETWEEN 7200 PRECEDING AND CURRENT ROW) AS Max_Temp_in_120frame

同样，我想计算第二个最高温度。

我已经尝试了以下 2 个版本的 NTH_VALUE，但它没有按预期工作。

NTH_VALUE(Temperature,2) OVER(PARTITION BY device_id ORDER BY event_ts_seconds RANGE BETWEEN 7200 PRECEDING AND CURRENT ROW) AS SecondMax_Temp_in_120frame

NTH_VALUE(Temperature,2) OVER(PARTITION BY device_id ORDER BY Temperature DESC RANGE BETWEEN 7200 PRECEDING AND CURRENT ROW) AS SecondMax_Temp_in_120frame

非常感谢您的帮助

Answer 1

考虑以下方法

SELECT * EXCEPT(Array_Temp_in_120frame, Two_Max_Temp_in_120frame),
  Two_Max_Temp_in_120frame[SAFE_OFFSET(0)] Max_Temp_in_120frame, 
  Two_Max_Temp_in_120frame[SAFE_OFFSET(1)] SecondMax_Temp_in_120frame 
FROM (
  SELECT *, ARRAY(
      SELECT Temperature FROM t.Array_Temp_in_120frame Temperature
      ORDER BY Temperature DESC LIMIT 2
    ) Two_Max_Temp_in_120frame
  FROM (
    SELECT *, ARRAY_AGG(Temperature) OVER(
        PARTITION BY device_type ORDER BY event_ts_seconds 
        RANGE BETWEEN 7200 PRECEDING AND CURRENT ROW
      ) AS Array_Temp_in_120frame
    FROM your_table
  ) t
)

Answer 2

简答

你可以这样做：

WITH sample AS (
  SELECT 
    ts,
    ROUND(10*RAND(), 1) as value
  FROM UNNEST (GENERATE_ARRAY(1, 10)) ts
)
, sample_with_moving_array AS (
    SELECT
        ts,
        value,
        ARRAY_AGG(value) OVER(ORDER BY ts RANGE BETWEEN 3 PRECEDING AND CURRENT ROW) AS moving_array,
    FROM sample
)
, sample_with_sorted_moving_array AS (
    SELECT
        ts,
        value,
        moving_array,
        ARRAY(SELECT value FROM UNNEST(moving_array) as value ORDER BY value DESC) as sorted_moving_array
    FROM sample_with_moving_array
)
SELECT
    ts,
    value,
    sorted_moving_array[SAFE_ORDINAL(1)] AS max_value_in_3_frame_alternate_version,
    sorted_moving_array[SAFE_ORDINAL(2)] AS second_max_value_in_3_frame,
FROM sample_with_sorted_moving_array

说明

首先，NTH_VALUE 不起作用，因为它是窗口函数FIRST_VALUE 的泛化，而不是MAX。

在 BigQuery 中，您可以分三步计算第二个最大值。 首先，使用ARRAY_AGG 计算值的移动数组：

ARRAY_AGG(value) OVER(ORDER BY ts RANGE BETWEEN 3 PRECEDING AND CURRENT ROW) AS moving_array

这会给你这样的东西：

ts	value	moving_array_as_string
1	0.6	[0.6]
2	5.4	[0.6, 5.4]
3	5.4	[0.6, 5.4, 5.4]
4	5.9	[0.6, 5.4, 5.4, 5.9]
5	4.6	[5.4, 5.4, 5.9, 4.6]
6	6	[5.4, 5.9, 4.6, 6]
7	8.7	[5.9, 4.6, 6, 8.7]
8	3.5	[4.6, 6, 8.7, 3.5]
9	4	[6, 8.7, 3.5, 4]
10	0.7	[8.7, 3.5, 4, 0.7]

完成后，使用 UNNEST 对数组进行排序，如下所示：

ARRAY(SELECT value FROM UNNEST(moving_array) as value ORDER BY value DESC) as sorted_moving_array

这会给你这样的东西（我没有显示排名，但你可以看到每个数组中的值已经排序）：

ts	value	sorted_moving_array_as_string
1	0.6	[0.6]
2	5.4	[5.4, 0.6]
3	5.4	[5.4, 5.4, 0.6]
4	5.9	[5.9, 5.4, 5.4, 0.6]
5	4.6	[5.9, 5.4, 5.4, 4.6]
6	6	[6, 5.9, 5.4, 4.6]
7	8.7	[8.7, 6, 5.9, 4.6]
8	3.5	[8.7, 6, 4.6, 3.5]
9	4	[8.7, 6, 4, 3.5]
10	0.7	[8.7, 4, 3.5, 0.7]

最后，你可以像这样得到sorted_moving_array的第二个元素：

sorted_moving_array[SAFE_ORDINAL(2)] second_max_value_in_3_frame

ORDINAL 表示您认为数组的索引从 1 而不是 0 开始。我发现使用ORDINAL(2) 而不是OFFSET(1) 获取第二个元素更具可读性。 SAFE_ 表示如果索引超出数组的范围，您希望它返回 NULL，而不是抛出错误。

最终查询如下所示：

WITH sample AS (
  SELECT 
    ts,
    ROUND(10*RAND(), 1) as value
  FROM UNNEST (GENERATE_ARRAY(1, 10)) ts
)
, sample_with_moving_array AS (
    SELECT
        ts,
        value,
        ARRAY_AGG(value) OVER(ORDER BY ts RANGE BETWEEN 3 PRECEDING AND CURRENT ROW) AS moving_array,
    FROM sample
)
, sample_with_sorted_moving_array AS (
    SELECT
        ts,
        value,
        moving_array,
        ARRAY(SELECT value FROM UNNEST(moving_array) as value ORDER BY value DESC) as sorted_moving_array
    FROM sample_with_moving_array
)
SELECT
    ts,
    value,
    MAX(value) OVER(ORDER BY ts RANGE BETWEEN 3 PRECEDING AND CURRENT ROW) AS max_value_in_3_frame,
    sorted_moving_array[SAFE_ORDINAL(1)] AS max_value_in_3_frame_alternate_version,
    sorted_moving_array[SAFE_ORDINAL(2)] AS second_max_value_in_3_frame,
    CONCAT("[", ARRAY_TO_STRING(ARRAY(SELECT CAST(value as STRING) FROM UNNEST(moving_array) value), ", "), "]") AS moving_array_as_string,
    CONCAT("[", ARRAY_TO_STRING(ARRAY(SELECT CAST(value as STRING) FROM UNNEST(sorted_moving_array) value), ", "), "]") AS sorted_moving_array_as_string
FROM sample_with_sorted_moving_array

返回：

ts	value	max_value_in_3_frame	max_value_in_3_frame_alternate_version	second_max_value_in_3_frame	moving_array_as_string	sorted_moving_array_as_string
1	0.6	0.6	0.6	null	[0.6]	[0.6]
2	5.4	5.4	5.4	0.6	[0.6, 5.4]	[5.4, 0.6]
3	5.4	5.4	5.4	5.4	[0.6, 5.4, 5.4]	[5.4, 5.4, 0.6]
4	5.9	5.9	5.9	5.4	[0.6, 5.4, 5.4, 5.9]	[5.9, 5.4, 5.4, 0.6]
5	4.6	5.9	5.9	5.4	[5.4, 5.4, 5.9, 4.6]	[5.9, 5.4, 5.4, 4.6]
6	6	6	6	5.9	[5.4, 5.9, 4.6, 6]	[6, 5.9, 5.4, 4.6]
7	8.7	8.7	8.7	6	[5.9, 4.6, 6, 8.7]	[8.7, 6, 5.9, 4.6]
8	3.5	8.7	8.7	6	[4.6, 6, 8.7, 3.5]	[8.7, 6, 4.6, 3.5]
9	4	8.7	8.7	6	[6, 8.7, 3.5, 4]	[8.7, 6, 4, 3.5]
10	0.7	8.7	8.7	4	[8.7, 3.5, 4, 0.7]	[8.7, 4, 3.5, 0.7]

作为控制措施，我还使用新旧方法计算了最大值，以检查两者的结果是否相同。

最后，您很幸运能够使用 BigQuery，因为我认为其他 SQL 引擎不支持此查询中使用的所有功能，至少不支持如此优雅的语法。