Bigquery Lag 导航功能忽略空值答案

【问题标题】：Bigquery Lag navigation function ignore nullsBigquery Lag 导航功能忽略空值
【发布时间】：2021-04-06 22:45:34
【问题描述】：

我有一张如下表：

WITH
  table1 AS (
  SELECT
    TIMESTAMP '2016-10-18 2:51:45' AS ts,
    'AB4' AS packId,
    FALSE AS stat,
    'x' AS position
  UNION ALL
  SELECT
    TIMESTAMP '2016-10-18 2:54:11',
    'AB4',
    FALSE,
    'x'
  UNION ALL
  SELECT
    TIMESTAMP '2016-10-18 3:01:17',
    'AB4',
    FALSE,
    'x'
  UNION ALL
  SELECT
    TIMESTAMP '2016-10-18 3:05:42',
    'AB4',
    TRUE,
    NULL
  UNION ALL
  SELECT
    TIMESTAMP '2016-10-18 3:06:24',
    'AB4',
    FALSE,
    'x'
  UNION ALL
  SELECT
    TIMESTAMP '2016-10-18 3:06:36',
    'AB4',
    TRUE,
    'y'
  UNION ALL
  SELECT
    TIMESTAMP '2016-10-18 3:07:41',
    'AB4',
    FALSE,
    'x')
SELECT
  ts,
  packId,
  stat,
  position,
  LAG(position) OVER prev_pos AS previous_position1,
  LAG(position,2) OVER prev_pos AS previous_position2
FROM
  table1
WINDOW
  prev_pos AS (
  PARTITION BY
    packId
  ORDER BY
    ts)

结果如下所示：

我想获得不为 null 的前一个值（从列位置），如结果表所示，我希望这些 null 值是前一个非 null 值，在本例中它们都是 'x' . （有数千行有空位置，所以最后一个非空位置当前行可能是它之前的一千行）

总结：如何在 bigquery 中忽略 Lag 的空值？

【问题讨论】：

标签： sql google-bigquery window-functions

【解决方案1】：

以下是 BigQuery 标准 SQL

WITH table1 AS (
  SELECT TIMESTAMP '2016-10-18 2:51:45' AS ts, 'AB4' AS packId, FALSE AS stat, 'x3' AS position UNION ALL
  SELECT TIMESTAMP '2016-10-18 2:54:11', 'AB4', FALSE, 'x2' UNION ALL
  SELECT TIMESTAMP '2016-10-18 3:01:17', 'AB4', FALSE, 'x1' UNION ALL
  SELECT TIMESTAMP '2016-10-18 3:05:42', 'AB4', TRUE, NULL UNION ALL
  SELECT TIMESTAMP '2016-10-18 3:06:24', 'AB4', FALSE, 'x' UNION ALL
  SELECT TIMESTAMP '2016-10-18 3:06:36', 'AB4', TRUE, 'y' UNION ALL
  SELECT TIMESTAMP '2016-10-18 3:07:41', 'AB4', FALSE, 'x'
)
SELECT
  ts,
  packId,
  stat,
  position,
  NTH_VALUE(position, 1 IGNORE NULLS) OVER prev_pos AS previous_position1,
  NTH_VALUE(position, 2 IGNORE NULLS) OVER prev_pos AS previous_position2
FROM  table1
WINDOW prev_pos AS 
  (PARTITION BY packId ORDER BY ts DESC ROWS BETWEEN 1 FOLLOWING and UNBOUNDED FOLLOWING)
-- ORDER BY ts

有输出

您可以看到不需要子查询 - 对原始查询的唯一更改是

将 LAG(position) 和 LAG(position,2) 分别替换为 - NTH_VALUE(position, 1 IGNORE NULLS) 和 NTH_VALUE(position, 2 IGNORE NULLS) 和
为prev_pos窗口添加适当的框架子句

【讨论】：

google 支付给@Mikhail 的钱不够，这个人是 stackoverflow 的英雄
:o) 哈哈 - 从来都不是 Google 员工 - 只是 GCP/BigQuery 的忠实粉丝 :o)

【解决方案2】：

BigQuery 使这有点困难。一种方法是枚举具有位置值的组。然后使用窗口框架规范从前一组和前一组中获取值。

诀窍是分配组。最简单的方法是对每行上或之后的非 NULL 值进行计数。这会将带有 NULL 值的每一行分配给 next 组。但是，组号是相反的。

这是结果查询：

SELECT t1.*,
       MAX(position) OVER (PARTITION BY packId ORDER BY grp DESC RANGE BETWEEN 1 preceding and 1 preceding) as prev1,
       MAX(position) OVER (PARTITION BY packId ORDER BY grp DESC RANGE BETWEEN 2 preceding and 2 preceding) as prev2,
FROM (SELECT t1.*,
             COUNT(position) OVER (PARTITION BY packid ORDER BY ts DESC) as grp
      FROM table1 t1 
     ) t1

【讨论】：

真棒@gordon-linoff
只有一个问题，如果我想在多个列上进行分区，那么可以将它添加到查询中的 packId 列旁边吗？像这样：SELECT t1.*, MAX(position) OVER (PARTITION BY packId,anotherCol ORDER BY grp DESC RANGE BETWEEN 1 preceding and 1 preceding) as prev1, MAX(position) OVER (PARTITION BY packId,anotherCol ORDER BY grp DESC RANGE BETWEEN 2 preceding and 2 preceding) as prev2, FROM (SELECT t1.*, COUNT(position) OVER (PARTITION BY packid,anotherCol ORDER BY ts DESC) as grp FROM table1 t1 ) t1
@Saba 。 . .额外的partition by 列应该没问题。