在 Postgresql 中过滤时间序列数据以返回前一条记录和后一条记录答案

【问题标题】：Filter timeseries data in Postgresql to return previous record and following records在 Postgresql 中过滤时间序列数据以返回前一条记录和后一条记录
【发布时间】：2022-01-11 13:34:07
【问题描述】：

我有一个包含传感器读数的大表，以及一个包含新读数的相应暂存表。

我正在尝试过滤从主表中获取的记录，以便仅获取用于计算增量的相关信息。

一个独立的简化示例：

在主表中为传感器 1 到 6 创建过去 7 天的虚拟数据。
为传感器 3 到 5 创建新的传感器数据，在过去 3 天中进行。

    DROP TABLE IF EXISTS tb_sensor_reading;
    DROP TABLE IF EXISTS stg_sensor_reading;
    CREATE TABLE tb_sensor_reading(
        sensor_id INTEGER
        , ts_reading TIMESTAMP
        , reading_value NUMERIC(8, 3)
    );
    CREATE TABLE stg_sensor_reading AS TABLE tb_sensor_reading WITH NO DATA;
    
    SELECT * FROM stg_sensor_reading;
    
    INSERT INTO tb_sensor_reading( 
         sensor_id 
         , ts_reading 
         , reading_value 
    ) 
    SELECT 
        generated_sensor_id 
        , generated_ts_reading 
        -- readings delta may vary between +25 and +49.9 between each reading 
        , EXTRACT(EPOCH FROM generated_ts_reading) / (3600 * 6 / 25) 
            - EXTRACT(EPOCH FROM NOW() - INTERVAL '7 days') / (3600 * 6 / 25) 
            + RANDOM()*24.9
    FROM 
        GENERATE_SERIES(1,6) AS generated_sensor_id 
        , GENERATE_SERIES( 
            NOW() - INTERVAL '7 days', NOW(), INTERVAL '6 hours' 
        ) AS generated_ts_reading 
    ; 
    -- generate dummy records in stg_reading 
    INSERT INTO stg_sensor_reading( 
         sensor_id 
         , ts_reading 
         , reading_value 
    ) 
    SELECT 
        generated_sensor_id 
        , generated_ts_reading 
        , EXTRACT(EPOCH FROM generated_ts_reading) / (3600 * 6 / 25) 
          - EXTRACT(EPOCH FROM NOW() - INTERVAL '7 days') / (3600 * 6 / 25) 
          + RANDOM()*24.9 
    FROM 
        GENERATE_SERIES(3, 5) AS generated_sensor_id 
        , GENERATE_SERIES( 
            NOW() - INTERVAL '3 days'
            , NOW() - INTERVAL '2 days'
            , INTERVAL '6 hours' 
        ) AS generated_ts_reading 
    ;
    SELECT * FROM tb_sensor_reading LIMIT 5 ;

结果

SELECT * FROM tb_sensor_reading LIMIT 5;

     sensor_id |         ts_reading         | reading_value
    -----------+----------------------------+---------------
             1 | 2022-01-03 13:25:16.417613 |         7.154
             1 | 2022-01-03 19:25:16.417613 |        46.736
             1 | 2022-01-04 01:25:16.417613 |        73.835
             1 | 2022-01-04 07:25:16.417613 |        89.147
             1 | 2022-01-04 13:25:16.417613 |       105.444
(5 rows) 

 SELECT * FROM stg_sensor_reading ;
 sensor_id |         ts_reading         | reading_value
-----------+----------------------------+---------------
         3 | 2022-01-07 18:40:19.978908 |       406.839
         3 | 2022-01-08 00:40:19.978908 |       435.935
         3 | 2022-01-08 06:40:19.978908 |       454.385
         3 | 2022-01-08 12:40:19.978908 |       495.853
         3 | 2022-01-08 18:40:19.978908 |       516.214
         4 | 2022-01-07 18:40:19.978908 |       410.096
         4 | 2022-01-08 00:40:19.978908 |       442.941
         4 | 2022-01-08 06:40:19.978908 |       462.816
         4 | 2022-01-08 12:40:19.978908 |       496.893
         4 | 2022-01-08 18:40:19.978908 |       517.247
         5 | 2022-01-07 18:40:19.978908 |       419.876
         5 | 2022-01-08 00:40:19.978908 |       425.990
         5 | 2022-01-08 06:40:19.978908 |       464.086
         5 | 2022-01-08 12:40:19.978908 |       486.731
         5 | 2022-01-08 18:40:19.978908 |       515.454
(15 rows)

我正在尝试从 tb_sensor_readings 中选择要返回的记录：

✅ 仅存在于 stg_sensor_reading 中的 sensor_id ：

SELECT * FROM tb_sensor_reading
WHERE sensor_id IN (
    SELECT DISTINCT sensor_id from stg_sensor_reading
)
ORDER BY sensor_id, ts_reading

LIMIT 5;

 sensor_id |         ts_reading         | reading_value
-----------+----------------------------+---------------
         3 | 2022-01-03 13:25:16.417613 |        24.669
         3 | 2022-01-03 19:25:16.417613 |        48.384
         3 | 2022-01-04 01:25:16.417613 |        56.889
         3 | 2022-01-04 07:25:16.417613 |        95.980
         3 | 2022-01-04 13:25:16.417613 |       120.536

❌ 每个 sensor_id 仅前一条记录和以下记录，即伪代码：

MAX(tb_sensor_reading.ts_reading) < MIN(stg_sensor_reading.ts_reading)
OR tb_sensor_reading.ts_reading > MIN(stg_sensor_reading.ts_reading)
ON tb_sensor_reading.sensor_id = stg_sensor_reading. sensor_id

我该怎么做：

为每个传感器返回正确的系列，即使它们的最小时间戳在彼此之间有很大差异？
在过滤器中包含之前具有相同 sensor_id 的记录？

非常感谢！

【问题讨论】：

试试lag和lead
@Bergi 感谢您的评论。我设法找到了一个使用 DISTINCT ON 的解决方案，但不确定与此类数据的 LAG 函数相比是否表现良好。

标签： postgresql

【解决方案1】：

我可以使用 UNION 运算符从 tb_reading 中选择所需的行。

第一个查询获取每个查询的最后读数 sensor_id 来自 tb_reading，发生在第一个新读数之前。
第二个子查询返回来自 tb_sensor_reading 的读数，对于每个 sensor_id，这发生在 staging 表中可用读数的同一时间段内。
如果查询的作用是限制计算增量的行数，那么 WHERE tb.sensor_id = stg.sensor_id 是不值得的。

(SELECT -- most recent previous reading for each sensor_id
    DISTINCT ON (sensor_id)
    sensor_id
    , ts_reading
    , reading_value
    FROM
      tb_sensor_reading tb
    WHERE sensor_id IN (SELECT DISTINCT sensor_id FROM stg_sensor_reading)
       AND ts_reading < ALL(
           SELECT MIN(ts_reading)
           FROM stg_sensor_reading stg
           WHERE tb.sensor_id = stg.sensor_id
    ) ORDER BY sensor_id, ts_reading DESC
  
)
UNION
    SELECT -- all readings that happened in the same period than stg
      sensor_id
      , ts_reading
      , reading_value
    FROM tb_sensor_reading tb
    WHERE
       sensor_id IN (SELECT DISTINCT sensor_id FROM stg_sensor_reading)
       AND ts_reading >= ALL(
          SELECT MIN(ts_reading)
          FROM stg_sensor_reading stg
          WHERE tb.sensor_id = stg.sensor_id
        )
       AND ts_reading <= ALL(
          SELECT MAX(ts_reading)
          FROM stg_sensor_reading stg
          WHERE tb.sensor_id = stg.sensor_id
        )
    ORDER BY sensor_id, ts_reading DESC ;

结果


 sensor_id |         ts_reading         | reading_value
-----------+----------------------------+---------------
         3 | 2022-01-08 18:40:19.973512 |       514.387
         3 | 2022-01-08 12:40:19.973512 |       490.200
         3 | 2022-01-08 06:40:19.973512 |       468.323
         3 | 2022-01-08 00:40:19.973512 |       438.597
         3 | 2022-01-07 18:40:19.973512 |       420.665
         4 | 2022-01-08 18:40:19.973512 |       524.803
         4 | 2022-01-08 12:40:19.973512 |       497.348
         4 | 2022-01-08 06:40:19.973512 |       468.300
         4 | 2022-01-08 00:40:19.973512 |       430.468
         4 | 2022-01-07 18:40:19.973512 |       411.204
         5 | 2022-01-08 18:40:19.973512 |       504.969
         5 | 2022-01-08 12:40:19.973512 |       482.629
         5 | 2022-01-08 06:40:19.973512 |       473.075
         5 | 2022-01-08 00:40:19.973512 |       432.396
         5 | 2022-01-07 18:40:19.973512 |       405.156
(15 rows)

【讨论】：