使用值数组聚合时间序列数据答案

【问题标题】：Aggregate timeseries data with value array使用值数组聚合时间序列数据
【发布时间】：2020-07-15 02:41:18
【问题描述】：

我在一个表中有传感器数据，按时间戳记，数组中有多个值。例如：

CREATE TABLE test_raw (
  ts timestamp without time zone NOT NULL,
  values real[]
);
INSERT INTO test_raw VALUES
  ('2020-7-14 00:00:00', ARRAY[1, 10]),
  ('2020-7-14 00:01:00', ARRAY[2, 20, 30]),
  ('2020-7-14 00:20:00', ARRAY[3, NULL, 30, 40]),
  ('2020-7-14 00:23:00', ARRAY[9, NULL, 50, 80]),
  ('2020-7-14 00:10:00', ARRAY[3, 30, 40]),
  ('2020-7-14 00:11:00', ARRAY[3, 30, NULL, 50])
;

该数组对应于设备收集的不同指标，例如，values[1] 可能是温度，values[2] 可能是湿度等。完整架构具有额外的列（例如设备 ID），用于指示数组的内容包含。

我现在想创建一个聚合/汇总表，例如，平均超过 10 分钟。如果值是标量而不是数组，我将编写以下视图（我将使用它来填充汇总表）：

CREATE VIEW test_raw_10m AS
SELECT
  floor(extract(epoch FROM ts)/600)*600 as ts,
  AVG(value) /* scalar value! */
FROM test_raw
GROUP BY ts;

但是值数组并不是那么简单。我看到了一个非常相关的问题的答案：Pairwise array sum aggregate function? 这使我想到以下内容，这似乎过于复杂：

WITH test_raw_10m AS (
  SELECT floor(extract(epoch FROM ts)/600)*600 as ts, values
  FROM test_raw
)
SELECT
  t.ts,
  ARRAY( SELECT
    AVG(value) as value
    FROM test_raw_10m tt, UNNEST(tt.values) WITH ORDINALITY x(value, rn)
    WHERE tt.ts = t.ts
    GROUP by x.rn
    ORDER by x.rn) AS values
FROM test_raw_10m AS t
GROUP BY ts
ORDER by ts
;

我的问题：有更好的方法吗？

为了完整起见，以下是给出上述示例数据的结果：

     ts     |     values
------------+----------------
 1594684800 | {1.5,15,30}
 1594685400 | {3,30,40,50}
 1594686000 | {6,NULL,40,60}
(3 rows)

这是查询计划：

                                        QUERY PLAN
-------------------------------------------------------------------------------------------
 Group  (cost=119.37..9490.26 rows=200 width=40)
   Group Key: t.ts
   CTE test_raw_10m
     ->  Seq Scan on test_raw  (cost=0.00..34.00 rows=1200 width=40)
   ->  Sort  (cost=85.37..88.37 rows=1200 width=8)
         Sort Key: t.ts
         ->  CTE Scan on test_raw_10m t  (cost=0.00..24.00 rows=1200 width=8)
   SubPlan 2
     ->  Sort  (cost=46.57..46.82 rows=100 width=16)
           Sort Key: x.rn
           ->  HashAggregate  (cost=42.00..43.25 rows=100 width=16)
                 Group Key: x.rn
                 ->  Nested Loop  (cost=0.00..39.00 rows=600 width=12)
                       ->  CTE Scan on test_raw_10m tt  (cost=0.00..27.00 rows=6 width=32)
                             Filter: (ts = t.ts)
                       ->  Function Scan on unnest x  (cost=0.00..1.00 rows=100 width=12)

【问题讨论】：

您可以在具有请求行为的数组上引入自己的聚合函数 - 但可能仅此而已 - 对于这种情况，我不确定自定义聚合是否比您的查询更快 - 可能不会 - 但它取决于数据。
您的查询对我来说看起来不错。

标签： postgresql

【解决方案1】：

如果我通过将FROM test_raw 更改为FROM test_raw WHERE ts >= <some timestamp> 之类的内容（在两个查询中）进行部分更新，则以下查询在我的真实数据集上的速度要快得多：

SELECT bucket as ts, ARRAY_AGG(v)
FROM (
  SELECT to_timestamp(floor(extract(epoch FROM ts)/600)*600) as bucket, AVG(values[i]) AS v
  FROM (SELECT ts, generate_subscripts(values, 1) AS i, values FROM test_raw) AS foo
  GROUP BY bucket, i
  ORDER BY bucket, i
) bar
GROUP BY bucket;

我认为ORDER BY bucket, i 不是必需的，但我不确定。

                                                                   QUERY PLAN

-----------------------------------------------------------------------------------------------------------------------------------------------
-
 GroupAggregate  (cost=228027.62..241630.12 rows=200 width=40) (actual time=0.948..1.209 rows=3 loops=1)
   Group Key: (to_timestamp((floor((date_part('epoch'::text, foo.ts) / '600'::double precision)) * '600'::double precision)))
   ->  GroupAggregate  (cost=228027.62..241027.62 rows=40000 width=20) (actual time=0.826..1.099 rows=11 loops=1)
         Group Key: (to_timestamp((floor((date_part('epoch'::text, foo.ts) / '600'::double precision)) * '600'::double precision))), foo.i
         ->  Sort  (cost=228027.62..231027.62 rows=1200000 width=44) (actual time=0.773..0.870 rows=20 loops=1)
               Sort Key: (to_timestamp((floor((date_part('epoch'::text, foo.ts) / '600'::double precision)) * '600'::double precision))), foo.i
               Sort Method: quicksort  Memory: 19kB
               ->  Subquery Scan on foo  (cost=0.00..33031.00 rows=1200000 width=44) (actual time=0.165..0.619 rows=20 loops=1)
                     ->  ProjectSet  (cost=0.00..6031.00 rows=1200000 width=44) (actual time=0.131..0.312 rows=20 loops=1)
                           ->  Seq Scan on test_raw  (cost=0.00..22.00 rows=1200 width=40) (actual time=0.034..0.070 rows=6 loops=1)
 Planning Time: 0.525 ms
 Execution Time: 1.504 ms
(12 rows)

【讨论】：