连续重复/重复的有序计数

Question

我非常怀疑我是否以最有效的方式执行此操作，这就是我在此处标记plpgsql 的原因。我需要在 20 亿行 上为 一千个测量系统 运行它。

您的测量系统通常会在失去连接时报告之前的值，并且它们经常会因为突发但有时会持续很长时间而失去连接。您需要汇总，但是当您这样做时，您需要查看它重复了多长时间并根据该信息制作各种过滤器。假设您正在测量汽车的 mpg，但它停留在 20 mpg 一个小时，然后移动到 20.1，依此类推。您需要在卡住时评估准确性。您还可以放置一些替代规则来查找汽车何时在高速公路上，并且使用窗口功能您可以生成汽车的“状态”并进行分组。废话不多说：

--here's my data, you have different systems, the time of measurement, and the actual measurement
--as well, the raw data has whether or not it's a repeat (hense the included window function
select * into temporary table cumulative_repeat_calculator_data
FROM
    (
    select 
    system_measured, time_of_measurement, measurement, 
    case when 
     measurement = lag(measurement,1) over (partition by system_measured order by time_of_measurement asc) 
     then 1 else 0 end as repeat
    FROM
    (
    SELECT 5 as measurement, 1 as time_of_measurement, 1 as system_measured
    UNION
    SELECT 150 as measurement, 2 as time_of_measurement, 1 as system_measured
    UNION
    SELECT 5 as measurement, 3 as time_of_measurement, 1 as system_measured
    UNION
    SELECT 5 as measurement, 4 as time_of_measurement, 1 as system_measured
    UNION
    SELECT 5 as measurement, 1 as time_of_measurement, 2 as system_measured
    UNION
    SELECT 5 as measurement, 2 as time_of_measurement, 2 as system_measured
    UNION
    SELECT 5 as measurement, 3 as time_of_measurement, 2 as system_measured
    UNION
    SELECT 5 as measurement, 4 as time_of_measurement, 2 as system_measured
    UNION
    SELECT 150 as measurement, 5 as time_of_measurement, 2 as system_measured
    UNION
    SELECT 5 as measurement, 6 as time_of_measurement, 2 as system_measured
    UNION
    SELECT 5 as measurement, 7 as time_of_measurement, 2 as system_measured
    UNION
    SELECT 5 as measurement, 8 as time_of_measurement, 2 as system_measured
    ) as data
) as data;

--unfortunately you can't have window functions within window functions, so I had to break it down into subquery
--what we need is something to partion on, the 'state' of the system if you will, so I ran a running total of the nonrepeats
--this creates a row that stays the same when your data is repeating - aka something you can partition/group on
select * into temporary table cumulative_repeat_calculator_step_1
FROM
    (
    select 
    *,
    sum(case when repeat = 0 then 1 else 0 end) over (partition by system_measured order by time_of_measurement asc) as cumlative_sum_of_nonrepeats_by_system
    from cumulative_repeat_calculator_data
    order by system_measured, time_of_measurement
) as data;

--finally, the query. I didn't bother showing my desired output, because this (finally) got it
--I wanted a sequential count of repeats that restarts when it stops repeating, and starts with the first repeat
--what you can do now is take the average measurement under some condition based on how long it was repeating, for example  
select *, 
case when repeat = 0 then 0
else
row_number() over (partition by cumlative_sum_of_nonrepeats_by_system, system_measured order by time_of_measurement) - 1
end as ordered_repeat
from cumulative_repeat_calculator_step_1
order by system_measured, time_of_measurement

那么，为了在一个巨大的桌子上运行它，你会做些什么不同的事情，或者你会使用什么替代工具？我在考虑 plpgsql，因为我怀疑这需要在数据库中或在数据插入过程中完成，尽管我通常在加载数据后使用数据。有没有办法在不诉诸子查询的情况下一次性完成？

我已经测试了一个替代方法，但它仍然依赖于子查询，我认为这更快。对于该方法，您可以使用 start_timestamp、end_timestamp、system 创建一个“开始和停止”表。然后加入更大的表，如果时间戳介于两者之间，则将其归类为处于该状态，这本质上是cumlative_sum_of_nonrepeats_by_system 的替代方案。但是，当您这样做时，您会以 1=1 的方式加入数千台设备和数千或数百万个“事件”。你认为这是一个更好的方法吗？

Answer 1

测试用例

首先，一种更有用的方式来呈现您的数据 - 甚至更好，在 sqlfiddle 中，准备好使用：

CREATE TEMP TABLE data(
   system_measured int
 , time_of_measurement int
 , measurement int
);

INSERT INTO data VALUES
 (1, 1, 5)
,(1, 2, 150)
,(1, 3, 5)
,(1, 4, 5)
,(2, 1, 5)
,(2, 2, 5)
,(2, 3, 5)
,(2, 4, 5)
,(2, 5, 150)
,(2, 6, 5)
,(2, 7, 5)
,(2, 8, 5);

简化查询

由于尚不清楚，我仅假设上述内容。
接下来，我将您的查询简化为：

WITH x AS (
   SELECT *, CASE WHEN lag(measurement) OVER (PARTITION BY system_measured
                               ORDER BY time_of_measurement) = measurement
                  THEN 0 ELSE 1 END AS step
   FROM   data
   )
   , y AS (
   SELECT *, sum(step) OVER(PARTITION BY system_measured
                            ORDER BY time_of_measurement) AS grp
   FROM   x
   )
SELECT * ,row_number() OVER (PARTITION BY system_measured, grp
                             ORDER BY time_of_measurement) - 1 AS repeat_ct
FROM   y
ORDER  BY system_measured, time_of_measurement;

现在，虽然使用纯 SQL 非常好用，但使用 plpgsql 函数会快很多，因为它可以在查询至少需要的单个表扫描中完成三个扫描。

使用 plpgsql 函数更快：

CREATE OR REPLACE FUNCTION x.f_repeat_ct()
  RETURNS TABLE (
    system_measured int
  , time_of_measurement int
  , measurement int, repeat_ct int
  )  LANGUAGE plpgsql AS
$func$
DECLARE
   r    data;     -- table name serves as record type
   r0   data;
BEGIN

-- SET LOCAL work_mem = '1000 MB';  -- uncomment an adapt if needed, see below!

repeat_ct := 0;   -- init

FOR r IN
   SELECT * FROM data d ORDER BY d.system_measured, d.time_of_measurement
LOOP
   IF  r.system_measured = r0.system_measured
       AND r.measurement = r0.measurement THEN
      repeat_ct := repeat_ct + 1;   -- start new array
   ELSE
      repeat_ct := 0;               -- start new count
   END IF;

   RETURN QUERY SELECT r.*, repeat_ct;

   r0 := r;                         -- remember last row
END LOOP;

END
$func$;

呼叫：

SELECT * FROM x.f_repeat_ct();

确保在这种 plpgsql 函数中始终对列名进行表限定，因为我们使用与输出参数相同的名称，如果没有限定，则优先。

数十亿行

如果您有 十亿 行，您可能需要拆分此操作。我引用手册here：

注意：RETURN NEXT 和 RETURN QUERY 的当前实现 在从函数返回之前存储整个结果集，如 上面讨论过。这意味着如果一个 PL/pgSQL 函数产生一个 非常大的结果集，性能可能很差：数据将被写入 到磁盘以避免内存耗尽，但函数本身不会 返回，直到生成整个结果集。一个未来 PL/pgSQL 版本可能允许用户定义集合返回 没有此限制的功能。目前，点在 哪些数据开始写入磁盘由work_mem 控制 配置变量。有足够内存的管理员 在内存中存储更大的结果集应该考虑增加这个 参数。

考虑一次为一个系统计算行，或为work_mem 设置足够高的值以应对负载。按照报价中提供的链接了解有关 work_mem 的更多信息。

一种方法是在您的函数中使用SET LOCAL 为work_mem 设置一个非常高的值，这仅对当前事务有效。我在函数中添加了注释行。 不要在全局范围内将其设置得非常高，因为这可能会破坏您的服务器。阅读手册。