【问题标题】：Aggregate rows ahead only where one value in the row ahead is less than another仅在前面行中的一个值小于另一个值时聚合前面的行
【发布时间】：2020-09-25 00:05:19
【问题描述】：

我的每台机器都有两个计数，p 和 r。

p 应该总是大于或等于r，但由于技术滞后和聚合周期短，情况并非总是如此，r 计数经常 - 但是并非总是 - 显示上一时期的数据。因为滞后的长度不是恒定的，所以无法确切知道r 值属于哪个时期。因此，我不能简单地将所有 r 计数统一向后移动，因为这可能会产生以前没有的其他差异。

这种情况无法改变，我必须按原样处理数据。

在下面的示例中，您可以看到 p 在机器 1 上短暂“暂停”，在机器 2 上显着减慢，但 r 计数继续返回更大的值在“暂停”之前比p 短时间：

-- Dummy data
declare @t table(d date,m int,p int,r int);
insert into @t values(getdate()-9,1,100,10),(getdate()-8,1,90 ,10),(getdate()-7,1,70 ,10),(getdate()-6,1,70 ,10),(getdate()-5,1,80 ,10),(getdate()-4,1,50 ,10),(getdate()-3,1,10 ,10),(getdate()-2,1,0  ,10),(getdate()-1,1,0  ,10),(getdate()+0,1,0  ,10),(getdate()+1,1,0  ,0),(getdate()+2,1,0  ,0),(getdate()+3,1,40 ,0),(getdate()+4,1,50 ,0),(getdate()+5,1,80 ,10),(getdate()-9,2,1100,100),(getdate()-8,2,190 ,100),(getdate()-7,2,170 ,100),(getdate()-6,2,170 ,100),(getdate()-5,2,180 ,100),(getdate()-4,2,150 ,100),(getdate()-3,2,110 ,100),(getdate()-2,2,10  ,100),(getdate()-1,2,10  ,100),(getdate()+0,2,10  ,100),(getdate()+1,2,10  ,0),(getdate()+2,2,10  ,0),(getdate()+3,2,140 ,0),(getdate()+4,2,150 ,0),(getdate()+5,2,180 ,100);
select * from @t order by m,d;

-- Output
+------------+---+------+-----+
|     d      | m |  p   |  r  |
+------------+---+------+-----+
| 2020-05-27 | 1 |  100 |  10 |
| 2020-05-28 | 1 |   90 |  10 |
| 2020-05-29 | 1 |   70 |  10 |
| 2020-05-30 | 1 |   70 |  10 |
| 2020-05-31 | 1 |   80 |  10 |
| 2020-06-01 | 1 |   50 |  10 |
| 2020-06-02 | 1 |   10 |  10 |
| 2020-06-03 | 1 |    0 |  10 |
| 2020-06-04 | 1 |    0 |  10 |
| 2020-06-05 | 1 |    0 |  10 |
| 2020-06-06 | 1 |    0 |   0 |
| 2020-06-07 | 1 |    0 |   0 |
| 2020-06-08 | 1 |   40 |   0 |
| 2020-06-09 | 1 |   50 |   0 |
| 2020-06-10 | 1 |   80 |  10 |
| 2020-05-27 | 2 | 1100 | 100 |
| 2020-05-28 | 2 |  190 | 100 |
| 2020-05-29 | 2 |  170 | 100 |
| 2020-05-30 | 2 |  170 | 100 |
| 2020-05-31 | 2 |  180 | 100 |
| 2020-06-01 | 2 |  150 | 100 |
| 2020-06-02 | 2 |  110 | 100 |
| 2020-06-03 | 2 |   10 | 100 |
| 2020-06-04 | 2 |   10 | 100 |
| 2020-06-05 | 2 |   10 | 100 |
| 2020-06-06 | 2 |   10 |   0 |
| 2020-06-07 | 2 |   10 |   0 |
| 2020-06-08 | 2 |  140 |   0 |
| 2020-06-09 | 2 |  150 |   0 |
| 2020-06-10 | 2 |  180 | 100 |
+------------+---+------+-----+

我需要能够在一定程度上及时向后调整这些 r 计数，以便将它们添加到前面的行中，以使每个 p 数字大于或等于相应的 r价值。

在上述m = 1 的示例中，输出可能类似于以下r 计数中的任何；我不关心调整的范围，只关心每一行的p >= r，并且调整只会及时倒退：

+------------+---+------+------+------+------+
|     d      | m |  p   |  r1  |  r2  |  r3  |
+------------+---+------+------+------+------+
| 2020-05-27 | 1 |  100 |   10 |   10 |   10 |
| 2020-05-28 | 1 |   90 |   10 |   10 |   10 |
| 2020-05-29 | 1 |   70 |   10 |   15 |   10 |
| 2020-05-30 | 1 |   70 |   20 |   20 |   10 |) Note how the original 30 r counts
| 2020-05-31 | 1 |   80 |   20 |   20 |   10 |} that didn't follow the rule
| 2020-06-01 | 1 |   50 |   20 |   15 |   40 |) have been moved back in time
| 2020-06-02 | 1 |   10 |   10 |   10 |   10 |
| 2020-06-03 | 1 |    0 |    0 |    0 |    0 |
| 2020-06-04 | 1 |    0 |    0 |    0 |    0 |
| 2020-06-05 | 1 |    0 |    0 |    0 |    0 |
| 2020-06-06 | 1 |    0 |    0 |    0 |    0 |
| 2020-06-07 | 1 |    0 |    0 |    0 |    0 |
| 2020-06-08 | 1 |   40 |    0 |    0 |    0 |
| 2020-06-09 | 1 |   50 |    0 |    0 |    0 |
| 2020-06-10 | 1 |   80 |   10 |   10 |   10 |
+------------+---+------+------+------+------+

我已尝试使用窗口函数和rows between 等解决此问题，但我不知道如何识别需要重新分配给前一周期的r 值，以及识别哪个@987654345 @ 值来分配它们。如果我取得任何进展，我会在下面添加，但非常感谢所有帮助。

尝试 1

我管理的最接近的是以下适用于上述情况，但是当您将p = 50 值更改为小于40 并且当我只想向后调整时，也会及时向前和向后调整：

with t as(
select row_number() over (partition by m order by d) as rn
      ,(row_number() over (partition by m order by d)-1) / 5 as gn
      ,*
from @t
where m = 1
)
select *
      ,case when p > r
            then r + (sum(case when p < r then r else 0 end) over (partition by gn) / sum(case when p > r then 1 else 0 end) over (partition by gn))
            else case when p = r
                      then r
                      else 0
                      end
            end as r_adj
from t;

尝试 2

这更接近了，但仍在向前和向后调整时间：

with t as(
select row_number() over (partition by m order by d) as rn
      ,(row_number() over (partition by m order by d)-1) / 10 as gn
      ,(row_number() over (partition by m order by d)+4) / 10 as gn2
      ,*
from @t
where m = 1
)
,r1 as(
select *
      ,case when p > r
            then r + (sum(case when p < r then r - p else 0 end) over (partition by gn) / sum(case when p > r then 1. else 0. end) over (partition by gn))
            else case when p = r
                      then r
                      else 0
                      end
            end as r_adj
from t
)
select d
      ,m
      ,p
      ,r
      ,case when p > r_adj
            then r_adj + (sum(case when p < r_adj then r_adj - p else 0 end) over (partition by gn2) / sum(case when p > r_adj then 1. else 0. end) over (partition by gn2))
            else case when p = r_adj
                      then r_adj
                      else r_adj - (r_adj - p)
                      end
            end as r_new
from r1
order by rn
;

【问题讨论】：

你能把你的SQL代码放上来吗？
我关心的是如何从源代码生成 r 字段。
r 字段是源。正如我所说，它如您在上面看到的那样到达，需要相应地调整以适应 p >= r 规则。
如果你给我看的是源表，那么你必须从那里回溯。您需要了解如何将 r 与“主键”的一种形式相关联，以便确定哪些记录是错误的，以及需要偏移多少行。目标是将 r 与适当的字段“重新对齐”。如果是我，我会要求填写该表的人对其进行量化。
正如我在 OP 中所说：这种情况无法改变，我必须按原样处理数据。 并非世界上所有的数据都来自另一个数据库。在这种情况下，数据直接来自连接到现实世界机器的传感器并按原样到达。没有“主键”可以回溯，也无法更改传感器的输出。

标签： sql sql-server time-series sql-server-2016 window-functions

【解决方案1】：

一种方法使用apply：

select t.*,
       t2.r as imputed_r
from t outer apply
     (select top (1) t2.*
      from t t2
      where t2.m = t.m and
            t2.d >= t.d and t2.r <= t.p
      order by t2.d desc
     ) t2;

Here 是一个 dbfiddle。

【讨论】：

我想你误解了我的问题。我不希望 r 计数消失，而是包含在 p 值足够大以容纳它的先前行中。表中p 和r 的总数不能改变。

【解决方案2】：

您可能可以稍微清理一下，但希望冗长可以帮助您了解正在发生的事情。

WITH r1 AS (
-- Find what the previous day's p value was so we can compare it with today's
SELECT
    *
    ,LAG(p) OVER (PARTITION BY m ORDER BY d) AS previous_p
FROM
    @t
)
,r2 AS (
-- Compute the change between p valuesin percentage
SELECT
    *
    ,CASE WHEN r1.previous_p > 0 THEN ((r1.p*1.0-r1.previous_p)/r1.previous_p) ELSE 0 END AS p_delta
FROM
    r1
)
,r3 AS (
-- Use the percentage change to check if a machine was "paused"
SELECT
    *
    ,CASE WHEN r2.p_delta > 0.8 OR (r2.previous_p = 0 AND r2.p > 0) THEN 1 ELSE 0 END AS is_new_session -- Adjust 0.8 to whatever percent change makes sense for a new session
FROM
    r2
)
, r4 AS (
-- Put each row for a machine into its corresponding group
SELECT
    *
    ,SUM(r3.is_new_session) OVER (PARTITION BY r3.m ORDER BY r3.d) AS session_group_id
FROM
    r3
)
, r5 AS (
-- Now we can calculate for each group how total p and r leftover are for each day
SELECT
    *
    ,CASE WHEN r4.p-r4.r > 0 AND r4.r > 0 THEN r4.p-r4.r ELSE 0 END AS remaining_p
    ,CASE WHEN r4.r-r4.p > 0 THEN r4.r-r4.p ELSE 0 END AS remaining_r
FROM
    r4
)
, r6 AS (
-- We need to get the accumulating remaining amounts
SELECT
    *
    ,SUM(r5.remaining_p) OVER (PARTITION BY r5.m, r5.session_group_id ORDER BY r5.d ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) AS total_remaining_p
    ,SUM(r5.remaining_r) OVER (PARTITION BY r5.m, r5.session_group_id ORDER BY r5.d ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) AS total_remaining_r
FROM
    r5
)
,r7 AS (
-- We then go backwards and subtract accumulating p out of the accumulated r
SELECT
    *
    ,r6.total_remaining_r - r6.total_remaining_p AS adjusted_total_remaining_r
FROM
    r6
)
,r8 AS (
-- Shift everything back one day so the adjusted remaining r can be added to the correct date
SELECT
    *
    ,LEAD(r7.adjusted_total_remaining_r) OVER (PARTITION BY r7.m, r7.session_group_id ORDER BY r7.d) AS next_adjusted_total_remaining_r
FROM
    r7
)
-- Finally, calculate what the corected r amount is for each day
SELECT
    r8.d
   ,r8.m
   ,r8.p
   ,r8.r
    ,CASE WHEN r8.remaining_p > 0 THEN -- If there is any remaing p left where we can put r amounts
        CASE WHEN r8.total_remaining_p > r8.total_remaining_r THEN -- Also check that we have more accumlated left than r
            CASE WHEN r8.next_adjusted_total_remaining_r > 0 THEN r8.r + r8.next_adjusted_total_remaining_r -- If we still have space, add the adjusted amount of what is left
            ELSE r8.r -- If we have run out of extra r, leave it as r
            END
        ELSE r8.p -- For those days that we had to adjust by the max amount possible, r + adjusted_r will just be p
        END
     ELSE r8.r - r8.remaining_r END AS corrected_r -- Remove the extra r where r was greater than p
FROM
    r8
ORDER BY r8.m,r8.d

输出：

+------------+---+------+-----+-------------+
|     d      | m |  p   |  r  | corrected_r |
+------------+---+------+-----+-------------+
| 2020-05-27 | 1 |  100 |  10 |          10 |
| 2020-05-28 | 1 |   90 |  10 |          10 |
| 2020-05-29 | 1 |   70 |  10 |          10 |
| 2020-05-30 | 1 |   70 |  10 |          10 |
| 2020-05-31 | 1 |   80 |  10 |          10 |
| 2020-06-01 | 1 |   50 |  10 |          40 |
| 2020-06-02 | 1 |   10 |  10 |          10 |
| 2020-06-03 | 1 |    0 |  10 |           0 |
| 2020-06-04 | 1 |    0 |  10 |           0 |
| 2020-06-05 | 1 |    0 |  10 |           0 |
| 2020-06-06 | 1 |    0 |   0 |           0 |
| 2020-06-07 | 1 |    0 |   0 |           0 |
| 2020-06-08 | 1 |   40 |   0 |           0 |
| 2020-06-09 | 1 |   50 |   0 |           0 |
| 2020-06-10 | 1 |   80 |  10 |          10 |
| 2020-05-27 | 2 | 1100 | 100 |         100 |
| 2020-05-28 | 2 |  190 | 100 |         100 |
| 2020-05-29 | 2 |  170 | 100 |         160 |
| 2020-05-30 | 2 |  170 | 100 |         170 |
| 2020-05-31 | 2 |  180 | 100 |         180 |
| 2020-06-01 | 2 |  150 | 100 |         150 |
| 2020-06-02 | 2 |  110 | 100 |         110 |
| 2020-06-03 | 2 |   10 | 100 |          10 |
| 2020-06-04 | 2 |   10 | 100 |          10 |
| 2020-06-05 | 2 |   10 | 100 |          10 |
| 2020-06-06 | 2 |   10 |   0 |           0 |
| 2020-06-07 | 2 |   10 |   0 |           0 |
| 2020-06-08 | 2 |  140 |   0 |           0 |
| 2020-06-09 | 2 |  150 |   0 |           0 |
| 2020-06-10 | 2 |  180 | 100 |         100 |
+------------+---+------+-----+-------------+

【讨论】：

谢谢，但您的脚本因连续多个超额r 而崩溃。如果您有兴趣，我想通了并发布了答案。
不知道您所说的“连续多次超额r 跌倒”是什么意思。您对@gordon-linoff 的回答的抱怨是他删除了r，但您也在您提出的答案中这样做了。我的不会丢失任何r。这就是问题所在吗？我保留了所有 r 值。我已根据您提供的数据更新了答案以显示结果。
看来我的实际虚拟数据是一个边缘情况，没有出现在我的实际数据中。对于脚本的问题，在具有p = 0 和r = 0 的第4 和第5 d 之间的m = 1 添加几行，您将看到问题。
我已经在我的回答中编辑了脚本来解决这个问题。

【解决方案3】：

使用下面的脚本设法让这个工作在可接受的非常小的误差范围内工作。

它通过检查前面的行来查看是否有多余的r，如果有，则将其添加到当前行。然后在其输出上运行相同的过程，有效地将任何多余的内容及时向后移动两行。这涵盖了现实世界数据中的大多数情况。

最后，它会检查是否有剩余的多余部分，并找到具有超过特定阈值的p 值的最新行的时间戳，以将多余部分移入。如果事先没有带有p > r 的行（例如，收到的第一个数据周期一开始就有这个问题），它会在未来获取最近的数据。

with t as(
select *
      ,case when lead(p - r) over (partition by m order by d) < 0
            then case when p - r < 0
                        then p
                        else r
                        end
                 + abs(lead(p - r,1,0) over (partition by m order by d))
            else case when p - r >= 0
                        then r
                        else r  + (p - r)
                        end
            end
as r1
from @t
)
,b2 as(
select *
      ,case when lead(p - r1) over (partition by m order by d) < 0
            then case when p - r1 < 0
                        then p
                        else r1
                        end
                 + abs(lead(p - r1,1,0) over (partition by m order by d))
            else case when p - r1 >= 0
                        then r1
                        else r1 + (p - r1)
                        end
            end
        as r2
from t
)
,d as(
select *
      ,case when p - r2 < 0
            then case when max(case when p - r2 > 30 then d else '19000101' end) over (partition by m order by d rows between unbounded preceding and 1 preceding) = '19000101'
                        then min(case when p - r2 > 30 then d else '29990101' end) over (partition by m order by d rows between 1 following and unbounded following)
                        else max(case when p - r2 > 30 then d else '19000101' end) over (partition by m order by d rows between unbounded preceding and 1 preceding)
                        end
            else null
            end as drep
from b2
)
,dover as
(
select drep
      ,m
      ,sum(p) as p
      ,sum(r2 - p) as r3
from d
where drep is not null
group by drep
        ,m
)
,f as(
select d.*
    ,dover.r3
    ,case when d.drep is null
        then d.r2 + isnull(dover.r3 /*- dover.p*/,0) -- abs(isnull(dover.p - (dover.r3 - dover.p),0))
        else d.r2 - (d.r2 - d.p)
        end as ra
from d
    left join dover
        on d.m = dover.m
            and d.d = dover.drep
)
select d,m,p,r,ra
from f
order by m,d

【讨论】：