每行查询最后 N 个相关行

Question

我有以下查询，它为每个 station 获取最新 N observations 的 id：

SELECT id
FROM (
  SELECT station_id, id, created_at,
         row_number() OVER(PARTITION BY station_id
                           ORDER BY created_at DESC) AS rn
  FROM (
      SELECT station_id, id, created_at
      FROM observations
  ) s
) s
WHERE rn <= #{n}
ORDER BY station_id, created_at DESC;

我在id、station_id、created_at 上有索引。

这是我想出的唯一解决方案，可以为每个站点获取多条记录。但是它很慢（81000 条记录的表需要 154.0 毫秒）。

如何加快查询速度？

Answer 1

仅当您不需要查询最新的实时数据时，这才是一个很好的答案。

准备（需要 postgresql 9.3）

drop materialized view test;
create materialized view test as select * from (
  SELECT station_id, id, created_at,
      row_number() OVER(
          PARTITION BY station_id
          ORDER BY created_at DESC
      ) as rn
  FROM (
      SELECT
          station_id,
          id,
          created_at
      FROM observations
  ) s
 ) q WHERE q.rn <= 100 -- use a value that will be your max limit number for further queries
ORDER BY station_id, rn DESC ;


create index idx_test on test(station_id,rn,created_at);

如何查询数据：

select * from test where rn<10 order by station_id,created_at;

您的原始查询在我的机器上是 281 毫秒，而这个新查询是 15 毫秒。

如何使用新数据更新视图：

refresh materialized view test;

我有另一个解决方案，它不需要物化视图并且可以处理实时的最新数据。但鉴于您不需要最新数据，这种物化视图的效率要高得多。

Answer 2

索引

首先，多列索引会有所帮助：

CREATE INDEX observations_special_idx
ON observations(station_id, created_at DESC, id)

created_at DESC 稍微好一点，但如果没有DESC，索引仍会以几乎相同的速度向后扫描。

假设created_at 定义为NOT NULL，否则在索引和查询中考虑DESC NULLS LAST：

• Sort by column ASC, but NULL values first?

最后一列id 仅在您从中获得index-only scan 时才有用，如果您不断添加大量新行，这可能不起作用。在这种情况下，请从索引中删除 id。

更简单的查询（仍然很慢）

简化您的查询，内部子选择无济于事：

SELECT id
FROM  (
  SELECT station_id, id, created_at
       , row_number() OVER (PARTITION BY station_id
                            ORDER BY created_at DESC) AS rn
  FROM   observations
  ) s
WHERE  rn <= #{n}  -- your limit here
ORDER  BY station_id, created_at DESC;

应该会快一点，但还是很慢。

快速查询

• 假设您有相对少数个观测站和相对许多个观测站。
• 还假设station_id id 定义为NOT NULL。

要真正快速，您需要松散索引扫描（尚未在 Postgres 中实现）。相关答案：

• Optimize GROUP BY query to retrieve latest row per user

如果您有一个单独的 stations 表（这似乎很可能），您可以使用 JOIN LATERAL 模拟它（Postgres 9.3+）：

SELECT o.id
FROM   stations s
CROSS  JOIN LATERAL (
   SELECT o.id
   FROM   observations o
   WHERE  o.station_id = s.station_id  -- lateral reference
   ORDER  BY o.created_at DESC
   LIMIT  #{n}  -- your limit here
   ) o
ORDER  BY s.station_id, o.created_at DESC;

如果您没有stations 的表，则最好创建并维护一个表。可能添加外键引用以强制关系完整性。

如果这不是一个选项，您可以即时提取这样的表。简单的选择是：

SELECT DISTINCT station_id FROM observations;
SELECT station_id FROM observations GROUP BY 1;

但是两者都需要顺序扫描并且速度很慢。使用 recursive CTE 使 Postgres 使用上述索引（或任何以 station_id 为前导列的 btree 索引）：

WITH RECURSIVE stations AS (
   (                  -- extra pair of parentheses ...
   SELECT station_id
   FROM   observations
   ORDER  BY station_id
   LIMIT  1
   )                  -- ... is required!
   UNION ALL
   SELECT (SELECT o.station_id
           FROM   observations o
           WHERE  o.station_id > s.station_id
           ORDER  BY o.station_id
           LIMIT  1)
   FROM   stations s
   WHERE  s.station_id IS NOT NULL  -- serves as break condition
   )
SELECT station_id
FROM   stations
WHERE  station_id IS NOT NULL;      -- remove dangling row with NULL

在上述简单查询中将其用作stations 表的直接替换：

WITH RECURSIVE stations AS (
   (
   SELECT station_id
   FROM   observations
   ORDER  BY station_id
   LIMIT  1
   )
   UNION ALL
   SELECT (SELECT o.station_id
           FROM   observations o
           WHERE  o.station_id > s.station_id
           ORDER  BY o.station_id
           LIMIT  1)
   FROM   stations s
   WHERE  s.station_id IS NOT NULL
   )
SELECT o.id
FROM   stations s
CROSS  JOIN LATERAL (
   SELECT o.id, o.created_at
   FROM   observations o
   WHERE  o.station_id = s.station_id
   ORDER  BY o.created_at DESC
   LIMIT  #{n}  -- your limit here
   ) o
WHERE  s.station_id IS NOT NULL
ORDER  BY s.station_id, o.created_at DESC;

这应该仍然比您的速度快 几个数量级。

db小提琴here
_旧sqlfiddle