postgresql - 两个大表之间的连接需要很长时间

Question

我确实有两个相当大的表，我需要在它们之间进行日期范围连接。不幸的是，查询需要 12 多个小时。我正在使用在 docker 中运行的 postgresql 10.5，最大。 5GB 内存和多达 12 个 CPU 内核可用。

基本上，在左表中，我确实有一个设备 ID 和一个日期范围列表（从 = Timestamp，到 = ValidUntil）。然后我想加入右表，其中包含所有设备的测量值（传感器数据），以便我只获得位于日期范围之一（来自左表）内的传感器数据。查询：

select
    A.*,
    B."Timestamp" as "PressureTimestamp",
    B."PropertyValue" as "Pressure"
from A
inner join B
    on  B."EquipmentId" =  A."EquipmentId"
    and B."Timestamp"   >= A."Timestamp"
    and B."Timestamp"   <  A."ValidUntil"

不幸的是，这个查询只使用了一个核心，这可能是它运行如此缓慢的原因。有没有办法重写查询以便并行化？

索引：

create index if not exists A_eq_timestamp_validUntil on public.A using btree ("EquipmentId", "Timestamp", "ValidUntil");
create index if not exists B_eq_timestamp on public.B using btree ("EquipmentId", "Timestamp");

表格：

-- contains 332,000 rows
CREATE TABLE A (
    "EquipmentId" bigint,
    "Timestamp" timestamp without time zone,
    "ValidUntil" timestamp without time zone
)
WITH ( OIDS = FALSE )

-- contains 70,000,000 rows
CREATE TABLE B
(
    "EquipmentId" bigint,
    "Timestamp" timestamp without time zone,
    "PropertyValue" double precision
)
WITH ( OIDS = FALSE )

执行计划（解释...输出）：

Nested Loop  (cost=176853.59..59023908.95 rows=941684055 width=48)
  ->  Bitmap Heap Scan on v2_pressure p  (cost=176853.16..805789.35 rows=9448335 width=24)
        Recheck Cond: ("EquipmentId" = 2956235)
        ->  Bitmap Index Scan on v2_pressure_eq  (cost=0.00..174491.08 rows=9448335 width=0)
              Index Cond: ("EquipmentId" = 2956235)"
  ->  Index Scan using v2_prs_eq_timestamp_validuntil on v2_prs prs  (cost=0.42..5.16 rows=100 width=32)
        Index Cond: (("EquipmentId" = 2956235) AND (p."Timestamp" >= "Timestamp") AND (p."Timestamp" < "ValidUntil"))

更新 1： 根据 cmets 修复了索引，这大大提高了性能

Answer 1

索引修正是解决缓慢的首要手段，但它只会在一定程度上有所帮助。鉴于您的表很大，我建议您尝试 Postgres Partition 。它有一些来自 postgres 的内置支持。

但您需要有一些过滤器/分区标准。我在您的查询中看不到任何 where 子句，因此无法建议。可能你可以试试设备ID。这也有助于实现并行性。

Answer 2

-- \i tmp.sql

CREATE TABLE A
        ( equipmentid bigint NOT NULL
        , ztimestamp timestamp without time zone NOT NULL
        , validuntil timestamp without time zone NOT NULL
        , PRIMARY KEY (equipmentid,ztimestamp)
        , UNIQUE (equipmentid,validuntil) -- mustbeunique, since the intervals dont overlap
        ) ;

-- contains 70,000,000 rows
CREATE TABLE B
        ( equipmentid bigint NOT NULL
        , ztimestamp timestamp without time zone NOT NULL
        , propertyvalue double precision
        , PRIMARY KEY (equipmentid,ztimestamp)
        ) ;

INSERT INTO B(equipmentid,ztimestamp,propertyvalue)
SELECT i,t, random()
FROM generate_series(1,1000) i
CROSS JOIN generate_series('2018-09-01','2018-09-30','1day'::interval) t
        ;


INSERT INTO A(equipmentid,ztimestamp,validuntil)
SELECT equipmentid,ztimestamp, ztimestamp+ '7 days'::interval
FROM B
WHERE date_part('dow', ztimestamp) =0
        ;

ANALYZE A;
ANALYZE B;

EXPLAIN
SELECT
    A.*,
    B.ztimestamp AS pressuretimestamp,
    B.propertyvalue AS pressure
FROM A
INNER JOIN B
    ON  B.equipmentid =  A.equipmentid
    AND B.ztimestamp   >= A.ztimestamp
    AND B.ztimestamp   <  A.validuntil
    WHERE A.equipmentid=333 -- I added this, the plan in the question also has a r estriction on Id
        ;

---

以及由此产生的计划：

---

SET
ANALYZE
ANALYZE
                                                 QUERY PLAN                                                 
------------------------------------------------------------------------------------------------------------
 Nested Loop  (cost=0.34..21.26 rows=17 width=40)
   ->  Index Scan using a_equipmentid_validuntil_key on a  (cost=0.17..4.34 rows=5 width=24)
         Index Cond: (equipmentid = 333)
   ->  Index Scan using b_pkey on b  (cost=0.17..3.37 rows=3 width=24)
         Index Cond: ((equipmentid = 333) AND (ztimestamp >= a.ztimestamp) AND (ztimestamp < a.validuntil))
(5 rowSET

---

这是我当前的random_page_cost=1.1;设置

设置为4.0后，我得到了和OP一样的方案：

---

SET
                                                                     QUERY PLAN                                                                     
----------------------------------------------------------------------------------------------------------------------------------------------------
 Nested Loop  (cost=35.13..54561.69 rows=1416136 width=40) (actual time=1.391..1862.275 rows=225540 loops=1)
   ->  Bitmap Heap Scan on aa2  (cost=34.71..223.52 rows=1345 width=24) (actual time=1.173..5.223 rows=1345 loops=1)
         Recheck Cond: (equipmentid = 5)
         Heap Blocks: exact=9
         ->  Bitmap Index Scan on aa2_equipmentid_validuntil_key  (cost=0.00..34.38 rows=1345 width=0) (actual time=1.047..1.048 rows=1345 loops=1)
               Index Cond: (equipmentid = 5)
   ->  Index Scan using bb2_pkey on bb2  (cost=0.42..29.87 rows=1053 width=24) (actual time=0.109..0.757 rows=168 loops=1345)
         Index Cond: ((equipmentid = 5) AND (ztimestamp >= aa2.ztimestamp) AND (ztimestamp < aa2.validuntil))
 Planning Time: 3.167 ms
 Execution Time: 2168.967 ms
(10 rows)