在 WHERE 语句中使用 OR 进行慢速 JOIN 查询答案

【问题标题】：Slow JOIN query with OR in WHERE statement在 WHERE 语句中使用 OR 进行慢速 JOIN 查询
【发布时间】：2011-03-18 14:25:17
【问题描述】：

这是我的问题的一个简单示例：

CREATE TABLE test1 (id SERIAL, key TEXT UNIQUE, value TEXT);
CREATE TABLE test2 (id SERIAL, key TEXT UNIQUE, value TEXT);

INSERT INTO test1 (key, value) 
SELECT i::TEXT, 'ABC' || i::TEXT 
FROM generate_series(0, 1000000) AS i;

INSERT INTO test2 (key, value) 
SELECT i::TEXT, 'ABC' || (i+1000)::TEXT 
FROM generate_series(0,  600000) AS i;

INSERT INTO test2 (key, value) 
SELECT i::TEXT, 'ABC' || (i+1000)::TEXT 
FROM generate_series(1000000, 1200000) AS i;

CREATE INDEX test1_key ON test1 (key);
CREATE INDEX test1_value ON test1 (value);
CREATE INDEX test2_key ON test2 (key);
CREATE INDEX test2_value ON test2 (value);

VACUUM FULL VERBOSE ANALYZE test1;
VACUUM FULL VERBOSE ANALYZE test2;

这是我当前正在使用的查询，但耗时超过 6 秒。

EXPLAIN ANALYZE 
SELECT test1.key AS key1, test1.value AS value1, 
       test2.key AS key2, test2.value AS value2
FROM test1 
LEFT OUTER JOIN test2 ON (test1.key = test2.key)
WHERE test1.value = 'ABC1234' OR test2.value = 'ABC1234';

 key1 | value1  | key2 | value2
------+---------+------+---------
 234  | ABC234  | 234  | ABC1234
 1234 | ABC1234 | 1234 | ABC2234
(2 rows)

                                                         QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------
 Hash Left Join  (cost=27344.05..79728.10 rows=2 width=32) (actual time=5428.635..6097.098 rows=2 loops=1)
   Hash Cond: (test1.key = test2.key)
   Filter: ((test1.value = 'ABC1234'::text) OR (test2.value = 'ABC1234'::text))
   ->  Seq Scan on test1  (cost=0.00..16321.01 rows=1000001 width=15) (actual time=0.009..1057.315 rows=1000001 loops=1)
   ->  Hash  (cost=13047.02..13047.02 rows=800002 width=17) (actual time=2231.964..2231.964 rows=800002 loops=1)
         Buckets: 65536  Batches: 2  Memory Usage: 14551kB
         ->  Seq Scan on test2  (cost=0.00..13047.02 rows=800002 width=17) (actual time=0.010..980.232 rows=800002 loops=1)
 Total runtime: 6109.042 ms
(8 rows)

在这两个表中，只有很少的数据集符合要求，但似乎没有观察到这一事实。我可以改为使用这样的查询：

EXPLAIN ANALYZE 
SELECT coalesce(test1.key, test3.key1) AS key1, coalesce(test1.value, test3.value1) AS value1,
       coalesce(test2.key, test3.key2) AS key2, coalesce(test2.value, test3.value2) AS value2
FROM (SELECT test1.key AS key1, test1.value AS value1, 
             test2.key AS key2, test2.value AS value2
      FROM (SELECT key, value FROM test1 WHERE value = 'ABC1234') AS test1
      FULL JOIN (SELECT key, value FROM test2 WHERE value = 'ABC1234') AS test2
      ON (test1.key = test2.key)) AS test3
LEFT OUTER JOIN test1 ON (test1.key = test3.key2)
LEFT OUTER JOIN test2 ON (test2.key = test3.key1)
WHERE test1.key IS NOT NULL;

 key1 | value1  | key2 | value2
------+---------+------+---------
 1234 | ABC1234 | 1234 | ABC2234
 234  | ABC234  | 234  | ABC1234
(2 rows)

                                                                QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------
 Nested Loop Left Join  (cost=0.00..33.56 rows=1 width=64) (actual time=0.075..0.083 rows=1 loops=1)
   ->  Nested Loop  (cost=0.00..25.19 rows=1 width=47) (actual time=0.066..0.072 rows=1 loops=1)
         ->  Nested Loop Left Join  (cost=0.00..16.80 rows=1 width=32) (actual time=0.051..0.054 rows=1 loops=1)
               ->  Index Scan using test2_value_key on test2  (cost=0.00..8.41 rows=1 width=17) (actual time=0.026..0.027 rows=1 loops=1)
                     Index Cond: (value = 'ABC1234'::text)
               ->  Index Scan using test1_key on test1  (cost=0.00..8.38 rows=1 width=15) (actual time=0.020..0.020 rows=0 loops=1)
                     Index Cond: (public.test1.key = public.test2.key)
                     Filter: (public.test1.value = 'ABC1234'::text)
         ->  Index Scan using test1_key on test1  (cost=0.00..8.38 rows=1 width=15) (actual time=0.011..0.013 rows=1 loops=1)
               Index Cond: ((public.test1.key IS NOT NULL) AND (public.test1.key = public.test2.key))
   ->  Index Scan using test2_key on test2  (cost=0.00..8.36 rows=1 width=17) (actual time=0.001..0.001 rows=0 loops=1)
         Index Cond: (public.test2.key = public.test1.key)
 Total runtime: 0.139 ms

下面的查询比较简单，但还是太慢了：

EXPLAIN ANALYZE
SELECT test1.key AS key1, test1.value AS value1, 
       test2.key AS key2, test2.value AS value2
FROM test1 
LEFT OUTER JOIN test2 ON (test1.key = test2.key)
WHERE test1.value = 'ABC1234'
   OR EXISTS (SELECT 1 FROM test2 t WHERE t.key = test1.key AND t.value = 'ABC1234');

 key1 | value1  | key2 | value2
------+---------+------+---------
 1234 | ABC1234 | 1234 | ABC2234
 234  | ABC234  | 234  | ABC1234
(2 rows)

                                                               QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------
 Merge Left Join  (cost=0.00..8446826.32 rows=500001 width=32) (actual time=615.706..1651.370 rows=2 loops=1)
   Merge Cond: (test1.key = test2.key)
   ->  Index Scan using test1_key on test1  (cost=0.00..8398983.25 rows=500001 width=15) (actual time=28.449..734.567 rows=2 loops=1)
         Filter: ((value = 'ABC1234'::text) OR (alternatives: SubPlan 1 or hashed SubPlan 2))
         SubPlan 1
           ->  Index Scan using test2_key on test2 t  (cost=0.00..8.36 rows=1 width=0) (never executed)
                 Index Cond: (key = $0)
                 Filter: (value = 'ABC1234'::text)
         SubPlan 2
           ->  Index Scan using test2_value on test2 t  (cost=0.00..8.37 rows=1 width=7) (actual time=0.376..0.380 rows=1 loops=1)
                 Index Cond: (value = 'ABC1234'::text)
   ->  Index Scan using test2_key on test2  (cost=0.00..39593.05 rows=800002 width=17) (actual time=0.019..498.456 rows=348894 loops=1)
 Total runtime: 1651.453 ms
(13 rows)

所以我的问题是：是否有一个简单的查询会导致类似的快速执行计划，例如第二个查询，或者可能是一个索引或对计划者的某种提示。

（我知道对于那个例子来说，只有一个包含两个值的表是合理的。但实际上这些表更复杂，并且表方案不能那么容易地更改。）

PostgreSQL Version: 9.0.3
shared_buffers = 64MB
effective_cache_size = 32MB
work_mem = 16MB
maintenance_work_mem = 32MB
temp_buffers = 8MB
wal_buffers= 1MB

编辑：正如 Kipotlov 所建议的，这里是 UNION 版本。为什么普通的OR查询没有选择这么好的方案？

EXPLAIN ANALYZE
SELECT test1.key AS key1, test1.value AS value1, 
       test2.key AS key2, test2.value AS value2
FROM test1 
LEFT OUTER JOIN test2 ON (test1.key = test2.key)
WHERE test1.value = 'ABC1234'
UNION
SELECT test1.key AS key1, test1.value AS value1, 
       test2.key AS key2, test2.value AS value2
FROM test1 
LEFT OUTER JOIN test2 ON (test1.key = test2.key)
WHERE test2.value = 'ABC1234';

 key1 | value1  | key2 | value2
------+---------+------+---------
 1234 | ABC1234 | 1234 | ABC2234
 234  | ABC234  | 234  | ABC1234
(2 rows)

                                                                   QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------
 Unique  (cost=33.64..33.66 rows=2 width=32) (actual time=0.114..0.119 rows=2 loops=1)
   ->  Sort  (cost=33.64..33.64 rows=2 width=32) (actual time=0.111..0.113 rows=2 loops=1)
         Sort Key: public.test1.key, public.test1.value, public.test2.key, public.test2.value
         Sort Method:  quicksort  Memory: 17kB
         ->  Append  (cost=0.00..33.63 rows=2 width=32) (actual time=0.046..0.097 rows=2 loops=1)
               ->  Nested Loop Left Join  (cost=0.00..16.81 rows=1 width=32) (actual time=0.044..0.050 rows=1 loops=1)
                     ->  Index Scan using test1_value_key on test1  (cost=0.00..8.44 rows=1 width=15) (actual time=0.023..0.024 rows=1 loops=1)
                           Index Cond: (value = 'ABC1234'::text)
                     ->  Index Scan using test2_key on test2  (cost=0.00..8.36 rows=1 width=17) (actual time=0.014..0.016 rows=1 loops=1)
                           Index Cond: (public.test1.key = public.test2.key)
               ->  Nested Loop  (cost=0.00..16.80 rows=1 width=32) (actual time=0.036..0.041 rows=1 loops=1)
                     ->  Index Scan using test2_value_key on test2  (cost=0.00..8.41 rows=1 width=17) (actual time=0.019..0.020 rows=1 loops=1)
                           Index Cond: (value = 'ABC1234'::text)
                     ->  Index Scan using test1_key on test1  (cost=0.00..8.38 rows=1 width=15) (actual time=0.013..0.015 rows=1 loops=1)
                           Index Cond: (public.test1.key = public.test2.key)
 Total runtime: 0.173 ms
(16 rows)

【问题讨论】：

您是否尝试使用 2 个查询和 'UNION' ？第一个查询使用第一个 where 子句 (test1.value)，第二个查询使用第二个 where 子句 (test2.value)。不知道会不会更快..
@Kipotlov 我添加了 UNION 查询。它和第二个查询一样快，但我不确定我是否可以将它用于我的实际问题。任何想法为什么正常的 OR 查询更喜欢顺序扫描？

标签： sql database performance postgresql join

【解决方案1】：

首先，感谢您提出非常详细的问题。很少有人会在提问之前对他们的问题进行如此详细的研究。

我一直在考虑这个问题，问题似乎是 PostgreSQL 想要加入 所有行，因为来自 test1 的每个不匹配行都可能在 test2 中匹配——反之亦然.

解决方案是强制规划器分两步执行查询。一种方法是您已经尝试过的大型 UNION 查询——强制它在单独的查询中考虑每个表达式。

另一种方法是强制规划器首先找到匹配的键，然后执行连接，这样就不会有歧义了：

EXPLAIN ANALYZE
SELECT test1.key AS key1, test1.value AS value1, 
       test2.key AS key2, test2.value AS value2
FROM (
    SELECT key FROM test1 WHERE value='ABC1234'
    UNION SELECT key FROM test2 WHERE value='ABC1234'
) AS matching_keys
INNER JOIN test1 USING (key)
LEFT OUTER JOIN test2 USING (key);

 Nested Loop Left Join  (cost=16.84..34.44 rows=2 width=32) (actual time=0.211..0.280 rows=2 loops=1)
   ->  Nested Loop  (cost=16.84..33.65 rows=2 width=15) (actual time=0.175..0.212 rows=2 loops=1)
         ->  Unique  (cost=16.84..16.85 rows=2 width=6) (actual time=0.132..0.136 rows=2 loops=1)
               ->  Sort  (cost=16.84..16.85 rows=2 width=6) (actual time=0.131..0.132 rows=2 loops=1)
                     Sort Key: public.test1.key
                     Sort Method: quicksort  Memory: 25kB
                     ->  Append  (cost=0.00..16.83 rows=2 width=6) (actual time=0.058..0.110 rows=2 loops=1)
                           ->  Index Scan using test1_value on test1  (cost=0.00..8.42 rows=1 width=6) (actual time=0.056..0.058 rows=1 loops=1)
                                 Index Cond: (value = 'ABC1234'::text)
                           ->  Index Scan using test2_value on test2  (cost=0.00..8.39 rows=1 width=7) (actual time=0.046..0.047 rows=1 loops=1)
                                 Index Cond: (value = 'ABC1234'::text)
         ->  Index Scan using test1_key on test1  (cost=0.00..8.38 rows=1 width=15) (actual time=0.032..0.033 rows=1 loops=2)
               Index Cond: (key = public.test1.key)
   ->  Index Scan using test2_key on test2  (cost=0.00..0.38 rows=1 width=17) (actual time=0.028..0.029 rows=1 loops=2)
         Index Cond: (public.test1.key = key)
 Total runtime: 0.390 ms
(16 rows)

同样，UNION 充当 OR 的角色。不幸的是，这种方法对于像value>'ABC1234' 这样的查询仍然表现不佳。你可以通过提高work_mem 来改进它。我在这里不知所措。

关于你的最后一个问题：

为什么普通的OR查询没有选择这么好的方案？

因为 PostgreSQL 规划器目前缺乏将 OR'ed 表达式拆分为单独的 UNION 查询的能力。有一些注意事项使它比看起来更难。

PostgreSQL 规划器已经相当精细，但到目前为止，利用手动重写 SQL 已经可能实现的优化并不是一个重要的优先事项。

【讨论】：

我不确定是否总是可以通过手动重写进行优化。例如，如果? 是用户提供的值并且条件是value < ?，则UNION 计划对于? = 'ABC1' 更好，但对于? = 'ABC9'，OR 计划更好。只有规划者知道value < 'ABC9' 的匹配项会比value < 'ABC1' 的匹配项多得多。有没有办法消除警告（通过添加 NOT NULL 约束或类似的东西）或者第一个 OR 查询总是使用顺序扫描？
我用另一种方法更新了我的回复。不幸的是，也好不到哪里去。

【解决方案2】：

我不知道哪种方式更好或更快。

但我注意到的第一件事是：您有两个表，每个表中的每个键列 (UNIQUE)。然后你从两个表中获取相同键的数据。

我的意思是，你为什么不在一开始就加入两张桌子，这样你就只需要从一张桌子上得到吗？

【讨论】：

我真正的两张表有更多的列，并且不只是由一个列连接。我只是简化了我的案例，同时仍然得到相同的结果。我不能那么容易地将我真正的两张表合并为一张。