如何使用嵌套在 OR 中的多个 AND 条件来索引 SQL答案

【问题标题】：How to Index SQL with multiple AND conditions nested inside OR如何使用嵌套在 OR 中的多个 AND 条件来索引 SQL
【发布时间】：2018-11-02 10:03:44
【问题描述】：

我想加快下面的sql（成本是19685.75）。我可以索引这个在 WHERE 语句中包含多个复杂嵌套 AND 条件并结合 OR 的 sql 吗？

SELECT DISTINCT
    ON ("crawler_url"."url") U0."id"
FROM "characteristics_text" U0 LEFT OUTER
JOIN "characteristics_text_urls"
    ON (U0."id" = "characteristics_text_urls"."text_id") LEFT OUTER
JOIN "crawler_url"
    ON ("characteristics_text_urls"."url_id" = "crawler_url"."id")
WHERE ( 
    (
        U0."publication_date" BETWEEN '2018-01-01' AND '2018-12-31'
        AND EXTRACT('month' FROM U0."publication_date") = 10
    )
        OR 
    (
        U0."publication_date" IS NULL
        AND U0."lastmod" BETWEEN '2018-01-01' AND '2018-12-31'
        AND EXTRACT('month' FROM U0."lastmod") = 10
    )
        OR 
    (
        U0."publication_date" IS NULL
        AND U0."lastmod" IS NULL
        AND U0."created_at" BETWEEN '2018-01-01 00:00:00+08:00' AND '2018-12-31 23:59:59.999999+08:00'
        AND EXTRACT('month' FROM U0."created_at" AT TIME ZONE 'Asia/Hong_Kong') = 10
    )
        OR 
    (
        U0."publication_date" >= '2018-08-01'
        AND U0."publication_date" < '2018-10-31'
    )
        OR 
    (
        U0."publication_date" IS NULL
        AND U0."lastmod" >='2018-08-01'
        AND U0."lastmod" < '2018-10-31'
    )
        OR 
    (
        U0."publication_date" IS NULL
        AND U0."lastmod" IS NULL
        AND U0."created_at" >= '2018-07-31 16:00:00+00:00'
        AND U0."created_at" < '2018-10-30 16:00:00+00:00'
    ) 
)
ORDER BY  "crawler_url"."url" ASC, U0."created_at" DESC

表格文本包含以下字段和索引（其他一些字段未显示）

                                            Table "public.characteristics_text"                                                                                                     
            Column         |           Type           |                             Modifiers                                                                                        
    ------------------------+--------------------------+-------------------------------------------------------------------                                                           
    id                     | integer                  | not null default nextval('characteristics_text_id_seq'::regclass)
    text                   | text                     | 
    created_at             | timestamp with time zone | not null
    lastmod                | date                     | 
    publication_date       | date                     | 
    Indexes:
        "characteristics_text_pkey" PRIMARY KEY, btree (id)
        "characteristics_text_fde81f11" btree (created_at)
        "characteristics_text_lastmod_3bff34c2_uniq" btree (lastmod)
        "characteristics_text_publication_date_772c1bda_uniq" btree (publication_date)
        "characteristics_text_publication_date_c6311385_idx" btree (publication_date, lastmod, created_at)

我为 created_at、lastmod 和 publication_date 添加了三个单一索引；以及这些字段的一个多列索引。

但在 postgres EXPAIN 查询中，这个 where 子句仍在使用 Seq Scan 而不是 Index Scan。

->  Seq Scan on characteristics_text u0  (cost=0.00..19685.75 rows=14535 width=12)
    Filter: (
            (
                (publication_date >= '2018-01-01'::date) AND 
                (publication_date <= '2018-12-31'::date) AND 
                (
                        date_part(
                            'month'::text, (publication_date)::timestamp without time zone
                ) = 10::double precision)
            ) OR 

                ((publication_date IS NULL) AND (lastmod >= '2018-01-01'::date) AND (lastmod <= '2018-12-31'::date) AND (date_part('month'::text, (lastmod)::timestamp without time zone) = 10::double precision)) OR ((publication_date IS NULL) AND (lastmod IS NULL) AND (created_at >= '2017-12-31 16:00:00+00'::timestamp with time zone) AND (created_at <= '2018-12-31 15:59:59.999999+00'::timestamp with time zone) AND (date_part('month'::text, timezone('Asia/Hong_Kong'::text, created_at)) = 10::double precision)) OR ((publication_date >= '2018-08-01'::date) AND (publication_date < '2018-10-31'::date)) OR ((publication_date IS NULL) AND (lastmod >= '2018-08-01'::date) AND (lastmod < '2018-10-31'::date)) OR ((publication_date IS NULL) AND (lastmod IS NULL) AND (created_at >= '2018-07-31 16:00:00+00'::timestamp with time zone) AND (created_at < '2018-10-30 16:00:00+00'::timestamp with time zone))
)

我的问题是：
1. 是否可以让 postgres 对这个复杂的 SELECT 子句使用索引扫描？
2. 是否需要为每个 AND 子句创建一个多列索引？例如在这个原因中创建索引(publication_date, lastmod)？

    (
        U0."publication_date" IS NULL
        AND U0."lastmod" BETWEEN '2018-01-01' AND '2018-12-31'
        AND EXTRACT('month' FROM U0."lastmod") = 10
    )

索引是否适用于搜索 IS NULL？搜索 IS NULL 的字段是否需要索引？

2018 年 4 月更新

当我尝试通过逐个测试字段来最小化查询时，字段publication_date 和last_mod 会分别触发索引扫描，而created_at 则不能：

是因为created_at 是时间戳吗？但是为什么索引不适用于时间戳？

explain SELECT DISTINCT
    ON ("crawler_url"."url") U0."id"
FROM "characteristics_text" U0 LEFT OUTER
JOIN "characteristics_text_urls"
    ON (U0."id" = "characteristics_text_urls"."text_id") LEFT OUTER
JOIN "crawler_url"
    ON ("characteristics_text_urls"."url_id" = "crawler_url"."id")
WHERE ( 
(
        U0."created_at" BETWEEN '2018-01-01 00:00:00+08:00' AND '2018-12-31 23:59:59.999999+08:00'
        AND EXTRACT('month' FROM U0."created_at" AT TIME ZONE 'Asia/Hong_Kong') = 10
    )   
)
ORDER BY  "crawler_url"."url" ASC, U0."created_at" DESC



Unique  (cost=18004.05..18006.01 rows=393 width=86)
->  Sort  (cost=18004.05..18005.03 rows=393 width=86)
        Sort Key: crawler_url.url, u0.created_at
        ->  Nested Loop Left Join  (cost=0.71..17987.11 rows=393 width=86)
            ->  Nested Loop Left Join  (cost=0.42..17842.25 rows=393 width=16)
                    ->  Seq Scan on characteristics_text u0  (cost=0.00..15467.37 rows=393 width=12)
                        Filter: ((created_at >= '2017-12-31 16:00:00+00'::timestamp with time zone) AND (created_at <= '2018-12-31 15:59:59.999999+00'::timestamp with time zone) AND (date_part('month'::text, timezone('Asia/Hong_Kong'::text, created_at)) = 10::double precision))
                    ->  Index Scan using characteristics_text_urls_65eb77fe on characteristics_text_urls  (cost=0.42..6.03 rows=1 width=8)
                        Index Cond: (u0.id = text_id)
            ->  Index Scan using crawler_url_pkey on crawler_url  (cost=0.29..0.36 rows=1 width=78)
                    Index Cond: (characteristics_text_urls.url_id = id)

publication_date似乎触发了索引扫描：

(
    U0."publication_date" IS NULL
    AND U0."lastmod" >='2018-08-01'
    AND U0."lastmod" < '2018-10-31'
)


Unique  (cost=17053.26..17085.63 rows=6473 width=86)
->  Sort  (cost=17053.26..17069.44 rows=6473 width=86)
        Sort Key: crawler_url.url, u0.created_at
        ->  Nested Loop Left Join  (cost=11130.73..16643.51 rows=6473 width=86)
            ->  Hash Right Join  (cost=11130.44..14257.63 rows=6473 width=16)
                    Hash Cond: (characteristics_text_urls.text_id = u0.id)
                    ->  Seq Scan on characteristics_text_urls  (cost=0.00..1858.01 rows=120601 width=8)
                    ->  Hash  (cost=11049.53..11049.53 rows=6473 width=12)
                        ->  Bitmap Heap Scan on characteristics_text u0  (cost=186.95..11049.53 rows=6473 width=12)
                                Recheck Cond: ((publication_date IS NULL) AND (lastmod >= '2018-08-01'::date) AND (lastmod < '2018-10-31'::date))
                                ->  Bitmap Index Scan on characteristics_text_publication_date_c6311385_idx  (cost=0.00..185.33 rows=6473 width=0)
                                    Index Cond: ((publication_date IS NULL) AND (lastmod >= '2018-08-01'::date) AND (lastmod < '2018-10-31'::date))
            ->  Index Scan using crawler_url_pkey on crawler_url  (cost=0.29..0.36 rows=1 width=78)
                    Index Cond: (characteristics_text_urls.url_id = id)

【问题讨论】：

首先使用您的 WHERE 条件之一将您的查询减少到该表上的 SELECT。看看它是否使用索引。如果是这样，请继续添加，直到找出其中哪一个导致 seq 扫描，然后从那里开始。
@eurotrash 似乎计划器自动切换到created_at 的 seq 扫描，请参阅上面更新的 4nov2018 部分
两个想法。首先摆脱对EXTRACT 的调用，这已经是无法与您的索引一起使用的东西。你应该只剩下U0."created_at" BETWEEN...。其次，您全年都在寻找created_at。表中的记录占很大比例吗？如果是这样，那可能就是它不使用索引的原因。如果不是，我强烈怀疑这与时区有关。我自己从不使用时区，因此无法在此提供建议，但我知道涉及时区的查询可能并不总是按预期使用索引。
@eurotrash 你是对的，2018 年全年有 100,000 条记录中的 60%。我实际上只想要 10 月的数据。我写了这个奇怪的EXTRACT 语句，因为它是从django 函数查找月份docs.djangoproject.com/en/dev/ref/models/querysets/#month 生成的。应该使用BETWEEN '2018-10-01 00:00:00+08:00' AND '2018-12-31 23:59:59.999999+08:00'

标签： sql postgresql performance indexing query-planner

【解决方案1】：

好的，全表扫描（seq_scan）实际上可以比多次索引扫描更快。这取决于您过滤条件的特定“选择性”。

首先，您的WHERE 子句有六个过滤条件，即ORed。这意味着如果你想使用索引，PostgreSQL 需要使用它 6 次，然后执行“索引 OR”来合并结果。那可能不便宜。

因此，首先，您需要了解 6 种过滤条件中的每一种的预期选择性是多少。也就是说，与表的总行数相比，选择了多少行。做它；几个简单的 SQL 查询将为您提供答案。在这里发布答案。

现在，如果所有六个选择性的总和超过 5%，那么全表扫描（您现在拥有的算法）会更快。不要为索引而烦恼。

否则，以下索引会有所帮助：

create index ix1 on characteristics_text (
  publication_date, 
  lastmod,
  created_at,
  1);

【讨论】：

是的，2018 年全年有 100,000 条记录中的 60%。从BEWEEN全年改为BETWEEN '2018-10-01 00:00:00+08:00' AND '2018-12-31 23:59:59.999999+08:00'进行索引扫描。

【解决方案2】：

我怀疑您是否会在这里获得有用的索引。您可能会考虑将这个查询分成 4 或 5 个部分，然后使用 UNION 再次将结果粘贴在一起。（UNION 将删除欺骗，WHILE UNION ALL 将返回所有行）。

UNION 是一项相当昂贵的操作，因此需要考虑返回多少行。如果删除足够多的行，使用索引可能会比 UNION 损失的效率更高。如果返回了许多行，则您当前的表单将达到预期效果。

【讨论】：

【解决方案3】：

2018 年全年 100,000 条记录中有 60%，这使得数据库使用了 seq 扫描。从 BEWEEN 全年更改为仅一个月即可进行索引扫描。

  AND U0."created_at" >= '2018-10-01 00:00:00+00:00'
    AND U0."created_at" <= '2018-10-31 23:59:59.999999+00:00')

完整的 SQL：

SELECT DISTINCT
    ON ("crawler_url"."url") U0."id"
FROM "characteristics_text" U0 LEFT OUTER
JOIN "characteristics_text_urls"
    ON (U0."id" = "characteristics_text_urls"."text_id") LEFT OUTER
JOIN "crawler_url"
    ON ("characteristics_text_urls"."url_id" = "crawler_url"."id")
WHERE (
        (U0."publication_date" >= '2018-10-01'
        AND U0."publication_date" <= '2018-11-01')

        OR (U0."publication_date" IS NULL
        AND U0."lastmod" >= '2018-10-01'
        AND U0."lastmod" <= '2018-11-01'
        )

        OR 

        (U0."publication_date" IS NULL
        AND U0."lastmod" IS NULL
        AND U0."created_at" >= '2018-10-01 00:00:00+00:00'
        AND U0."created_at" <= '2018-10-31 23:59:59.999999+00:00')

        OR 

        (U0."publication_date" >= '2018-08-01'
        AND U0."publication_date" < '2018-10-31')

        OR 

        (U0."publication_date" IS NULL
        AND U0."lastmod" >= '2018-08-01'
        AND U0."lastmod" < '2018-10-31')

        OR 

        (U0."publication_date" IS NULL
        AND U0."lastmod" IS NULL
        AND U0."created_at" >= '2018-07-31 16:00:00+00:00'
        AND U0."created_at" < '2018-10-30 16:00:00+00:00')
    )
ORDER BY  "crawler_url"."url" ASC

EXPLAIN 语句显示每个 AND 条件的索引扫描，因此总共有 6 个索引扫描。

Unique  (cost=22885.16..22962.39 rows=15446 width=88)
->  Sort  (cost=22885.16..22923.77 rows=15446 width=88)
        Sort Key: crawler_url.url
        ->  Hash Right Join  (cost=18669.29..21068.51 rows=15446 width=88)
            Hash Cond: (crawler_url.id = characteristics_text_urls.url_id)
            ->  Seq Scan on crawler_url  (cost=0.00..1691.88 rows=55288 width=88)
            ->  Hash  (cost=18476.21..18476.21 rows=15446 width=8)
                    ->  Hash Right Join  (cost=14982.09..18476.21 rows=15446 width=8)
                        Hash Cond: (characteristics_text_urls.text_id = u0.id)
                        ->  Seq Scan on characteristics_text_urls  (cost=0.00..1907.25 rows=115525 width=8)
                        ->  Hash  (cost=14789.01..14789.01 rows=15446 width=4)
                                ->  Bitmap Heap Scan on characteristics_text u0  (cost=516.57..14789.01 rows=15446 width=4)
                                    Recheck Cond: (((publication_date >= '2018-10-01'::date) AND (publication_date <= '2018-11-01'::date)) OR ((publication_date IS NULL) AND (lastmod >= '2018-10-01'::date) AND (lastmod <= '2018-11-01'::date)) OR ((publication_date IS NULL) AND (lastmod IS NULL) AND (created_at >= '2018-10-01 00:00:00+00'::timestamp with time zone) AND (created_at <= '2018-10-31 23:59:59.999999+00'::timestamp with time zone)) OR ((publication_date >= '2018-08-01'::date) AND (publication_date < '2018-10-31'::date)) OR ((publication_date IS NULL) AND (lastmod >= '2018-08-01'::date) AND (lastmod < '2018-10-31'::date)) OR ((publication_date IS NULL) AND (lastmod IS NULL) AND (created_at >= '2018-07-31 16:00:00+00'::timestamp with time zone) AND (created_at < '2018-10-30 16:00:00+00'::timestamp with time zone)))
                                    ->  BitmapOr  (cost=516.57..516.57 rows=16081 width=0)
                                            ->  Bitmap Index Scan on characteristics_text_publication_date_772c1bda_uniq  (cost=0.00..4.53 rows=11 width=0)
                                                Index Cond: ((publication_date >= '2018-10-01'::date) AND (publication_date <= '2018-11-01'::date))
                                            ->  Bitmap Index Scan on characteristics_text_publication_date_c6311385_idx  (cost=0.00..6.49 rows=166 width=0)
                                                Index Cond: ((publication_date IS NULL) AND (lastmod >= '2018-10-01'::date) AND (lastmod <= '2018-11-01'::date))
                                            ->  Bitmap Index Scan on characteristics_text_publication_date_c6311385_idx  (cost=0.00..14.61 rows=413 width=0)
                                                Index Cond: ((publication_date IS NULL) AND (lastmod IS NULL) AND (created_at >= '2018-10-01 00:00:00+00'::timestamp with time zone) AND (created_at <= '2018-10-31 23:59:59.999999+00'::timestamp with time zone))
                                            ->  Bitmap Index Scan on characteristics_text_publication_date_772c1bda_uniq  (cost=0.00..74.61 rows=3419 width=0)
                                                Index Cond: ((publication_date >= '2018-08-01'::date) AND (publication_date < '2018-10-31'::date))
                                            ->  Bitmap Index Scan on characteristics_text_publication_date_c6311385_idx  (cost=0.00..108.20 rows=3503 width=0)
                                                Index Cond: ((publication_date IS NULL) AND (lastmod >= '2018-08-01'::date) AND (lastmod < '2018-10-31'::date))
                                            ->  Bitmap Index Scan on characteristics_text_publication_date_c6311385_idx  (cost=0.00..284.95 rows=8569 width=0)
                                                Index Cond: ((publication_date IS NULL) AND (lastmod IS NULL) AND (created_at >= '2018-07-31 16:00:00+00'::timestamp with time zone) AND (created_at < '2018-10-30 16:00:00+00'::timestamp with time zone))

【讨论】：