MySQL - 查询获取速度非常慢

【问题标题】：MySQL - query fetching very slowMySQL - 查询获取速度非常慢
【发布时间】：2021-04-12 14:06:40
【问题描述】： 
CREATE TABLE `offers` (
  `id` mediumint(8) unsigned NOT NULL auto_increment,
  `cap_id` varchar(255),
  `deposit_value` varchar(255) default NULL,
  `term` varchar(255) default NULL,
  `annual_mileage` varchar(255) default NULL,
  `finance_type` varchar(255) default NULL,
  `monthly_payment` mediumint default NULL,
  `stock` varchar(255) default NULL,
  PRIMARY KEY (`id`)
) AUTO_INCREMENT=1;

INSERT INTO `offers` (`cap_id`,`deposit_value`,`term`,`annual_mileage`,`finance_type`,`monthly_payment`,`stock`) VALUES ("93897","6","24","12000","B",363,"0"),("90626","1","24","12000","P",810,"0"),("93607","6","36","25000","B",172,"1"),("35877","1","48","10000","B",678,"1"),("77134","3","24","10000","P",454,"0"),("47080","6","36","10000","B",737,"0"),("46392","3","24","30000","P",261,"0"),("21418","3","36","20000","B",390,"1"),("80175","9","48","10000","B",713,"1"),("91487","6","48","12000","P",778,"1");
INSERT INTO `offers` (`cap_id`,`deposit_value`,`term`,`annual_mileage`,`finance_type`,`monthly_payment`,`stock`) VALUES ("18311","9","24","20000","B",384,"0"),("41740","9","24","12000","P",674,"1"),("69030","9","24","10000","P",518,"0"),("53342","3","36","8000","B",746,"1"),("65566","6","36","30000","P",145,"1"),("63172","6","48","5000","P",698,"1"),("79712","1","48","30000","B",330,"1"),("90505","1","36","8000","B",458,"0"),("42393","1","36","20000","B",363,"1"),("70454","9","24","5000","B",673,"1");
INSERT INTO `offers` (`cap_id`,`deposit_value`,`term`,`annual_mileage`,`finance_type`,`monthly_payment`,`stock`) VALUES ("81215","1","48","8000","B",472,"0"),("76538","3","24","15000","B",226,"0"),("05094","9","36","12000","P",721,"1"),("57363","9","48","5000","B",777,"1"),("23233","1","48","12000","B",381,"0"),("40542","3","48","12000","P",610,"0"),("63824","3","24","12000","B",761,"1"),("17686","3","24","5000","P",893,"1"),("57669","9","48","30000","P",805,"0"),("21864","1","24","25000","P",530,"0");
INSERT INTO `offers` (`cap_id`,`deposit_value`,`term`,`annual_mileage`,`finance_type`,`monthly_payment`,`stock`) VALUES ("48360","3","48","12000","P",159,"0"),("88614","3","48","20000","B",730,"0"),("17693","1","24","10000","B",298,"0"),("34049","6","48","20000","B",728,"0"),("15038","9","24","10000","P",720,"1"),("31809","1","36","20000","P",237,"0"),("49277","9","48","25000","P",235,"1"),("54607","1","24","12000","P",661,"1"),("65098","1","48","20000","P",548,"1"),("76440","9","48","10000","P",495,"1");

CREATE TABLE `offers_lowest` (
  `id` mediumint(8) unsigned NOT NULL auto_increment,
  `date` DATE,
  `cap_id` varchar(255),
  `deposit` varchar(255) default NULL,
  `term` varchar(255) default NULL,
  `mileage` varchar(255) default NULL,
  `finance_type` varchar(255) default NULL,
  `lowest_price` mediumint default NULL,
  `stock` varchar(255) default NULL,
  PRIMARY KEY (`id`)
) AUTO_INCREMENT=1;

INSERT INTO `offers_lowest` (`date`,`cap_id`,`deposit`,`term`,`mileage`,`finance_type`,`lowest_price`,`stock`) VALUES ("2021-04-09","93897","6","24","12000","B",363,"0"),("2021-04-010","90626","1","24","12000","P",810,"0"),("2021-04-010","93607","6","36","25000","B",172,"1"),("2021-04-10","35877","1","48","10000","B",678,"1"),("2021-04-11","77134","3","24","10000","P",454,"0"),("2021-04-11","47080","6","36","10000","B",737,"0"),("2021-04-11","46392","3","24","30000","P",261,"0"),("2021-04-11","21418","3","36","20000","B",390,"1"),("2021-04-12","80175","9","48","10000","B",713,"1"),("2021-04-12","91487","6","48","12000","P",778,"1");
INSERT INTO `offers_lowest` (`date`,`cap_id`,`deposit`,`term`,`mileage`,`finance_type`,`lowest_price`,`stock`) VALUES ("2021-04-09","18311","9","24","20000","B",384,"0"),("2021-04-010","41740","9","24","12000","P",674,"1"),("2021-04-010","69030","9","24","10000","P",518,"0"),("2021-04-10","53342","3","36","8000","B",746,"1"),("2021-04-11","65566","6","36","30000","P",145,"1"),("2021-04-11","63172","6","48","5000","P",698,"1"),("2021-04-11","79712","1","48","30000","B",330,"1"),("2021-04-11","90505","1","36","8000","B",458,"0"),("2021-04-12","42393","1","36","20000","B",363,"1"),("2021-04-12","70454","9","24","5000","B",673,"1");
INSERT INTO `offers_lowest` (`date`,`cap_id`,`deposit`,`term`,`mileage`,`finance_type`,`lowest_price`,`stock`) VALUES ("2021-04-09","81215","1","48","8000","B",472,"0"),("2021-04-09","76538","3","24","15000","B",226,"0"),("2021-04-010","05094","9","36","12000","P",721,"1"),("2021-04-10","57363","9","48","5000","B",777,"1"),("2021-04-11","23233","1","48","12000","B",381,"0"),("2021-04-11","40542","3","48","12000","P",610,"0"),("2021-04-11","63824","3","24","12000","B",761,"1"),("2021-04-11","17686","3","24","5000","P",893,"1"),("2021-04-12","57669","9","48","30000","P",805,"0"),("2021-04-12","21864","1","24","25000","P",530,"0");
INSERT INTO `offers_lowest` (`date`,`cap_id`,`deposit`,`term`,`mileage`,`finance_type`,`lowest_price`,`stock`) VALUES ("2021-04-09","48360","3","48","12000","P",159,"0"),("2021-04-09","88614","3","48","20000","B",730,"0"),("2021-04-010","17693","1","24","10000","B",298,"0"),("2021-04-10","34049","6","48","20000","B",728,"0"),("2021-04-11","15038","9","24","10000","P",720,"1"),("2021-04-11","31809","1","36","20000","P",237,"0"),("2021-04-11","49277","9","48","25000","P",235,"1"),("2021-04-11","54607","1","24","12000","P",661,"1"),("2021-04-12","65098","1","48","20000","P",548,"1"),("2021-04-12","76440","9","48","10000","P",495,"1");

CREATE INDEX idx_profile_grouping ON offers (cap_id, deposit_value, term, annual_mileage);

CREATE INDEX idx_specials_query ON offers_lowest (cap_id, deposit, term, mileage);

SQLFiddle

所以，我正在尝试加入两个表格 - 一个是我们平台上的有效报价列表（我们是一个汽车租赁比较网站），另一个是自定义表格，它每天记录每辆车的最佳价格（cap_id )，以及所有可能的财务状况，即 9 个月的存款价值、24 个月的合同和每年 8,000 英里等。

最终目标是确定什么是“好”价格，输出包含MIN(offers.monthly_payment) 和MIN(offers_lowest.lowest_price)，并计算百分比差异，历史价格数据取自 7 天日期范围不包括当天。

offers_lowest 每天从offers 聚合，找到最小值monthly_payment 分组方式：cap_id, deposit_value, term, annual_mileage, finance_type

这是我目前想出的：

SELECT 
    s.cap_id,
    s.deposit,
    s.term,
    s.mileage,
    s.best_price,
    f.previous_best,
    (( s.best_price - f.previous_best ) / f.previous_best ) * 100 AS difference,
    s.stock
FROM
    (SELECT
        o.cap_id,
        o.deposit_value as deposit,
        o.term,
        o.annual_mileage as mileage,
        o.finance_type,
        MIN(o.monthly_payment) AS best_price,
        o.stock,
        o.brand_id
    FROM
        offers o USE INDEX(idx_profile_grouping)
    WHERE o.finance_type = 'P'
    GROUP BY o.cap_id, o.deposit_value, o.term, o.annual_mileage
    ) s 
    
INNER JOIN

    (SELECT
        ol.cap_id,
        ol.deposit,
        ol.term,
        ol.mileage,
        MIN(lowest_price) as previous_best
    FROM
        offers_lowest ol USE INDEX(idx_specials_query)
    WHERE finance_type = 'P'
        AND ol.date > CURDATE() - INTERVAL 7 DAY
        AND ol.date <= CURDATE() - INTERVAL 1 DAY
    GROUP BY ol.cap_id, ol.deposit, ol.term, ol.mileage
    ) f ON s.cap_id = f.cap_id AND s.deposit = f.deposit AND s.term = f.term AND s.mileage = f.mileage
GROUP BY s.cap_id, s.deposit, s.term, s.mileage

问题在于速度。当我单独运行任一子查询时，运行时间 10 分钟）。

索引是我在 cap_id、deposit、term 和 mileage 字段（两个表）上创建的索引，旨在加快处理速度。

解释返回这个：

# id select_type table partitions type possible_keys key key_len ref rows filtered Extra
1 PRIMARY ALL 965741 100.00 Using where; Using temporary; Using filesort
1 PRIMARY ref <auto_key0> <auto_key0> 10 f.cap_id f.deposit f.term f.mileage
3 DERIVED ol index idx_specials_query idx_specials_query 13 17386818 5.55 Using where
2 DERIVED o index idx_profile_grouping idx_profile_grouping 10 4800964 50.00 Using where

并且 SHOW PROFILE 会返回这个：

Status Duration
starting 0.000017
checking query cache for query 0.000171
checking permissions 0.000006
checking permissions 0.000005
Opening tables 0.000020
init 0.000109
System lock 0.000009
optimizing 0.000005
optimizing 0.000011
statistics 0.000033
preparing 0.000037
Sorting result 0.000007
optimizing 0.000015
statistics 0.000017
preparing 0.000017
Sorting result 0.000012
statistics 0.000046
preparing 0.000021
Creating tmp table 0.000020
Sorting result 0.000007
executing 0.000013
Sending data 0.000012
executing 0.000004
Sending data 68.205032
converting HEAP to ondisk 0.542756
Sending data 3.132642
executing 0.000019
Sending data 7.968966
converting HEAP to ondisk 0.746200
Sending data 22.506716
converting HEAP to ondisk 0.567816
Sending data 4.886870
Creating sort index 2.151519
end 0.000019
query end 0.000010
removing tmp table 0.002051
query end 0.000019
closing tables 0.000005
removing tmp table 0.001385
closing tables 0.000009
removing tmp table 0.002780
closing tables 0.000017
freeing items 0.000069
cleaned up 0.000007
cleaning up 0.000021

由此看来，我还没有写出最有效的查询..

我们使用 AWS RDS 实例，运行 MySQL 5.7.12。

如果我可以提供任何其他内容来添加背景，请告诉我。

【问题讨论】：

  • @Strawberry 查看更新的 SQLFiddle，谢谢！

标签： mysql


【解决方案1】：
  • FROM ( SELECT ... ) JOIN ( SELECT ... ) 效率低下。尝试将其中至少一个移动到外部SELECT。 （如果您为JOIN 添加合适的索引，这可能会消除“创建排序索引”。它还可能消除大部分神秘的“发送数据”。）
  • 该日期范围仅包括 6 个 DATE。
  • 如果一行有 MIN(offers.monthly_payment) 而另一行有 MIN(offers_lowest.lowest_price) 怎么办？
  • 不要将 VARCHAR(255) 用于数值。例如，MEDIUMINT 表示里程。 （这可能会消除“将 HEAP 转换为磁盘”。）
  • 请注意 PROFILE 是多么无用；大多数时候它喜欢说“发送数据”（占总数的 95%）。我可以确定 4% 并建议修复，见上文。另外 1% 是不可避免的噪音。
  • 第一个GROUP BY 不合适，因为它没有说明要为stock 或brand_id 赋予哪个值。如有必要，请在找到其余信息后使用额外的 JOIN 获取它们。
  • offers_lowest需要INDEX(finance_type, date)
  • 摆脱“索引提示”(USE INDEX...)

【讨论】：

  • 非常感谢您的详细回复-finance_type 上的索引，日期-我应该修改原始索引以包含这两个还是可以只使用这些列创建一个单独的索引？
  • @AlexMidmore - 是的，修改现有索引。 （添加INDEX(a,b)时，删除INDEX(a)。）
  • 我在加入“brands”表时也注意到了一个问题 - b.name（品牌名称）的输出对于每一行都是相同的，这告诉我它没有正确加入，但是我不知道为什么
  • @AlexMidmore - 我认为您需要使用相关查询和SHOW CREATE TABLE brand 和EXPLAIN SELECT ... 开始一个新问题，以及一些数据来显示问题。
猜你喜欢
  • 1970-01-01
  • 2015-12-03
  • 2015-11-21
  • 2017-10-30
  • 1970-01-01
  • 2020-08-14
  • 2021-12-09
  • 1970-01-01
  • 1970-01-01
相关资源
最近更新 更多
热门标签
Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode