在 MySQL 5.7 中优化查询答案

【问题标题】：optimize a query in MySQL 5.7在 MySQL 5.7 中优化查询
【发布时间】：2020-05-16 10:44:37
【问题描述】：

这是我的系统：

Linux Ubuntu 18.04 LTS
MySQL 5.7

我有一个查询（如下）需要很长时间才能完成。

需要 9 秒才能完成。在等待网页完成加载时，这对于用户来说是难以承受的。

此外，数据集庞大且不断增长。事件表有 250,000 行。我倾向于每天添加 1200 到 1800 行。

为了帮助优化，我想添加索引等，但对于如何（以及是否）可以使用我拥有的派生/连接选择查询来做到这一点，我感到很困惑。

这是我的查询。（有谁知道如何限制输出的宽度，这样我们就不必向右滚动才能看到整行？）

mysql> explain select OUTSIDE.ownerUID as Owner, 
  OUTSIDE.propUID as Property, 
  OUTSIDE.camname as 'Camera Name', 
  OUTSIDE.direction as Direction, 
  OUTSIDE.camtimestamp as 'Event Time', 
  convert_tz(now(),'UTC','US/Central') as Now, 
  sec_to_time(convert_tz(now(),'UTC','US/Central') - OUTSIDE.CamTimeStamp) as 'Elapsed Time' 
from events OUTSIDE, 
  (select camname,max(camtimestamp) as maxtimestamp from events group by camname) as INSIDE 
where OUTSIDE.camname = INSIDE.camname 
  AND OUTSIDE.camtimestamp = INSIDE.maxtimestamp;
+----+-------------+------------+------------+-------+----------------------+--------------+---------+---------------------+--------+----------+-------------+
| id | select_type | table      | partitions | type  | possible_keys        | key          | key_len | ref                 | rows   | filtered | Extra       |
+----+-------------+------------+------------+-------+----------------------+--------------+---------+---------------------+--------+----------+-------------+
|  1 | PRIMARY     | <derived2> | NULL       | ALL   | NULL                 | NULL         | NULL    | NULL                | 263103 |   100.00 | Using where |
|  1 | PRIMARY     | OUTSIDE    | NULL       | ref   | camtimestamp,camname | camtimestamp | 6       | INSIDE.maxtimestamp |      1 |    99.73 | Using where |
|  2 | DERIVED     | events     | NULL       | index | camname              | camname      | 257     | NULL                | 263103 |   100.00 | NULL        |
+----+-------------+------------+------------+-------+----------------------+--------------+---------+---------------------+--------+----------+-------------+
3 rows in set, 1 warning (0.03 sec)

查询结果如下：

mysql> select OUTSIDE.ownerUID as Owner, 
  OUTSIDE.propUID as Property, 
  OUTSIDE.camname as 'Camera Name', 
  OUTSIDE.direction as Direction, 
  OUTSIDE.camtimestamp as 'Event Time', 
  convert_tz(now(),'UTC','US/Central') as Now, 
  sec_to_time(convert_tz(now(),'UTC','US/Central') - OUTSIDE.CamTimeStamp) as 'Elapsed Time' 
from events OUTSIDE, 
  (select camname,max(camtimestamp) as maxtimestamp from events group by camname) as INSIDE 
where OUTSIDE.camname = INSIDE.camname 
  AND OUTSIDE.camtimestamp = INSIDE.maxtimestamp;
+-------+----------+-------------+-----------+---------------------+---------------------+--------------+
| Owner | Property | Camera Name | Direction | Event Time          | Now                 | Elapsed Time |
+-------+----------+-------------+-----------+---------------------+---------------------+--------------+
|     1 |        1 | wls1        | In        | 2020-01-30 12:27:31 | 2020-01-30 12:29:53 | 00:03:42     |
|     1 |        1 | wls2        | Out       | 2020-01-30 12:25:29 | 2020-01-30 12:29:53 | 00:07:04     |
+-------+----------+-------------+-----------+---------------------+---------------------+--------------+
2 rows in set (6.49 sec)

================================================ =========== 感谢所有回答问题或发表评论的人。我采纳了 Uueerdo 的综合索引建议，得出以下结果：

mysql> create index CamNameCamTime on events (camname,camtimestamp);

mysql> select OUTSIDE.ownerUID as Owner,
           OUTSIDE.propUID as Property,
           OUTSIDE.camname as 'Camera Name',
           OUTSIDE.direction as Direction,
           OUTSIDE.camtimestamp as 'Event Time',
           convert_tz(now(),'UTC','US/Central') as Now,
           sec_to_time(convert_tz(now(),'UTC','US/Central') - OUTSIDE.CamTimeStamp) as 'Elapsed Time' 
      from events OUTSIDE,
         (select camname,max(camtimestamp) as maxtimestamp 
           from events group by camname) as INSIDE
      where OUTSIDE.camname = INSIDE.camname AND OUTSIDE.camtimestamp = INSIDE.maxtimestamp;
+-------+----------+-------------+-----------+---------------------+---------------------+--------------+
| Owner | Property | Camera Name | Direction | Event Time          | Now                 | Elapsed Time |
+-------+----------+-------------+-----------+---------------------+---------------------+--------------+
|     1 |        1 | wls1        | In        | 2020-01-30 18:43:19 | 2020-01-30 18:44:33 | 00:01:54     |
|     1 |        1 | wls2        | Out       | 2020-01-30 18:41:51 | 2020-01-30 18:44:33 | 00:04:42     |
+-------+----------+-------------+-----------+---------------------+---------------------+--------------+
2 rows in set (0.00 sec)

mysql> explain select OUTSIDE.ownerUID as Owner,
    ->            OUTSIDE.propUID as Property,
    ->            OUTSIDE.camname as 'Camera Name',
    ->            OUTSIDE.direction as Direction,
    ->            OUTSIDE.camtimestamp as 'Event Time',
    ->            convert_tz(now(),'UTC','US/Central') as Now,
    ->            sec_to_time(convert_tz(now(),'UTC','US/Central') - OUTSIDE.CamTimeStamp) as 'Elapsed Time'
    ->       from events OUTSIDE,
    ->          (select camname,max(camtimestamp) as maxtimestamp
    ->            from events group by camname) as INSIDE
    ->       where OUTSIDE.camname = INSIDE.camname AND OUTSIDE.camtimestamp = INSIDE.maxtimestamp;
+----+-------------+------------+------------+-------+-------------------------------------+----------------+---------+------------------------------------+------+----------+--------------------------+
| id | select_type | table      | partitions | type  | possible_keys                       | key            | key_len | ref                                | rows | filtered | Extra                    |
+----+-------------+------------+------------+-------+-------------------------------------+----------------+---------+------------------------------------+------+----------+--------------------------+
|  1 | PRIMARY     | <derived2> | NULL       | ALL   | NULL                                | NULL           | NULL    | NULL                               |    2 |   100.00 | Using where              |
|  1 | PRIMARY     | OUTSIDE    | NULL       | ref   | camtimestamp,camname,CamNameCamTime | CamNameCamTime | 263     | INSIDE.camname,INSIDE.maxtimestamp |    1 |   100.00 | NULL                     |
|  2 | DERIVED     | events     | NULL       | range | camname,CamNameCamTime              | CamNameCamTime | 257     | NULL                               |    2 |   100.00 | Using index for group-by |
+----+-------------+------------+------------+-------+-------------------------------------+----------------+---------+------------------------------------+------+----------+--------------------------+
3 rows in set, 1 warning (0.00 sec)

成功了！

【问题讨论】：

在(camname, camtimestamp) 上添加复合索引应该有助于提高子查询和连接的性能。此外，虽然我认为它不一定会提高性能，但我还建议切换到显式 JOIN 语法，而不是继续使用过时的隐式“逗号”连接。
Uueerdo，您的建议非常有效！请考虑回答这个问题，以便我可以授予您解决方案。请参阅上面的编辑，其中总结了我对您的解决方案的发现。
这是执行“groupwise-max”的最快方法，但它确实需要该索引。

标签： mysql select optimization groupwise-maximum

【解决方案1】：

MySQL 不能在查询中对表的每个引用使用多个索引，因此camname 和camtimestamp 上的两个简单索引在这种情况下可能仅具有有限的用处。这样的索引非常适合选择日期范围内的记录，或名称的所有记录；但是在搜索每个名称的最大值时，camname 索引的用处可能会有问题（因为无论如何它都必须检查每个名称的每个 camtimestamp），而 camstimestamp 仍然需要搜索每条记录以确保所有 camnames占（最小的 camtimestamp 可能是特定 camname 的唯一一个）。

使用(camname, camtimestamp) 上的复合索引，可以在子查询中快速识别每个camname 的最大值，然后可以再次使用该索引在表中挑选出与结果匹配的行子查询。

【讨论】：

由于这里有一个子查询，我建议将您的开头句从“查询中的表上的一个索引”更改为“每个SELECT 一个索引”。

【解决方案2】：

你可以试试下面的sql吗？

 SELECT OUTSIDE.owneruid 
       AS Owner, 
       OUTSIDE.propuid 
       AS Property, 
       OUTSIDE.camname 
       AS 'Camera Name', 
       OUTSIDE.direction 
       AS Direction, 
       OUTSIDE.camtimestamp 
       AS 'Event Time', 
       Convert_tz(Now(), 'UTC', 'US/Central') 
       AS Now, 
       Sec_to_time(Convert_tz(Now(), 'UTC', 'US/Central') - 
       OUTSIDE.camtimestamp)
       AS 'Elapsed Time' 
FROM   events OUTSIDE 

WHERE  OUTSIDE.camtimestamp = (SELECT Max(camtimestamp)
        FROM   events WHERE
        events.camname = OUTSIDE.camname);

=====================

这里是原始的 Questor。我执行了上面的 SQL，查询从未完成。我终于不得不按下 CTRL C 来结束它。我重新检查了数据库连接并尝试了其他成功完成的通用查询并重试了上面的 SQL。它做了同样的事情。

这里是查询的解释：

mysql> explain SELECT OUTSIDE.owneruid
    ->        AS Owner,
    ->        OUTSIDE.propuid
    ->        AS Property,
    ->        OUTSIDE.camname
    ->        AS 'Camera Name',
    ->        OUTSIDE.direction
    ->        AS Direction,
    ->        OUTSIDE.camtimestamp
    ->        AS 'Event Time',
    ->        Convert_tz(Now(), 'UTC', 'US/Central')
    ->        AS Now,
    ->        Sec_to_time(Convert_tz(Now(), 'UTC', 'US/Central') -
    ->        OUTSIDE.camtimestamp)
    ->        AS 'Elapsed Time'
    -> FROM   events OUTSIDE
    ->
    -> WHERE  OUTSIDE.camtimestamp = (SELECT Max(camtimestamp)
    ->         FROM   events WHERE
    ->         events.camname = OUTSIDE.camname);
+----+--------------------+---------+------------+------+------------------------+----------------+---------+---------------------+--------+----------+-------------+
| id | select_type        | table   | partitions | type | possible_keys          | key            | key_len | ref                 | rows   | filtered | Extra       |
+----+--------------------+---------+------------+------+------------------------+----------------+---------+---------------------+--------+----------+-------------+
|  1 | PRIMARY            | OUTSIDE | NULL       | ALL  | NULL                   | NULL           | NULL    | NULL                | 264783 |   100.00 | Using where |
|  2 | DEPENDENT SUBQUERY | events  | NULL       | ref  | camname,CamNameCamTime | CamNameCamTime | 257     | lpr.OUTSIDE.camname | 263842 |   100.00 | Using index |
+----+--------------------+---------+------------+------+------------------------+----------------+---------+---------------------+--------+----------+-------------+
2 rows in set, 2 warnings (0.00 sec)

【讨论】：

在这种情况下，将子查询更改为相关联极不可能有帮助。
错误 1054 (42S22)：“where 子句”中的未知列“OUTSIDE.camname”
还是不行。这就是我得到的。我试了两次。我不明白为什么错误说明了它的作用。 ======== ERROR 1064 (42000): 你的 SQL 语法有错误；检查与您的 MySQL 服务器版本相对应的手册，以在第 19 行的“WHERE OUTSIDE.camtimestamp = (SELECT Max(camtimestamp) FROM events WH”附近使用正确的语法
, 需要从 OUTSIDE 之后删除
@user3055756：只是添加了通用的想法，没有尝试查询...是的，请删除 Uueerdo 提到的逗号

【解决方案3】：

如果 (camname, camtimestamp) 上的建议评论复合索引没有帮助，这可能是向您的数据库添加触发器的场景。如果您一直在寻找给定摄像机名称的最新时间戳，我会在摄像机名称表中添加一列，例如 LastEventID。然后，在您插入事件表期间，只需发出一个简化的

update CameraNameTable set 
      LastEventID = EventIDOfRecordJustInserted 
   where 
      CameraName = CameraNameOfEventInserted

那么，您的查询可以像

一样简单

select
      E.*,
      [your additional time, convert value columns]
   from
      CameraTable CT
         JOIN Events E
            on CT.LastEventID = E.ID

要在首次添加此新列后准备好您的相机名称表，您只需

Update CameraTable 
      JOIN ( select CamName, max( ID ) maxEventID
                from Events
                group by CamName ) PQ
         on CameraTable.CamName = PQ.CamName
   set LastEventID = PQ.maxEventID

您的触发器可能会在插入中添加亚秒级计时，但即使您每天只做 2,000 次，也就是每分钟 4 次，每天 8 小时。我不认为这有什么意义。一旦插入下一个事件 ID 记录，您的表将始终更新，因此您的查询永远不需要继续应用 max() 和加入。您是根据一个确切的 ID 加入的，该 ID 将是索引的一部分，并且只有 1:1 匹配。

【讨论】：

我喜欢这种方法。我越了解 MySQL 的局限性，我就越明白在某些情况下这可能是必要的。上面的综合指数建议奏效了。我的查询时间看起来只有几分之一秒。