Amazon Redshift 真空吸尘器的单独队列答案

【问题标题】：Separate queue for Amazon Redshift vacuumsAmazon Redshift 真空吸尘器的单独队列
【发布时间】：2018-04-27 22:09:13
【问题描述】：

我正在考虑在 WLM（工作负载管理）中为真空查询设置一个单独的队列，并将并发设置为 1。我选择并发 1 是因为在任何给定时刻只能运行一个真空查询。我正在尝试考虑这个实现是否会有任何问题。

当我运行以下查询来检查一个真空查询占用了多少个槽时，很明显，任何多级真空查询在任何给定时间点都只占用 1 个槽。

SELECT wlm.query, wlm.slot_count, trim(q.text), queue_start_time, queue_end_time, total_queue_time
FROM stl_wlm_query wlm, stl_querytext q
WHERE wlm.query = q.query
AND text like '%Vacuum my_awesome_table%'

ORDER BY queue_start_time DESC;

  query  | slot_count |                                     btrim                               |      queue_start_time      |       queue_end_time
---------+------------+-------------------------------------------------------------------------+----------------------------+-----------------------------
 1013242 |          1 | Vacuum my_awesome_table integrity check after vacuum execution          | 2018-04-27 16:46:47.90601  | 2018-04-27 16:46:47.90601
 1013237 |          1 | Vacuum my_awesome_table merge (increment: 2 remaining rows: 77234725)   | 2018-04-27 16:46:40.785284 | 2018-04-27 16:46:40.785284
 1013235 |          1 | Vacuum my_awesome_table merge (increment: 1)                            | 2018-04-27 16:46:29.789227 | 2018-04-27 16:46:29.789227
 1013232 |          1 | Vacuum my_awesome_table sort (partition: 35 remaining rows: 331972)     | 2018-04-27 16:46:16.978124 | 2018-04-27 16:46:16.978124
 1013231 |          1 | Vacuum my_awesome_table sort (partition: 34 remaining rows: 1458914)    | 2018-04-27 16:46:15.059653 | 2018-04-27 16:46:15.059653
 1013229 |          1 | Vacuum my_awesome_table sort (partition: 33 remaining rows: 2586321)    | 2018-04-27 16:46:13.641356 | 2018-04-27 16:46:13.641356
 1013228 |          1 | Vacuum my_awesome_table sort (partition: 32 remaining rows: 3713038)    | 2018-04-27 16:46:12.233741 | 2018-04-27 16:46:12.233741
 1013227 |          1 | Vacuum my_awesome_table sort (partition: 31 remaining rows: 4839275)    | 2018-04-27 16:46:10.838661 | 2018-04-27 16:46:10.838661
 1013226 |          1 | Vacuum my_awesome_table sort (partition: 30 remaining rows: 5965341)    | 2018-04-27 16:46:09.421457 | 2018-04-27 16:46:09.421457
 1013225 |          1 | Vacuum my_awesome_table sort (partition: 29 remaining rows: 7090572)    | 2018-04-27 16:46:07.93862  | 2018-04-27 16:46:07.93862
 1013224 |          1 | Vacuum my_awesome_table sort (partition: 28 remaining rows: 8215908)    | 2018-04-27 16:46:06.441227 | 2018-04-27 16:46:06.441227
 1013223 |          1 | Vacuum my_awesome_table sort (partition: 27 remaining rows: 9341191)    | 2018-04-27 16:46:05.009684 | 2018-04-27 16:46:05.009684
 1013222 |          1 | Vacuum my_awesome_table sort (partition: 26 remaining rows: 10467621)   | 2018-04-27 16:46:03.54458  | 2018-04-27 16:46:03.54458
 1013221 |          1 | Vacuum my_awesome_table sort (partition: 25 remaining rows: 11594203)   | 2018-04-27 16:46:01.998305 | 2018-04-27 16:46:01.998305
 1013218 |          1 | Vacuum my_awesome_table sort (partition: 24 remaining rows: 12720513)   | 2018-04-27 16:46:00.528971 | 2018-04-27 16:46:00.528971
 1013217 |          1 | Vacuum my_awesome_table sort (partition: 23 remaining rows: 13844984)   | 2018-04-27 16:45:59.071698 | 2018-04-27 16:45:59.071698
 1013216 |          1 | Vacuum my_awesome_table sort (partition: 22 remaining rows: 14970941)   | 2018-04-27 16:45:57.596597 | 2018-04-27 16:45:57.596597
 1013215 |          1 | Vacuum my_awesome_table sort (partition: 21 remaining rows: 16097323)   | 2018-04-27 16:45:56.253103 | 2018-04-27 16:45:56.253103
 1013214 |          1 | Vacuum my_awesome_table sort (partition: 20 remaining rows: 17223270)   | 2018-04-27 16:45:54.634498 | 2018-04-27 16:45:54.634498
 1013213 |          1 | Vacuum my_awesome_table sort (partition: 19 remaining rows: 18351994)   | 2018-04-27 16:45:53.265236 | 2018-04-27 16:45:53.265236
 1013212 |          1 | Vacuum my_awesome_table sort (partition: 18 remaining rows: 19477834)   | 2018-04-27 16:45:51.741294 | 2018-04-27 16:45:51.741294
 1013211 |          1 | Vacuum my_awesome_table sort (partition: 17 remaining rows: 20605101)   | 2018-04-27 16:45:50.338666 | 2018-04-27 16:45:50.338666
 1013210 |          1 | Vacuum my_awesome_table sort (partition: 16 remaining rows: 21730370)   | 2018-04-27 16:45:48.698214 | 2018-04-27 16:45:48.698214
 1013209 |          1 | Vacuum my_awesome_table sort (partition: 15 remaining rows: 22856152)   | 2018-04-27 16:45:46.27666  | 2018-04-27 16:45:46.27666
 1013208 |          1 | Vacuum my_awesome_table sort (partition: 14 remaining rows: 23981990)   | 2018-04-27 16:45:43.603719 | 2018-04-27 16:45:43.603719
 1013206 |          1 | Vacuum my_awesome_table sort (partition: 13 remaining rows: 25107349)   | 2018-04-27 16:45:40.848632 | 2018-04-27 16:45:40.848632
 1013205 |          1 | Vacuum my_awesome_table sort (partition: 12 remaining rows: 26233207)   | 2018-04-27 16:45:38.097134 | 2018-04-27 16:45:38.097134
 1013204 |          1 | Vacuum my_awesome_table sort (partition: 11 remaining rows: 27359056)   | 2018-04-27 16:45:35.316781 | 2018-04-27 16:45:35.316781
 1013203 |          1 | Vacuum my_awesome_table sort (partition: 10 remaining rows: 28486363)   | 2018-04-27 16:45:33.815825 | 2018-04-27 16:45:33.815825
 1013202 |          1 | Vacuum my_awesome_table sort (partition: 9 remaining rows: 29612051)    | 2018-04-27 16:45:32.262505 | 2018-04-27 16:45:32.262505
 1013201 |          1 | Vacuum my_awesome_table sort (partition: 8 remaining rows: 30738238)    | 2018-04-27 16:45:30.867315 | 2018-04-27 16:45:30.867315
 1013200 |          1 | Vacuum my_awesome_table sort (partition: 7 remaining rows: 31864515)    | 2018-04-27 16:45:29.476297 | 2018-04-27 16:45:29.476297
 1013197 |          1 | Vacuum my_awesome_table sort (partition: 6 remaining rows: 32989591)    | 2018-04-27 16:45:28.023365 | 2018-04-27 16:45:28.023365
 1013196 |          1 | Vacuum my_awesome_table sort (partition: 5 remaining rows: 34115286)    | 2018-04-27 16:45:26.607642 | 2018-04-27 16:45:26.607642
 1013195 |          1 | Vacuum my_awesome_table sort (partition: 4 remaining rows: 35241021)    | 2018-04-27 16:45:25.179251 | 2018-04-27 16:45:25.179251
 1013194 |          1 | Vacuum my_awesome_table sort (partition: 3 remaining rows: 36367702)    | 2018-04-27 16:45:23.167024 | 2018-04-27 16:45:23.167024
 1013193 |          1 | Vacuum my_awesome_table sort (partition: 2 remaining rows: 37493086)    | 2018-04-27 16:45:21.550219 | 2018-04-27 16:45:21.550219
 1013192 |          1 | Vacuum my_awesome_table sort (partition: 1)                             | 2018-04-27 16:45:13.472849 | 2018-04-27 16:45:13.472849
 1013188 |          1 | Vacuum my_awesome_table integrity check before vacuum execution         | 2018-04-27 16:45:13.083657 | 2018-04-27 16:45:13.083657
 1008690 |          1 | Vacuum my_awesome_table integrity check after vacuum execution          | 2018-04-27 09:00:23.795686 | 2018-04-27 09:00:23.795686
 1008686 |          1 | Vacuum my_awesome_table merge (increment: 2 remaining rows: 115824559)  | 2018-04-27 09:00:15.527474 | 2018-04-27 09:00:15.527474
 1008680 |          1 | Vacuum my_awesome_table merge (increment: 1)                            | 2018-04-27 09:00:00.946362 | 2018-04-27 09:00:00.946362

【问题讨论】：

标签： amazon-redshift vacuum

【解决方案1】：

使用多个队列的想法是阻止一组查询干扰另一组查询。将您的 VACUUM 放在单独的队列中意味着您的其他查询不会受到长期运行的 VACUUM 的影响。

对于 VACUUM，并发 1 是可以的。

更重要的是您为其分配的内存量 - 越多越好，但您也不想在 VACUUM不运行时浪费内存。

因此，如果您只在一天的部分中运行 VACUUM，则剩余时间的队列将被浪费。这可能不适合您的情况。

【讨论】：

感谢约翰的见解。我的想法是一样的，你的回复证实了我的想法。关于如何从其他较短查询的路径中清除长期运行的真空作业有什么想法吗？
是否可以安排 WLM 设置脚本？比如设置真空队列的最高优先级，运行真空，然后设置回正常模式
@PratikKhadloya VACUUM 完全由您自己控制——您需要自己发出命令。所以，使用SET query_group TO <vacuum_queue>，然后发出 VACUUM 命令。更可取的选择是使用普通队列，但在晚上不使用集群时运行 VACUUM。您甚至可以使用set wlm_query_slot_count to <num> 运行 VACUUM，以便在其运行时分配多个插槽给 VACUUM。这为其提供了更多内存，因此它应该运行得更快。
感谢您的信息。我们每天运行 3 次vacuum，因为以前每天运行一次是不够的，因为我们每小时更新很多表。
您的表格是否按时间排序？如果是这样，按时间顺序加载它们意味着您不需要 VACUUM。类似地，以 SORT KEY 顺序加载可避免抽真空。详情：Managing Vacuum Times