MySQL 调优 - prestashop

Question

我使用 Prestashop+ NGINX + PHP-FPM，有超过 30.000 个列表 + 变体，由外部 API 服务器同步。

我的 prestashop 正在运行：

AMD Ryzen™ 5 3600（6 核）+ 64GB RAM + 2xNVME 500GB (RAID 1) + 千兆以太网/网络

我不是专家，我很想得到一些建议

/etc/my.cnf

[mysql]
port                            = 3306

[mysqld]
# === Required Settings ===
basedir                         = /usr
bind_address                    = 127.0.0.1 # Change to 0.0.0.0 to allow remote connections
datadir                         = /var/lib/mysql
max_allowed_packet              = 256M
max_connect_errors              = 1000000
port                            = 3306
skip_external_locking
skip_name_resolve
tmpdir                          = /tmp
user                            = mysql

# === SQL Compatibility Mode ===
# Enable for b/c with databases created in older MySQL/MariaDB versions
# (e.g. when using null dates)
#sql_mode                       = ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION,ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES

# === InnoDB Settings ===
default_storage_engine          = InnoDB
innodb_buffer_pool_instances    = 42     # Use 1 instance per 1GB of InnoDB pool size
innodb_buffer_pool_size         = 42G    # Use up to 70-80% of RAM
innodb_file_per_table           = 1
innodb_flush_log_at_trx_commit  = 0
innodb_flush_method             = O_DIRECT
innodb_log_buffer_size          = 16M
innodb_log_file_size            = 2G
innodb_stats_on_metadata        = 0
performance_schema = ON

#innodb_temp_data_file_path     = ibtmp1:64M:autoextend:max:20G # Control the maximum size for the ibtmp1 file
#innodb_thread_concurrency      = 4     # Optional: Set to the number of CPUs on your system (minus 1 or 2) to better
                                        # contain CPU usage. E.g. if your system has 8 CPUs, try 6 or 7 and check
                                        # the overall load produced by MySQL/MariaDB.
innodb_read_io_threads          = 64
innodb_write_io_threads         = 64
#innodb_io_capacity             = 1000  # Max is 2000

# === MyISAM Settings ===
# The following 3 options are ONLY supported by MariaDB & up to MySQL 5.7
# Do NOT un-comment on MySQL 8.x+
query_cache_limit              = 10M    # UPD
query_cache_size               = 50000M   # UPD
query_cache_type               = 1     # Enabled by default

key_buffer_size                 = 32M   # UPD

low_priority_updates            = 1
concurrent_insert               = 2

# === Connection Settings ===
max_connections                 = 750   # UPD - Important: high no. of connections = high RAM consumption

back_log                        = 512
thread_cache_size               = 100
thread_stack                    = 192K

interactive_timeout             = 180
wait_timeout                    = 180

# For MySQL 5.7+ only (disabled by default)
#max_execution_time             = 30000 # Set a timeout limit for SELECT statements (value in milliseconds).
                                        # This option may be useful to address aggressive crawling on large sites,
                                        # but it can also cause issues (e.g. with backups). So use with extreme caution and test!
                                        # More info at: https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_max_execution_time

# For MariaDB 10.1.1+ only (disabled by default)
#max_statement_time             = 30    # The equivalent of "max_execution_time" in MySQL 5.7+ (set above)
                                        # The variable is of type double, thus you can use subsecond timeout.
                                        # For example you can use value 0.01 for 10 milliseconds timeout.
                                        # More info at: https://mariadb.com/kb/en/aborting-statements/

# === Buffer Settings ===
innodb_sort_buffer_size         = 2M    # UPD
join_buffer_size                = 8M    # UPD
read_buffer_size                = 3M    # UPD
read_rnd_buffer_size            = 4M    # UPD
sort_buffer_size                = 4M    # UPD

# === Table Settings ===
# In systemd managed systems like Ubuntu 16.04+ or CentOS 7+, you need to perform an extra action for table_open_cache & open_files_limit
# to be overriden (also see comment next to open_files_limit).
# E.g. for MySQL 5.7, please check: https://dev.mysql.com/doc/refman/5.7/en/using-systemd.html
# and for MariaDB check: https://mariadb.com/kb/en/library/systemd/
table_definition_cache          = 40000 # UPD
table_open_cache                = 40000 # UPD
open_files_limit                = 80000 # UPD - This can be 2x to 3x the table_open_cache value or match the system's
                                        # open files limit usually set in /etc/sysctl.conf or /etc/security/limits.conf
                                        # In systemd managed systems this limit must also be set in:
                                        # /etc/systemd/system/mysqld.service.d/override.conf (for MySQL 5.7+) and
                                        # /etc/systemd/system/mariadb.service.d/override.conf (for MariaDB)

max_heap_table_size             = 128M
tmp_table_size                  = 128M

# === Search Settings ===
ft_min_word_len                 = 3     # Minimum length of words to be indexed for search results

# === Logging ===
log_error                       = /var/lib/mysql/mysql_error.log
log_queries_not_using_indexes   = 1
long_query_time                 = 5
slow_query_log                  = 1     # Disabled for production
slow_query_log_file             = /var/lib/mysql/mysql_slow.log

[mysqldump]
# Variable reference
# For MySQL 5.7+:  https://dev.mysql.com/doc/refman/5.7/en/mysqldump.html
# For MariaDB:     https://mariadb.com/kb/en/library/mysqldump/
quick
quote_names
max_allowed_packet              = 512M

tuning-primer.sh

[root@server2 /]# curl -L https://raw.githubusercontent.com/BMDan/tuning-primer.sh/master/tuning-primer.sh | bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 63986  100 63986    0     0   201k      0 --:--:-- --:--:-- --:--:--  202k

 -- MYSQL PERFORMANCE TUNING PRIMER --
      - By: Matthew Montgomery -

MySQL Version 10.2.38-MariaDB-log x86_64

Uptime = 3 days 21 hrs 47 min 51 sec
Avg. qps = 18
Total Questions = 6119342
Threads Connected = 11

Server has been running for over 48hrs.
It should be safe to follow these recommendations

To find out more information on how each of these
runtime variables effects performance visit:
http://dev.mysql.com/doc/refman/10.2/en/server-system-variables.html
Visit http://www.mysql.com/products/enterprise/advisors.html
for info about MySQL's Enterprise Monitoring and Advisory Service

SLOW QUERIES
The slow query log is enabled.
Current long_query_time = 5.000000 sec.
Since startup, 102804 out of 6119356 queries have taken longer than <long_query_time-when-they-were-executed> to complete.
Your long_query_time may be too high, I typically set this under 5 sec.

BINARY UPDATE LOG
The binary update log is NOT enabled.
You will not be able to do point in time recovery
See http://dev.mysql.com/doc/refman/10.2/en/point-in-time-recovery.html

WORKER THREADS
Current thread_cache_size = 100
Current threads_cached = 42
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine

MAX CONNECTIONS
Current max_connections = 750
Current threads_connected = 11
Historic max_used_connections = 54
The number of used connections is 7% of the configured maximum.
You are using less than 10% of your configured max_connections.
Lowering max_connections could help to avoid an over-allocation of memory
See "MEMORY USAGE" section to make sure you are not over-allocating

INNODB STATUS
Current InnoDB index space = 137 M
Current InnoDB data space = 255 M
Current InnoDB buffer pool free = 99 %
Current innodb_buffer_pool_size = 42.00 G
Depending on how much space your innodb indexes take up it may be safe
to increase this value to up to 2 / 3 of total system memory

MEMORY USAGE
Max Memory Ever Allocated : 91.88 G
Configured Max Per-thread Buffers : 14.05 G
Configured Max Global Buffers : 90.87 G
Configured Max Memory Limit : 104.92 G
Physical Memory : 62.75 G

Max memory limit exceeds 90% of physical memory

KEY BUFFER
Current MyISAM index space = 185 K
Current key_buffer_size = 32 M
Key cache miss rate is 1 : 697
Key buffer free ratio = 81 %
Your key_buffer_size seems to be fine

QUERY CACHE
Query cache is enabled
Current query_cache_size = 48.82 G
Current query_cache_used = 1.17 G
Current query_cache_limit = 10 M
Current Query cache Memory fill ratio = 2.39 %
Current query_cache_min_res_unit = 4 K
Your query_cache_size seems to be too high.
Perhaps you can use these resources elsewhere
MySQL won't cache query results that are larger than query_cache_limit in size

SORT OPERATIONS
Current sort_buffer_size = 4 M
Current read_rnd_buffer_size = 4 M
Sort buffer seems to be fine

JOINS
Current join_buffer_size = 8.00 M
You have had 15352 queries where a join could not use an index properly
join_buffer_size >= 4 M
This is not advised
You should enable "log-queries-not-using-indexes"
Then look for non indexed joins in the slow query log.

OPEN FILES LIMIT
Current open_files_limit = 32768 files
The open_files_limit should typically be set to at least 2x-3x
that of table_cache if you have heavy MyISAM usage.
Your open_files_limit value seems to be fine

TABLE CACHE
Current table_open_cache = 15994 tables
Current table_definition_cache = 40000 tables
You have a total of 483 tables
You have 557 open tables.
The table_cache value seems to be fine

TEMP TABLES
Current max_heap_table_size = 128 M
Current tmp_table_size = 128 M
Of 146965 temp tables, 6% were created on disk
Created disk tmp tables ratio seems fine

TABLE SCANS
Current read_buffer_size = 3 M
Current table scan ratio = 88 : 1
read_buffer_size seems to be fine

TABLE LOCKING
Current Lock Wait ratio = 1 : 37
You may benefit from selective use of InnoDB.

Answer 1

当前query_cache_size = 48.82 G -- 不！停在50M左右；当它那么大时，它会变得非常低效。

在慢日志上使用 pt-query-digest；然后让我们讨论前几个查询。这可能对性能有很大帮助。

log_queries_not_using_indexes = 1 -- 关闭；否则会使慢日志变得混乱。

我们将在第二遍中解决这个问题：

max_connections                 = 750   # UPD
table_definition_cache          = 40000 # UPD
table_open_cache                = 40000 # UPD

如果您想了解更多分析，请参阅http://mysql.rjweb.org/doc.php/mysql_analysis#tuning

摘要

查询 1

    UPDATE        ps_stock_available sa
       INNER JOIN ps_mmo_connector_product_map pm
                ON sa.id_product = pm.id_product_shop 
            SET sa.quantity = '10'
            WHERE sa.quantity > '10'\G

摘要中的第一个查询是一个似乎以quantity 为上限的更新。据推测，您可以通过永远将值更改为超过 10 来消除它？这将减轻服务器的一半负担。如果您不想这样做，请提供这些以便我检查索引：

SHOW CREATE TABLE `presta`.`ps_stock_available`\G
SHOW CREATE TABLE `presta`.`ps_mmo_connector_product_map`\G

但是……有什么问题？这意味着没有太多事情发生：

# Overall: 281.15k total, 799 unique, 0.54 QPS, 0.01x concurrency 

这表示“最差”查询（即更新）不是很慢或很频繁：

# Exec time     48   1313s    18ms      1s    85ms   253ms    88ms    53ms

如果您期望服务器变得越来越繁忙，那么我将深入挖掘。但目前，服务器唯一会做的就是打鼾。

当你得到这些创建时，也抓住这些，这样我就可以检查接下来的几个查询：

SHOW CREATE TABLE `presta`.`ps_category_product`\G
SHOW CREATE TABLE `presta`.`ps_category`\G
SHOW CREATE TABLE `presta`.`ps_category_shop`\G

SHOW CREATE TABLE `presta`.`ps_search_word`\G
SHOW CREATE TABLE `presta`.`ps_search_index`\G

SHOW TABLE STATUS FROM `presta` LIKE 'ps_tag_count'\G

我看到REPLACE 似乎正在重新计算标签计数。乍一看，这似乎是一种非常低效的方法。但是看看查询有多复杂，改进它可能并不实际。你是 Query 4 的意图吗？