一、pt-osc的工作原理
（1.1）基本原理步骤
1、创建一个和源表一样表结构的新表
2、在新表执行DDL语句
3、在源表创建三个触发器分别对应insert、update、delete操作
4、从源表拷贝数据到新表，拷贝过程中源表通过触发器把新的DML操作更新到新表中
5、rename源表到old表中，把新表rename为源表，默认最后删除源表
general log种已经算比较详细了,但不同参数可能结果还是会有不少区别,此处不多分析,精简一下osc的几个重要的步骤如下：
| 步骤 | 操作 |
|---|---|
| step1 | 检查原表是否由主键和触发器 |
| step2 | 被操作表是S表，创建tmp-S数据表 |
| step3 | 在S表上创建insert update delete触发器 |
| step4 | 全量数据同步过去 |
| step5 | 全量同步的过程中,新产生的增量（变化）数据就触发到tmp-S表中 |
| step6 | 新旧表重命名(元数据锁,短暂锁表) |
| step7 | 删除旧表,删除触发器 |
general log 提炼全过程
step1：略 step2： (root@localhost) [(none)]> truncate mysql.general_log; Query OK, 0 rows affected (1.65 sec) (root@localhost) [(none)]> set global general_log = 1; Query OK, 0 rows affected (0.00 sec) (root@localhost) [(none)]> set global log_output = 'table'; Query OK, 0 rows affected (0.01 sec) step 3： pt-online-schema-change --alter "add index index_c (c)" --socket=/tmp/mysql.sock --user=root --password=123 D=test,t=sbtest1 --execute No slaves found. See --recursion-method if host VM_221_162_centos has slaves. Not checking slave lag because no slaves were found and --check-slave-lag was not specified. Operation, tries, wait: analyze_table, 10, 1 copy_rows, 10, 0.25 create_triggers, 10, 1 drop_triggers, 10, 1 swap_tables, 10, 1 update_foreign_keys, 10, 1 Altering `test`.`sbtest1`... Creating new table... Created new table test._sbtest1_new OK. Altering new table... Altered `test`.`_sbtest1_new` OK. 2017-11-30T18:28:19 Creating triggers... 2017-11-30T18:28:19 Created triggers OK. 2017-11-30T18:28:19 Copying approximately 493200 rows... 2017-11-30T18:28:41 Copied rows OK. 2017-11-30T18:28:41 Analyzing new table... 2017-11-30T18:28:41 Swapping tables... 2017-11-30T18:28:41 Swapped original and new tables OK. 2017-11-30T18:28:41 Dropping old table... 2017-11-30T18:28:41 Dropped old table `test`.`_sbtest1_old` OK. 2017-11-30T18:28:41 Dropping triggers... 2017-11-30T18:28:41 Dropped triggers OK. Successfully altered `test`.`sbtest1`. 上面已经可以看出个大概过程了 step 4： 这一步详细分5块分析如下： (root@localhost) [(none)]> set global log_output = 'file'; Query OK, 0 rows affected (0.00 sec) (root@localhost) [(none)]> set global general_log = 0; Query OK, 0 rows affected (0.01 sec) (root@localhost) [mysql]> select argument from mysql.general_log; root@localhost on test using Socket set autocommit=1 SHOW VARIABLES LIKE 'innodb\_lock_wait_timeout' SET SESSION innodb_lock_wait_timeout=1 SHOW VARIABLES LIKE 'lock\_wait_timeout' SET SESSION lock_wait_timeout=60 SHOW VARIABLES LIKE 'wait\_timeout' SET SESSION wait_timeout=10000 SELECT @@SQL_MODE SET @@SQL_QUOTE_SHOW_CREATE = 1/*!40101, @@SQL_MODE='NO_AUTO_VALUE_ON_ZERO,ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION'*/ SELECT @@server_id /*!50038 , @@hostname*/ 说明： 1、设置session级的变量 SET SESSION innodb_lock_wait_timeout=1 SET SESSION lock_wait_timeout=60 SET SESSION wait_timeout=10000 －－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－ SHOW VARIABLES LIKE 'version%' SHOW ENGINES SHOW VARIABLES LIKE 'innodb_version' SHOW VARIABLES LIKE 'innodb_stats_persistent' SELECT @@SERVER_ID SHOW GRANTS FOR CURRENT_USER() SHOW FULL PROCESSLIST SHOW SLAVE HOSTS SHOW GLOBAL STATUS LIKE 'Threads_running' SHOW GLOBAL STATUS LIKE 'Threads_running' SELECT CONCAT(@@hostname, @@port) SHOW TABLES FROM `test` LIKE 'sbtest1' SELECT VERSION() SHOW TRIGGERS FROM `test` LIKE 'sbtest1' /*!40101 SET @OLD_SQL_MODE := @@SQL_MODE, @@SQL_MODE := '', @OLD_QUOTE := @@SQL_QUOTE_SHOW_CREATE, @@SQL_QUOTE_SHOW_CREATE := 1 */ USE `test` SHOW CREATE TABLE `test`.`sbtest1` /*!40101 SET @@SQL_MODE := @OLD_SQL_MODE, @@SQL_QUOTE_SHOW_CREATE := @OLD_QUOTE */ EXPLAIN SELECT * FROM `test`.`sbtest1` WHERE 1=1 SELECT table_schema, table_name FROM information_schema.key_column_usage WHERE referenced_table_schema='test' AND referenced_table_name='sbtest1' SHOW VARIABLES LIKE 'wsrep_on' /*!40101 SET @OLD_SQL_MODE := @@SQL_MODE, @@SQL_MODE := '', @OLD_QUOTE := @@SQL_QUOTE_SHOW_CREATE, @@SQL_QUOTE_SHOW_CREATE := 1 */ 说明： 1、查看变量,当前用户的权限,slave信息,版本信息等 2、检查sbtest1是否存在触发器 3、执行计划 4、检查sbtest1是否存在外键关联 －－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－ USE `test` SHOW CREATE TABLE `test`.`sbtest1` /*!40101 SET @@SQL_MODE := @OLD_SQL_MODE, @@SQL_QUOTE_SHOW_CREATE := @OLD_QUOTE */ CREATE TABLE `test`.`_sbtest1_new` ( `id` int(11) NOT NULL AUTO_INCREMENT, `k` int(11) NOT NULL DEFAULT '0', `c` char(120) NOT NULL DEFAULT '', `pad` char(60) NOT NULL DEFAULT '', PRIMARY KEY (`id`), KEY `k_1` (`k`) ) ENGINE=InnoDB AUTO_INCREMENT=500001 DEFAULT CHARSET=latin1 ALTER TABLE `test`.`_sbtest1_new` add index index_c (c) /*!40101 SET @OLD_SQL_MODE := @@SQL_MODE, @@SQL_MODE := '', @OLD_QUOTE := @@SQL_QUOTE_SHOW_CREATE, @@SQL_QUOTE_SHOW_CREATE := 1 */ USE `test` SHOW CREATE TABLE `test`.`_sbtest1_new` /*!40101 SET @@SQL_MODE := @OLD_SQL_MODE, @@SQL_QUOTE_SHOW_CREATE := @OLD_QUOTE */ SELECT TRIGGER_SCHEMA, TRIGGER_NAME, DEFINER, ACTION_STATEMENT, SQL_MODE, CHARACTER_SET_CLIENT, COLLATION_CONNECTION, EVENT_MANIPULATION, ACTION_TIMING FROM INFORMATION_SCHEMA.TRIGGERS WHERE EVENT_MANIPULATION = 'DELETE' AND ACTION_TIMING = 'AFTER' AND TRIGGER_SCHEMA = 'test' AND EVENT_OBJECT_TABLE = 'sbtest1' SELECT TRIGGER_SCHEMA, TRIGGER_NAME, DEFINER, ACTION_STATEMENT, SQL_MODE, CHARACTER_SET_CLIENT, COLLATION_CONNECTION, EVENT_MANIPULATION, ACTION_TIMING FROM INFORMATION_SCHEMA.TRIGGERS WHERE EVENT_MANIPULATION = 'UPDATE' AND ACTION_TIMING = 'AFTER' AND TRIGGER_SCHEMA = 'test' AND EVENT_OBJECT_TABLE = 'sbtest1' SELECT TRIGGER_SCHEMA, TRIGGER_NAME, DEFINER, ACTION_STATEMENT, SQL_MODE, CHARACTER_SET_CLIENT, COLLATION_CONNECTION, EVENT_MANIPULATION, ACTION_TIMING FROM INFORMATION_SCHEMA.TRIGGERS WHERE EVENT_MANIPULATION = 'INSERT' AND ACTION_TIMING = 'AFTER' AND TRIGGER_SCHEMA = 'test' AND EVENT_OBJECT_TABLE = 'sbtest1' SELECT TRIGGER_SCHEMA, TRIGGER_NAME, DEFINER, ACTION_STATEMENT, SQL_MODE, CHARACTER_SET_CLIENT, COLLATION_CONNECTION, EVENT_MANIPULATION, ACTION_TIMING FROM INFORMATION_SCHEMA.TRIGGERS WHERE EVENT_MANIPULATION = 'DELETE' AND ACTION_TIMING = 'BEFORE' AND TRIGGER_SCHEMA = 'test' AND EVENT_OBJECT_TABLE = 'sbtest1' SELECT TRIGGER_SCHEMA, TRIGGER_NAME, DEFINER, ACTION_STATEMENT, SQL_MODE, CHARACTER_SET_CLIENT, COLLATION_CONNECTION, EVENT_MANIPULATION, ACTION_TIMING FROM INFORMATION_SCHEMA.TRIGGERS WHERE EVENT_MANIPULATION = 'UPDATE' AND ACTION_TIMING = 'BEFORE' AND TRIGGER_SCHEMA = 'test' AND EVENT_OBJECT_TABLE = 'sbtest1' SELECT TRIGGER_SCHEMA, TRIGGER_NAME, DEFINER, ACTION_STATEMENT, SQL_MODE, CHARACTER_SET_CLIENT, COLLATION_CONNECTION, EVENT_MANIPULATION, ACTION_TIMING FROM INFORMATION_SCHEMA.TRIGGERS WHERE EVENT_MANIPULATION = 'INSERT' AND ACTION_TIMING = 'BEFORE' AND TRIGGER_SCHEMA = 'test' AND EVENT_OBJECT_TABLE = 'sbtest1' CREATE TRIGGER `pt_osc_test_sbtest1_del` AFTER DELETE ON `test`.`sbtest1` FOR EACH ROW DELETE IGNORE FROM `test`.`_sbtest1_new` WHERE `test`.`_sbtest1_new`.`id` <=> OLD.`id` CREATE TRIGGER `pt_osc_test_sbtest1_upd` AFTER UPDATE ON `test`.`sbtest1` FOR EACH ROW BEGIN DELETE IGNORE FROM `test`.`_sbtest1_new` WHERE !(OLD.`id` <=> NEW.`id`) AND `test`.`_sbtest1_new`.`id` <=> OLD.`id`;REPLACE INTO `test`.`_sbtest1_new` (`id`, `k`, `c`, `pad`) VALUES (NEW.`id`, NEW.`k`, NEW.`c`, NEW.`pad`);END CREATE TRIGGER `pt_osc_test_sbtest1_ins` AFTER INSERT ON `test`.`sbtest1` FOR EACH ROW REPLACE INTO `test`.`_sbtest1_new` (`id`, `k`, `c`, `pad`) VALUES (NEW.`id`, NEW.`k`, NEW.`c`, NEW.`pad`) 说明： 1、根据原表的表结构结创建一张新表 2、对新表上的c字段加索引,这里依然用的是alter 3、检查原表上触发器情况,5.6开始同一张表上不能存在同一个动作的触发器 4、针对新表创建三个触发器,DELETE,UPDATE和INSERT(重点看下三个触发器内容) －－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－ EXPLAIN SELECT * FROM `test`.`sbtest1` WHERE 1=1 SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `test`.`sbtest1` FORCE INDEX(`PRIMARY`) ORDER BY `id` LIMIT 1 /*first lower boundary*/ SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `test`.`sbtest1` FORCE INDEX (`PRIMARY`) WHERE `id` IS NOT NULL ORDER BY `id` LIMIT 1 /*key_len*/ EXPLAIN SELECT /*!40001 SQL_NO_CACHE */ * FROM `test`.`sbtest1` FORCE INDEX (`PRIMARY`) WHERE `id` >= '1' /*key_len*/ EXPLAIN SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `test`.`sbtest1` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1')) ORDER BY `id` LIMIT 999, 2 /*next chunk boundary*/ SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `test`.`sbtest1` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1')) ORDER BY `id` LIMIT 999, 2 /*next chunk boundary*/ EXPLAIN SELECT `id`, `k`, `c`, `pad` FROM `test`.`sbtest1` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1')) AND ((`id` <= '1000')) LOCK IN SHARE MODE /*explain pt-online-schema-change 16157 copy nibble*/ INSERT LOW_PRIORITY IGNORE INTO `test`.`_sbtest1_new` (`id`, `k`, `c`, `pad`) SELECT `id`, `k`, `c`, `pad` FROM `test`.`sbtest1` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1')) AND ((`id` <= '1000')) LOCK IN SHARE MODE /*pt-online-schema-change 16157 copy nibble*/ SHOW WARNINGS SHOW GLOBAL STATUS LIKE 'Threads_running' EXPLAIN SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `test`.`sbtest1` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1001')) ORDER BY `id` LIMIT 3787, 2 /*next chunk boundary*/ SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `test`.`sbtest1` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1001')) ORDER BY `id` LIMIT 3787, 2 /*next chunk boundary*/ EXPLAIN SELECT `id`, `k`, `c`, `pad` FROM `test`.`sbtest1` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1001')) AND ((`id` <= '4788')) LOCK IN SHARE MODE /*explain pt-online-schema-change 16157 copy nibble*/ INSERT LOW_PRIORITY IGNORE INTO `test`.`_sbtest1_new` (`id`, `k`, `c`, `pad`) SELECT `id`, `k`, `c`, `pad` FROM `test`.`sbtest1` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1001')) AND ((`id` <= '4788')) LOCK IN SHARE MODE /*pt-online-schema-change 16157 copy nibble*/ SHOW WARNINGS SHOW GLOBAL STATUS LIKE 'Threads_running' 说明： 1、chunk太多,此处只贴两组 2、以chunk为单位进行目标表数据的拷贝（根据pk或uk分片）,有专门的参数指定怎么分片 3、在拷贝的过程中,对目标表的相关记录加了lock in share mode mode保证数据一致性,此时,会堵塞客户端对这些记录的DML操作 4、LOW_PRIORITY插入,降低优先级插入,等表上无其他操作时才插 5、SHOW GLOBAL STATUS LIKE 'Threads_running'检查当前正在运行的Threads数量,默认Threads_running=25,如果未指定最大值,则会取当前值的120%作为最大值,如果超过阀值则会暂停数据拷贝 6、很多explain语句？原因是没指定chunk大小,第一次默认分1000条记录,后面chunk具体多少根据执行计划判断成本,不影响系统正常运行则执行insert －－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－ RENAME TABLE `test`.`sbtest1` TO `test`.`_sbtest1_old`, `test`.`_sbtest1_new` TO `test`.`sbtest1` DROP TABLE IF EXISTS `test`.`_sbtest1_old` DROP TRIGGER IF EXISTS `test`.`pt_osc_test_sbtest1_del` DROP TRIGGER IF EXISTS `test`.`pt_osc_test_sbtest1_upd` DROP TRIGGER IF EXISTS `test`.`pt_osc_test_sbtest1_ins` SHOW TABLES FROM `test` LIKE '\_sbtest1\_new' 说明： 1、ANALYZE更新新表的统计信息 2、新老两张表重命名 3、删除原表 4、删除触发器