【发布时间】：2021-04-18 08:32:24
【问题描述】：
以下是与此问题相关的基础架构元素：
- RDS Aurora，在 MySql 5.7 上运行，并在基于行的模式下启用了 binlog
- 运行 Kafka 集群的 AWS MSK（我不知道这是否重要，但它运行在 2.7.0 版本上）
- 带有 Kafka-connect（作为 Docker 映像）的 EC2 实例运行使用 Debezium 的源连接器
我正在尝试使用 Debezium 从 Aurora 实例中获取所有更改并将它们放入 Kafka，但不幸的是，它不起作用。这是我正在使用的连接器配置：
{
"name": "aurora-connector-test",
"config": {
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"tasks.max": "1",
"database.hostname": "URL",
"database.port": "3306",
"database.user": "debezium",
"database.password": "PASSWORD",
"database.server.id": "129056",
"database.server.name": "aurora-connector-db-test",
"database.allowPublicKeyRetrieval":"true",
"database.include.list": "DATABASE",
"database.history.kafka.bootstrap.servers": "BROKER1:9092,BROKER2:9092,BROKER3:9092",
"database.history.kafka.topic": "schema-changes.DATABASE",
"transforms": "route",
"transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.route.regex": "([^.]+)\\.([^.]+)\\.([^.]+)",
"transforms.route.replacement": "$3"
}
}
使用上述配置创建连接器后，我可以看到 Debezium 能够连接到 Aurora 数据库，因为它在日志中列出了所有可用的表。可悲的是，在某些时候，我收到Communications link failure 错误。这是一大块日志：
connect_1 | 2021-01-12T08:47:39.106265621Z 2021-01-12 08:47:39,106 INFO MySQL|aurora-connector-db-test|snapshot Step 5: committing transaction [io.debezium.connector.mysql.SnapshotReader]
connect_1 | 2021-01-12T08:47:39.107645625Z 2021-01-12 08:47:39,107 ERROR MySQL|aurora-connector-db-test|snapshot Failed due to error: Aborting snapshot due to error when last running 'SHOW FULL TABLES IN `sys` where Table_Type = "BASE TABLE"': Communications link failure
connect_1 | 2021-01-12T08:47:39.113018742Z org.apache.kafka.connect.errors.ConnectException: Communications link failure
connect_1 | 2021-01-12T08:47:39.113022120Z
connect_1 | 2021-01-12T08:47:39.113025636Z The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. Error code: 0; SQLSTATE: 08S01.
connect_1 | 2021-01-12T08:47:39.113029640Z at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230)
connect_1 | 2021-01-12T08:47:39.113033236Z at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:207)
connect_1 | 2021-01-12T08:47:39.113036512Z at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:847)
connect_1 | 2021-01-12T08:47:39.113039728Z at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
connect_1 | 2021-01-12T08:47:39.113055546Z at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
connect_1 | 2021-01-12T08:47:39.113059492Z at java.base/java.lang.Thread.run(Thread.java:834)
connect_1 | 2021-01-12T08:47:39.113062958Z Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
connect_1 | 2021-01-12T08:47:39.113066762Z
connect_1 | 2021-01-12T08:47:39.113070570Z The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
connect_1 | 2021-01-12T08:47:39.113074098Z at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)
connect_1 | 2021-01-12T08:47:39.113077538Z at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
connect_1 | 2021-01-12T08:47:39.113080901Z at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:835)
connect_1 | 2021-01-12T08:47:39.113084338Z at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:455)
connect_1 | 2021-01-12T08:47:39.113087583Z at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:240)
connect_1 | 2021-01-12T08:47:39.113090489Z at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:199)
connect_1 | 2021-01-12T08:47:39.113093761Z at io.debezium.jdbc.JdbcConnection.lambda$patternBasedFactory$1(JdbcConnection.java:191)
connect_1 | 2021-01-12T08:47:39.113096988Z at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:789)
connect_1 | 2021-01-12T08:47:39.113100397Z at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:784)
connect_1 | 2021-01-12T08:47:39.113103714Z at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:748)
connect_1 | 2021-01-12T08:47:39.113107314Z ... 3 more
connect_1 | 2021-01-12T08:47:39.113110461Z Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure
connect_1 | 2021-01-12T08:47:39.113113684Z
connect_1 | 2021-01-12T08:47:39.113117610Z The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
connect_1 | 2021-01-12T08:47:39.113121639Z at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
connect_1 | 2021-01-12T08:47:39.113125289Z at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
connect_1 | 2021-01-12T08:47:39.113128514Z at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
connect_1 | 2021-01-12T08:47:39.113131552Z at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
connect_1 | 2021-01-12T08:47:39.113134638Z at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
connect_1 | 2021-01-12T08:47:39.113137876Z at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
connect_1 | 2021-01-12T08:47:39.113140913Z at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
connect_1 | 2021-01-12T08:47:39.113148186Z at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
connect_1 | 2021-01-12T08:47:39.113151641Z at com.mysql.cj.protocol.a.NativeSocketConnection.connect(NativeSocketConnection.java:91)
connect_1 | 2021-01-12T08:47:39.113155079Z at com.mysql.cj.NativeSession.connect(NativeSession.java:152)
connect_1 | 2021-01-12T08:47:39.113158553Z at com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:955)
connect_1 | 2021-01-12T08:47:39.113162110Z at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:825)
connect_1 | 2021-01-12T08:47:39.113165344Z ... 10 more
connect_1 | 2021-01-12T08:47:39.113168317Z Caused by: java.net.ConnectException: Connection refused (Connection refused)
connect_1 | 2021-01-12T08:47:39.113171590Z at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
connect_1 | 2021-01-12T08:47:39.113174792Z at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
connect_1 | 2021-01-12T08:47:39.113178326Z at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
connect_1 | 2021-01-12T08:47:39.113181678Z at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
connect_1 | 2021-01-12T08:47:39.113185328Z at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
connect_1 | 2021-01-12T08:47:39.113188827Z at java.base/java.net.Socket.connect(Socket.java:609)
connect_1 | 2021-01-12T08:47:39.113192354Z at com.mysql.cj.protocol.StandardSocketFactory.connect(StandardSocketFactory.java:155)
connect_1 | 2021-01-12T08:47:39.113195838Z at com.mysql.cj.protocol.a.NativeSocketConnection.connect(NativeSocketConnection.java:65)
connect_1 | 2021-01-12T08:47:39.113199398Z ... 13 more
connect_1 | 2021-01-12T08:47:39.113249277Z 2021-01-12 08:47:39,113 ERROR || WorkerSourceTask{id=aurora-connector-test-0} Task is being killed and will not recover until manually restarted [org.apache.kafka.connect.runtime.WorkerTask]
connect_1 | 2021-01-12T08:47:39.113256190Z 2021-01-12 08:47:39,113 INFO MySQL|aurora-connector-db-test|task Stopping MySQL connector task [io.debezium.connector.mysql.MySqlConnectorTask]
connect_1 | 2021-01-12T08:47:39.113259635Z 2021-01-12 08:47:39,113 INFO MySQL|aurora-connector-db-test|task ChainedReader: Stopping the snapshot reader [io.debezium.connector.mysql.ChainedReader]
connect_1 | 2021-01-12T08:47:39.113263070Z 2021-01-12 08:47:39,113 INFO MySQL|aurora-connector-db-test|task Discarding 0 unsent record(s) due to the connector shutting down [io.debezium.connector.mysql.SnapshotReader]
connect_1 | 2021-01-12T08:47:39.113266516Z 2021-01-12 08:47:39,113 INFO MySQL|aurora-connector-db-test|task Discarding 0 unsent record(s) due to the connector shutting down [io.debezium.connector.mysql.SnapshotReader]
connect_1 | 2021-01-12T08:47:39.113420628Z 2021-01-12 08:47:39,113 INFO MySQL|aurora-connector-db-test|task [Producer clientId=aurora-connector-test-dbhistory] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. [org.apache.kafka.clients.producer.KafkaProducer]
connect_1 | 2021-01-12T08:47:39.114679266Z 2021-01-12 08:47:39,114 INFO MySQL|aurora-connector-db-test|task Connector task finished all work and is now shutdown [io.debezium.connector.mysql.MySqlConnectorTask]
connect_1 | 2021-01-12T08:47:39.114799622Z 2021-01-12 08:47:39,114 INFO || [Producer clientId=connector-producer-aurora-connector-test-0] Closing the Kafka producer with timeoutMillis = 30000 ms. [org.apache.kafka.clients.producer.KafkaProducer]
我知道我在 AWS 上的安全组配置正确，因为我能够从 EC2 实例远程连接到 Aurora 实例。此外，我在同一个 EC2 实例上使用了非常相似的 MySQL 连接器配置而不是 Aurora，它能够在同一个 Kafka 中创建消息 - 因此我认为问题出在 Aurora 配置上（我使用的是默认配置启用基于行的二进制日志），但也许我错了。
你们对这个问题有什么想法吗？
【问题讨论】：
标签： mysql apache-kafka amazon-aurora debezium