即使没有流量，MariaDb-failover 也会在 CPU 上运行且很健谈

【问题标题】：MariaDb-failover is running high on CPU and chatty even when there is no traffic即使没有流量，MariaDb-failover 也会在 CPU 上运行且很健谈
【发布时间】：2020-10-05 12:54:40
【问题描述】：

我们有一个用 scala 2.12.x 和 Play framework 2.5.x 编写的 API。 API 使用 MariaDb 连接器/J 2.5.4 连接到 AWS aurora 集群，例如 jdbc:mysql:aurora://some-aurora-cluster

功能上一切正常，除了我们注意到 CPU 使用率很高，即使没有流量也是如此。一些研究表明：

[ec2-user@ip-xxx-xxx-xxx-xxx ~]$ top -H

    …
 6373 root      20   0 4644452 990888  21540 S 14.6 12.6   1:15.68 MariaDb-failove
 6374 root      20   0 4644452 990888  21540 S 13.6 12.6   1:16.11 MariaDb-failove
 6305 root      20   0 4644452 990888  21540 S 13.3 12.6   1:14.31 MariaDb-failove
 6375 root      20   0 4644452 990888  21540 S 12.3 12.6   1:14.59 MariaDb-failove
 6372 root      20   0 4644452 990888  21540 S 11.3 12.6   1:15.78 MariaDb-failove
    …

上面的cmd显示了很多MariaDb-failover。我不确定它的作用以及为什么有多个它忙于高 CPU 使用率。

[ec2-user@ip-xxx-xxx-xxx-xxx ~]$ netstat -a | less

…
tcp6       0      0 ip-xxx-xxx-xxx-31.:37446 ip-xxx-xxx-xxx-129:mysql TIME_WAIT
tcp6       0      0 ip-xxx-xxx-xxx-31.:37108 ip-xxx-xxx-xxx-129:mysql TIME_WAIT
tcp6       0      0 ip-xxx-xxx-xxx-31.:37648 ip-xxx-xxx-xxx-129:mysql TIME_WAIT
tcp6       0      0 ip-xxx-xxx-xxx-31.:36934 ip-xxx-xxx-xxx-129:mysql TIME_WAIT
tcp6       0      0 ip-xxx-xxx-xxx-31.:36870 ip-xxx-xxx-xxx-129:mysql TIME_WAIT
tcp6       0      0 ip-xxx-xxx-xxx-31.:37254 ip-xxx-xxx-xxx-129:mysql TIME_WAIT
tcp6       0      0 ip-xxx-xxx-xxx-31.:37902 ip-xxx-xxx-xxx-129:mysql TIME_WAIT
…

有很多 TIME_WAIT。这也很奇怪，因为在我执行这个 cmd 时没有流量。

[ec2-user@ip-xxx-xxx-xxx-xxx ~]$ netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n

      1 established)
      1 Foreign
      1 SYN_SENT
      6 LISTEN
     37 ESTABLISHED
    851 TIME_WAIT

有数百个TIME_WAIT；每次执行 cmd 时，数字都会发生变化。

有没有人知道这是正常现象还是我需要担心的事情？

如果您还有其他问题，请告诉我。

=========== 更多信息

ps -aux | grep java

得到 PID：2655

jstack 2655 > threaddump.log

这是内容（已删减）：

2020-06-16 16:44:39
Full thread dump OpenJDK 64-Bit Server VM (25.252-b09 mixed mode):

"MariaDb-failover-5" #276 daemon prio=5 os_prio=0 tid=0x00007f4b0400b000 nid=0x18e7 waiting on condition [0x00007f4b0c6d6000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.mariadb.jdbc.internal.protocol.AuroraProtocol.loop(AuroraProtocol.java:269)
    at org.mariadb.jdbc.internal.failover.impl.AuroraListener.reconnectFailedConnection(AuroraListener.java:203)
    at org.mariadb.jdbc.internal.failover.thread.FailoverLoop.doRun(FailoverLoop.java:84)
    at org.mariadb.jdbc.internal.failover.thread.TerminableRunnable.run(TerminableRunnable.java:80)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

"MariaDb-failover-4" #275 daemon prio=5 os_prio=0 tid=0x00007f4b0400a000 nid=0x18e6 waiting on condition [0x00007f4afbefd000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.mariadb.jdbc.internal.protocol.AuroraProtocol.loop(AuroraProtocol.java:269)
    at org.mariadb.jdbc.internal.failover.impl.AuroraListener.reconnectFailedConnection(AuroraListener.java:203)
    at org.mariadb.jdbc.internal.failover.thread.FailoverLoop.doRun(FailoverLoop.java:84)
    at org.mariadb.jdbc.internal.failover.thread.TerminableRunnable.run(TerminableRunnable.java:80)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

"MariaDb-failover-3" #274 daemon prio=5 os_prio=0 tid=0x00007f4b04009000 nid=0x18e5 waiting on condition [0x00007f4afbbfa000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.mariadb.jdbc.internal.protocol.AuroraProtocol.loop(AuroraProtocol.java:269)
    at org.mariadb.jdbc.internal.failover.impl.AuroraListener.reconnectFailedConnection(AuroraListener.java:203)
    at org.mariadb.jdbc.internal.failover.thread.FailoverLoop.doRun(FailoverLoop.java:84)
    at org.mariadb.jdbc.internal.failover.thread.TerminableRunnable.run(TerminableRunnable.java:80)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

"MariaDb-failover-2" #273 daemon prio=5 os_prio=0 tid=0x00007f4b04008000 nid=0x18e4 waiting on condition [0x00007f4b0c5d5000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.mariadb.jdbc.internal.protocol.AuroraProtocol.loop(AuroraProtocol.java:269)
    at org.mariadb.jdbc.internal.failover.impl.AuroraListener.reconnectFailedConnection(AuroraListener.java:203)
    at org.mariadb.jdbc.internal.failover.thread.FailoverLoop.doRun(FailoverLoop.java:84)
    at org.mariadb.jdbc.internal.failover.thread.TerminableRunnable.run(TerminableRunnable.java:80)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

"MariaDb-failover-1" #265 daemon prio=5 os_prio=0 tid=0x00007f4b2407e800 nid=0x18a1 waiting on condition [0x00007f4b0c2d4000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.mariadb.jdbc.internal.protocol.AuroraProtocol.loop(AuroraProtocol.java:269)
    at org.mariadb.jdbc.internal.failover.impl.AuroraListener.reconnectFailedConnection(AuroraListener.java:203)
    at org.mariadb.jdbc.internal.failover.thread.FailoverLoop.doRun(FailoverLoop.java:84)
    at org.mariadb.jdbc.internal.failover.thread.TerminableRunnable.run(TerminableRunnable.java:80)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

"VM Thread" os_prio=0 tid=0x00007f4b540db800 nid=0xf37 runnable 

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f4b54069000 nid=0xf35 runnable 

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f4b5406a800 nid=0xf36 runnable 

"VM Periodic Task Thread" os_prio=0 tid=0x00007f4b54132800 nid=0xf3e waiting on condition 

JNI global references: 2520

十六进制 LWP 的 PID：

6373 18e5
6374 18e6
6305 18a1
6375 18e7
6372 18e4

=========== 更多信息

我们的 API 有 5 种不同的数据库配置 - 5 种不同的数据库。每个都有一个连接字符串，例如jdbc:mysql:aurora://some-aurora-cluster

请注意，aurora 模式用于更好地体验故障转移，因此它解释了 5 个轻量级进程。但他们很健谈，导致许多TIME_WAIT 并可能导致 CPU 使用率升高。

以前有没有人遇到过这种情况，您是如何缓解这种情况的？我仍然想使用aurora 模式（或等效的模式），这样我们就不必在数据库故障转移时重新启动应用程序。

【问题讨论】：

    标签： mysql scala playframework amazon-aurora mariadb-connector-c


    【解决方案1】：

    经过几天的搜索和研究，我终于钻研了MariaDb驱动代码，特别是在线程转储显示的区域，并一直跟踪代码堆栈。

    https://github.com/mariadb-corporation/mariadb-connector-j/blob/master/src/main/java/org/mariadb/jdbc/internal/protocol/AuroraProtocol.java

    我找到了对设置 retriesAllDown 的引用，其默认值为 120。进一步阅读 MariaDb 驱动程序知识库页面，我还发现了另一个设置 failoverLoopRetries，其默认值为 120 .

    您可以在此处阅读有关 MariaDb 驱动程序设置的更多信息：https://github.com/mariadb-corporation/mariadb-connector-j/blob/3bc66153b51aca188afc50ff35a0123f16c099ed/src/main/java/org/mariadb/jdbc/util/DefaultOptions.java

    对于我们的团队和 API，我们对 12 的值（默认值的 10%）感到满意，并决定对这两种设置都使用该值，所以这里是修改后的连接字符串：

    jdbc:mysql:aurora://some-aurora-cluster?retriesAllDown=12&failoverLoopRetries=12

    这显着降低了 CPU 使用率，并且仍然保持了我们需要的故障转移功能。

    希望这个答案能帮助别人。在它至少对其他 10 人有所帮助之前，我不会将其标记为原始问题的答案。

    【讨论】：

      猜你喜欢
      • 2018-05-13
      • 2015-04-17
      • 1970-01-01
      • 2019-10-28
      • 2023-04-08
      • 2016-06-18
      • 1970-01-01
      • 2022-07-04
      • 1970-01-01
      相关资源
      最近更新 更多
      热门标签
      Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode