【发布时间】:2015-07-20 13:41:23
【问题描述】:
我正在 Docker 中运行 RethinkDB。在我们搬到新的数据中心之前,一切都运行得很好(但我不确定这是否与迁移有关)。这就是正在发生的事情。
我启动了 rethinkdb 容器,一段时间内一切都运行良好。一段时间后(在一小时或更长时间之间变化),我在 Docker 日志中看到以下内容(以黄色突出显示):
我完全不知道为什么它会随机从系统接收 SIGTERM。任何想法将不胜感激!
编辑:我正在为 SIGTERM 添加日志文件的 sn-p。根据时间戳,似乎没有任何类型的模式。
2015-07-15T16:15:02.888762613 663165.661585s notice: Server got SIGTERM from pid 0, uid 0; shutting down...
2015-07-17T17:02:11.562306701 13322.914561s notice: Server got SIGTERM from pid 0, uid 0; shutting down...
2015-07-19T18:31:12.499022237 96786.220054s notice: Server got SIGTERM from pid 0, uid 0; shutting down...
2015-07-20T13:52:44.493304030 69690.608865s notice: Server got SIGTERM from pid 0, uid 0; shutting down...
编辑 2:我在 Docker 之外运行了 RethinkDB,我在日志中看到:错误:工作进程无法与主进程重新同步。不确定是否有什么需要担心的。它似乎根本不会影响 RethinkDB 实例(所有客户端都保持连接)。
2015-07-21T06:53:10.663375859 0.116098s info: Automatically using cache size of 10702 MB
2015-07-21T06:53:10.676277261 0.128998s notice: Listening for intracluster connections on port 29015
2015-07-21T06:53:10.684504354 0.137225s notice: Listening for client driver connections on port 28015
2015-07-21T06:53:10.685485550 0.138206s notice: Listening for administrative HTTP connections on port 8080
2015-07-21T06:53:10.686313405 0.139034s notice: Listening on addresses: 127.0.0.1, 172.17.42.1, 192.151.151.122, ::1, fe80::1879:43ff:fe5e:bdb2%34, fe80::62eb:69ff:fe07:d986%2, fe80::b837:f2ff:fecd:d5cd%4
2015-07-21T06:53:10.686316632 0.139037s notice: Server ready, "0aa312e817ef_nrx" 069ac5b3-9f43-4bbe-9022-c1f006790e99
2015-07-21T06:53:11.558116243 1.010837s error: worker process failed to resynchronize with main process
2015-07-21T06:53:11.558122179 1.010843s notice: A newer version of the RethinkDB server is available: 2.0.4. You can read the changelog at <https://github.com/rethinkdb/rethinkdb/releases>.
编辑 3: 我在这里发现了另一个问题,我认为这可能是真正的问题。重新思考适配器(在应用程序中)保持与数据库服务器的连接已建立,这耗尽了系统中可用的文件描述符/端口。这是lsof 的示例打印输出。 注意这只是一个简短的列表。当多人使用系统时,有成百上千个保持打开状态
node 11633 [username_ommitted] 201u IPv4 0x53153575d33d64bb 0t0 TCP 192.168.1.142:61041->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 202u IPv4 0x53153575d33fa65b 0t0 TCP 192.168.1.142:61053->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 203u IPv4 0x53153575dd6a5d8b 0t0 TCP 192.168.1.142:61043->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 204u IPv4 0x53153575bff6717b 0t0 TCP 192.168.1.142:61044->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 206u IPv4 0x53153575d33e54bb 0t0 TCP 192.168.1.142:61049->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 207u IPv4 0x53153575d33ef4bb 0t0 TCP 192.168.1.142:61050->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 208u IPv4 0x53153575d33f2a4b 0t0 TCP 192.168.1.142:61051->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 209u IPv4 0x53153575c333a17b 0t0 TCP 192.168.1.142:61054->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 210u IPv4 0x53153575d33b47fb 0t0 TCP 192.168.1.142:61056->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 211u IPv4 0x53153575d33de17b 0t0 TCP 192.168.1.142:61057->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 212u IPv4 0x53153575d33f065b 0t0 TCP 192.168.1.142:61058->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 213u IPv4 0x53153575bff67a4b 0t0 TCP 192.168.1.142:61059->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 216u IPv4 0x53153575dd68f31b 0t0 TCP 192.168.1.142:61062->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 217u IPv4 0x53153575dd675a4b 0t0 TCP 192.168.1.142:61063->[RETHINK_IP]:28015 (ESTABLISHED)
【问题讨论】:
-
感觉像 logrotate 或类似的东西。你能监控它发生的时间,看看是否有规律吗?在任何情况下,它看起来确实像主机正在杀死某些东西......(甚至可能是被监控守护进程或排序杀死的长时间运行的进程)你检查过你的主机/操作吗?
-
@Gekkie 是的,我会尝试从日志中获取时间戳。 rethinkdb 是否内置了自己的日志轮换?我假设确实如此。有什么方法可以检查配置吗?
-
你可以
docker stats container_id文档docs.docker.com/reference/commandline/stats或者进入容器docker exec -it container_id bash文档docs.docker.com/reference/commandline/exec进行调试/监控/检查 -
@Gekkie 我添加了日志文件输出。似乎没有任何模式:/
-
您能否发布实际的日志文件文本而不是日志文件的图片?如果遇到同样的问题,这通常会使人们更容易通过搜索引擎找到问题,并且也会让那些试图帮助你的人更轻松。谢谢!