【问题标题】:RabbitMQ delay my message randomlyRabbitMQ 随机延迟我的消息
【发布时间】:2017-08-15 03:33:03
【问题描述】:

这是我的rabbitmqctl status

[{pid,32074},
 {running_applications,
     [{rabbitmq_management,"RabbitMQ Management Console","3.2.2"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.2"},
      {rabbit,"RabbitMQ","3.2.2"},
      {os_mon,"CPO  CXC 138 46","2.2.7"},
      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.2"},
      {webmachine,"webmachine","1.10.3-rmq3.2.2-gite9359c7"},
      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.2-git680dba8"},
      {xmerl,"XML parser","1.2.10"},
      {inets,"INETS  CXC 138 49","5.7.1"},
      {mnesia,"MNESIA  CXC 138 12","4.5"},
      {amqp_client,"RabbitMQ AMQP Client","3.2.2"},
      {sasl,"SASL  CXC 138 11","2.1.10"},
      {stdlib,"ERTS  CXC 138 10","1.17.5"},
      {kernel,"ERTS  CXC 138 10","2.14.5"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:32:32] [rq:32] [async-threads:30] [kernel-poll:true]\n"},
 {memory,
     [{total,1954308048},
      {connection_procs,619048024},
      {queue_procs,166111144},
      {plugins,4423520},
      {other_proc,46207032},
      {mnesia,44407568},
      {mgmt_db,331614464},
      {msg_index,6694584},
      {other_ets,30005328},
      {binary,63825504},
      {code,17629100},
      {atom,6531121},
      {other_system,617810659}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,53967541043},
 {disk_free_limit,50000000},
 {disk_free,51883839488},
 {file_descriptors,
     [{total_limit,655260},
      {total_used,12659},
      {sockets_limit,589732},
      {sockets_used,12657}]},
 {processes,[{limit,1048576},{used,125740}]},
 {run_queue,1},
 {uptime,33320350}]

我的队列大部分时间都是空的。但是我的 RabbitMQ 中有 84440 个交换和 8917 个队列。

我的问题是我发送的消息可能会延迟很长时间。

我试图从现场获取一些信息。这是我的监控图表,是什么时候发生延迟的。 我们可以看到概述和我的交换很好。但是我在队列中的消息是延迟交付和确认。

但从我的应用日志中。

2017-08-10 17:23:08.738 4219 INFO trove.openstack.common.rpc.amqp [-] [ProxyCallback]Received Message with Timestamp:2017-08-10T17:23:08, duration:0.737949s, unique_id:a17186068cae447bbada7a0f24ff45ef

17:23 消息立即收到。然后我将 ACK 发送回 MQ:

2017-08-10 17:23:08.739 4219 DEBUG trove.openstack.common.rpc.common [-] Consume Massage with ACK True

虽然我的17:43 消息延迟了 215.895117 秒,但我真的很困惑。

2017-08-10 17:43:53.895 4219 INFO trove.openstack.common.rpc.amqp [-] [ProxyCallback]Received Message with Timestamp:2017-08-10T17:40:18, duration:215.895117s, unique_id:dc04b94c8fa64978bc9d681b020f4500

【问题讨论】:

    标签: rabbitmq


    【解决方案1】:

    最后,我发现这是我的网络问题。 有时消息会在传递中丢失,并且 RabbitMQ 需要很长时间超时并重试消息传递。 结果我的消息延迟了。


    问题源于安全组规则,在 900s 没有数据包发送后,网关将断开 tcp 连接而无需重置(只是丢弃数据包)。

    系统范围的 tcp keep alive(tcp_keepalive_intvl) 设置为 1200s。

    所以我只是将tcp_keepalive_intvlsysctl 设置为600s,问题就解决了。

    【讨论】: