【发布时间】：2016-07-09 01:42:08
【问题描述】：
我最近将 ES 从版本 1.1.0 升级到了 1.6。所有配置都保持不变，但几乎 5% 的调用开始抛出 EsRejectedExecutionException，这在我运行 ES 1.1.0 时从未发生过
完整的堆栈跟踪如下：
org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to execute phase [merge], [reduce]
at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction$1.onFailure(TransportSearchQueryAndFetchAction.java:93)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.onRejection(AbstractRunnable.java:65)
at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:85)
at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.moveToSecondPhase(TransportSearchQueryAndFetchAction.java:78)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.innerMoveToSecondPhase(TransportSearchTypeAction.java:403)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:202)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onResult(TransportSearchTypeAction.java:178)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onResult(TransportSearchTypeAction.java:175)
at org.elasticsearch.search.action.SearchServiceTransportAction$12.handleResponse(SearchServiceTransportAction.java:346)
at org.elasticsearch.search.action.SearchServiceTransportAction$12.handleResponse(SearchServiceTransportAction.java:337)
at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:163)
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:132)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 500) on org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction$1@4cd69494
at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:79)
升级时所有配置都保持不变，所以我想知道是什么导致了这个问题。线程池大小似乎已满，所以我尝试通过增加线程池和Queue_size来进行测试。但我无法理解导致问题的从 1.1 到 1.6 的变化。
我阅读了发行说明，但没有找到与此相关的任何内容。我正在搜索任何配置参数的任何默认值更改，但我没有找到任何似乎相关的东西。
我发现this 问题与我的问题非常相似，但在版本 1.1.2 中已解决，因此这不应该导致错误。
如果有人能给我指点如何找到问题，那将是一个很大的帮助。
【问题讨论】：
-
你有多少个节点，每个节点有多少 CPU？
-
@Val 我有 2 个集群，每个集群 3 个节点。每个节点有 32 个 CPU。使用相同的配置，我从未在 1.1 上收到异常，但在 1.6 中出现错误
-
我阅读了stackoverflow.com/questions/27793530/…，这将缓解问题但不能解决问题。我认为这与 2 集群配置有关。
-
对于 1.6，search thread pool size 小于 1.3 (the documentation for 1.1 is not available anymore on the website)。 3x # of available processors 用于 1.3 和 int((# of available_processors * 3) / 2) + 1 用于 1.6。这或许可以解释它。
-
@AndreiStefan 我有 32 个 CPU，我已将搜索线程池大小设置为 20。这应该不是问题，因为它在 1.1 中使用此配置工作得非常好
标签： java elasticsearch