即使有 0 个未分配的分片，Elasticsearch 集群状态仍为黄色答案

【问题标题】：Elasticsearch cluster status still yellow even with 0 unassigned shards即使有 0 个未分配的分片，Elasticsearch 集群状态仍为黄色
【发布时间】：2019-11-22 02:59:22
【问题描述】：

我们有一种情况，未分配的分片为 0，但 Elasticsearch 集群状态仍为黄色。另一方面，一些索引数据正在重复。我们看到相同的数据输入超过 3 到 4 次。下面是集群的当前状态

{
  "cluster_name": "elasticsearch",
  "status": "yellow",
  "timed_out": false,
  "number_of_nodes": 5,
  "number_of_data_nodes": 2,
  "active_primary_shards": 3888,
  "active_shards": 7775,
  "relocating_shards": 0,
  "initializing_shards": 1,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 99.98713991769547
}

Elasticsearch 日志

[2019-11-22T02:42:19,929][INFO ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-b94556bc6-q62jc] [gc][1605322] overhead, spent [347ms] collecting in the last [1.1s]
[2019-11-22T02:42:20,424][WARN ][o.e.x.m.MonitoringService] [elasticsearch-master-b94556bc6-q62jc] monitoring execution failed
org.elasticsearch.xpack.monitoring.exporter.ExportException: Exception when closing export bulk
    at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$1$1.<init>(ExportBulk.java:95) ~[x-pack-monitoring-6.4.3.jar:6.4.3]
    at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$1.onFailure(ExportBulk.java:93) [x-pack-monitoring-6.4.3.jar:6.4.3]
    at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound$1.onResponse(ExportBulk.java:206) [x-pack-monitoring-6.4.3.jar:6.4.3]
    at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound$1.onResponse(ExportBulk.java:200) [x-pack-monitoring-6.4.3.jar:6.4.3]
    at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:96) [x-pack-core-6.4.3.jar:6.4.3]
    at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.lambda$doFlush$0(ExportBulk.java:164) [x-pack-monitoring-6.4.3.jar:6.4.3]
    at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:68) [elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$1(LocalBulk.java:115) [x-pack-monitoring-6.4.3.jar:6.4.3]
    at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:68) [elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:50) [elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:91) [elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173) [elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) [elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) [elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87) [elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76) [elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) [elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.client.support.AbstractClient.bulk(AbstractClient.java:484) [elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:74) [x-pack-core-6.4.3.jar:6.4.3]
    at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.doFlush(LocalBulk.java:108) [x-pack-monitoring-6.4.3.jar:6.4.3]
    at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.flush(ExportBulk.java:60) [x-pack-monitoring-6.4.3.jar:6.4.3]
    at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.lambda$doFlush$1(ExportBulk.java:154) [x-pack-monitoring-6.4.3.jar:6.4.3]
    at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:81) [x-pack-core-6.4.3.jar:6.4.3]
    at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:170) [x-pack-monitoring-6.4.3.jar:6.4.3]
    at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.flushAndClose(ExportBulk.java:84) [x-pack-monitoring-6.4.3.jar:6.4.3]
    at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.close(ExportBulk.java:74) [x-pack-monitoring-6.4.3.jar:6.4.3]
    at org.elasticsearch.xpack.monitoring.exporter.Exporters.export(Exporters.java:196) [x-pack-monitoring-6.4.3.jar:6.4.3]
    at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:258) [x-pack-monitoring-6.4.3.jar:6.4.3]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.4.3.jar:6.4.3]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514) [?:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.4.3.jar:6.4.3]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
    at java.lang.Thread.run(Thread.java:844) [?:?]
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulks
    at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.lambda$doFlush$0(ExportBulk.java:156) [x-pack-monitoring-6.4.3.jar:6.4.3]
    ... 29 more
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulk [default_local]
    ... 28 more
Caused by: java.lang.IllegalStateException: There are no ingest nodes in this cluster, unable to forward request to an ingest node.
    at org.elasticsearch.action.ingest.IngestActionForwarder.randomIngestNode(IngestActionForwarder.java:58) ~[elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.action.ingest.IngestActionForwarder.forwardIngestRequest(IngestActionForwarder.java:51) ~[elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:136) ~[elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:85) ~[elasticsearch-6.4.3.jar:6.4.3]
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.4.3.jar:6.4.3]
    ... 23 more

这可能是什么原因？我们如何解决这个问题？

【问题讨论】：

发布您的 elasticsearch 日志。您的 _health 响应说有 1 个分片处于初始化状态。
@ArchitSaxena 使用弹性搜索日志更新问题。

标签： elasticsearch

【解决方案1】：

您可以使用Cluster allocation explain API：GET /_cluster/allocation/explain，让 Elasticsearch “解释”它无法分配分片的原因

【讨论】：

集群现在是绿色的。但是，它在绿色和黄色之间波动。此外，数据正在重复
Elasticsearch 偶尔会重新定位分片以优化负载分布是正常的，但最终重新定位的分片会被清理掉。如果您看到集群状态发生变化，这可能表明您面临延迟问题（网络速度慢）。在集群状态从绿色变为黄色时检查 API 并发布响应。
感谢丹尼尔的回复。我们如何检查网络问题？我无法进行任何 API 调用。我试过 /_cat/shards 超时了
这个话题太笼统了。如果您无法再与 Elasticsearch 通信（就像您刚才所做的那样），您就会检测到网络问题。当 Elasticsearch 由于某个节点不再可访问而采取行动时，您还可以在日志中看到条目。我建议首先调查丢失的分片主题。您只需要能够与集群中的一个节点进行通信。在节点的主机上本地工作并查询 ES。您需要确保http.host 包含值'local'。因此，您可以了解集群的外观以及无法分配分片的原因。