由于据称磁盘可用空间不足，ElasticSearch 错误地停止了索引答案

【问题标题】：ElasticSearch wrongly halting indexing because of supposedly low disk free space由于据称磁盘可用空间不足，ElasticSearch 错误地停止了索引
【发布时间】：2019-11-24 20:48:44
【问题描述】：

我在一个足够大（满足我的需要）磁盘上安装了一个非常简单的 ES6：

root@ip-172-17-20-222:/home/admin# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7.8G     0  7.8G   0% /dev
tmpfs           1.6G  155M  1.4G  10% /run
/dev/nvme0n1p1   59G   34G   24G  59% /
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
tmpfs           1.6G     0  1.6G   0% /run/user/1000

如您所见，磁盘上有大量可用空间。

我使用它的 http API 配置了 ES：

curl -XPUT -H "Content-Type: application/json"  "http://172.17.20.222:9200/_cluster/settings" -d'
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.low": "85%",
    "cluster.routing.allocation.disk.watermark.high": "90%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "95%",
    "cluster.info.update.interval": "1m"
  }
}'
curl -XPUT -H "Content-Type: application/json" http://172.17.20.222:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

但几分钟后：

[2019-11-22T16:02:46,889][INFO ][o.e.c.r.a.DiskThresholdMonitor] [C25At0X] low disk watermark [85%] exceeded on [C25At0XLRAyaZ00zegY20A][C25At0X][/var/lib/elasticsearch/nodes/0] free: 6gb[10.2%], replicas will not be assigned to this node
[2019-11-22T16:03:46,890][WARN ][o.e.c.r.a.DiskThresholdMonitor] [C25At0X] flood stage disk watermark [95%] exceeded on [C25At0XLRAyaZ00zegY20A][C25At0X][/var/lib/elasticsearch/nodes/0] free: 2.6gb[4.4%], all indices on this node will be marked read-only
[2019-11-22T16:04:46,892][WARN ][o.e.c.r.a.DiskThresholdMonitor] [C25At0X] flood stage disk watermark [95%] exceeded on [C25At0XLRAyaZ00zegY20A][C25At0X][/var/lib/elasticsearch/nodes/0] free: 2.5gb[4.3%], all indices on this node will be marked read-only
[2019-11-22T16:05:46,897][WARN ][o.e.c.r.a.DiskThresholdMonitor] [C25At0X] flood stage disk watermark [95%] exceeded on [C25At0XLRAyaZ00zegY20A][C25At0X][/var/lib/elasticsearch/nodes/0] free: 2.5gb[4.2%], all indices on this node will be marked read-only
[2019-11-22T16:06:46,899][WARN ][o.e.c.r.a.DiskThresholdMonitor] [C25At0X] flood stage disk watermark [95%] exceeded on [C25At0XLRAyaZ00zegY20A][C25At0X][/var/lib/elasticsearch/nodes/0] free: 2.4gb[4.1%], all indices on this node will be marked read-only
[2019-11-22T16:07:46,901][WARN ][o.e.c.r.a.DiskThresholdMonitor] [C25At0X] flood stage disk watermark [95%] exceeded on [C25At0XLRAyaZ00zegY20A][C25At0X][/var/lib/elasticsearch/nodes/0] free: 2.3gb[4%], all indices on this node will be marked read-only
[2019-11-22T16:08:46,903][WARN ][o.e.c.r.a.DiskThresholdMonitor] [C25At0X] flood stage disk watermark [95%] exceeded on [C25At0XLRAyaZ00zegY20A][C25At0X][/var/lib/elasticsearch/nodes/0] free: 2.3gb[3.9%], all indices on this node will be marked read-only
[2019-11-22T16:09:46,905][WARN ][o.e.c.r.a.DiskThresholdMonitor] [C25At0X] flood stage disk watermark [95%] exceeded on [C25At0XLRAyaZ00zegY20A][C25At0X][/var/lib/elasticsearch/nodes/0] free: 2.2gb[3.8%], all indices on this node will be marked read-only
[2019-11-22T16:10:46,908][INFO ][o.e.c.r.a.DiskThresholdMonitor] [C25At0X] rerouting shards: [one or more nodes has gone under the high or low watermark]

如果我理解正确，ES 会停止索引，因为它认为我的磁盘中没有足够的可用空间，但这似乎不正确。

我不认为问题与 inode 相关，根据：

root@ip-172-17-20-222:/home/admin# ulimit
unlimited
root@ip-172-17-20-222:/home/admin# df -ih
Filesystem     Inodes IUsed IFree IUse% Mounted on
udev             2.0M   269  2.0M    1% /dev
tmpfs            2.0M   683  2.0M    1% /run
/dev/nvme0n1p1   3.8M   44K  3.8M    2% /
tmpfs            2.0M     1  2.0M    1% /dev/shm
tmpfs            2.0M     2  2.0M    1% /run/lock
tmpfs            2.0M    15  2.0M    1% /sys/fs/cgroup
tmpfs            2.0M    11  2.0M    1% /run/user/1000

还有什么可能导致这种行为？

【问题讨论】：

标签： elasticsearch

【解决方案1】：

free: 2.5gb[4.3%]

这些数字与df 来自同一来源，但变化很快。看起来有些东西每分钟消耗大约 100MB 的磁盘空间。您将需要跟踪您的磁盘使用情况，因为它会随着时间的推移而发生变化，以便通过df 也能捕捉到这种情况。

【讨论】：