【问题标题】:Elasticsearch service failing when /tmp is mounted noexec [duplicate]挂载 /tmp 时 Elasticsearch 服务失败 noexec [重复]
【发布时间】:2021-03-04 16:55:31
【问题描述】:

/tmpnoexec 一起挂载时,Elasticsearch 服务失败。当我尝试使用exec 重新挂载它时,服务正在启动。

以下链接是否仍然有效或有永久解决方案或变通方法?

https://www.elastic.co/guide/en/elasticsearch/reference/current/executable-jna-tmpdir.html

问。我想知道我们如何更改它试图在 /tmp 中写入的 Elasticsearch 日志的路径? (类似 '/var/' 或 '/opt' 的东西已经被挂载为 exec)

问。您还可以告诉我,如果此日志是临时的,并且弹性搜索服务会在一段时间后将其删除吗?如果是,那么只有将路径更改为“/var”或“/opt”之类的东西才有意义

以下是日志。

1.

systemctl status elasticsearch.service

elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/elasticsearch.service.d
           └─startup-timeout.conf
   Active: failed (Result: exit-code) since Fri 2020-11-20 14:52:12 IST; 16min ago
     Docs: http://www.elastic.co
  Process: 48882 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 48882 (code=exited, status=1/FAILURE)
   CGroup: /system.slice/elasticsearch.service

Nov 20 14:51:34 <HOSTNAME> systemd[1]: Starting Elasticsearch...
Nov 20 14:51:38 <HOSTNAME> elasticsearch[48882]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will ...release.
Nov 20 14:52:12 <HOSTNAME> elasticsearch[48882]: ERROR: [1] bootstrap checks failed
Nov 20 14:52:12 <HOSTNAME> elasticsearch[48882]: [1]: system call filters failed to install; check the logs and fix your configuration or disable sy...own risk
Nov 20 14:52:12 <HOSTNAME> systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 20 14:52:12 <HOSTNAME> systemd[1]: Failed to start Elasticsearch.
Nov 20 14:52:12 <HOSTNAME> systemd[1]: Unit elasticsearch.service entered failed state.
Nov 20 14:52:12 <HOSTNAME> systemd[1]: elasticsearch.service failed.

journalctl -fu elasticsearch

-- Logs begin at Tue 2020-11-17 13:09:05 IST. --
Nov 20 14:51:38 <HOSTNAME> elasticsearch[48882]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Nov 20 14:52:12 <HOSTNAME> elasticsearch[48882]: ERROR: [1] bootstrap checks failed
Nov 20 14:52:12 <HOSTNAME> elasticsearch[48882]: [1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
Nov 20 14:52:12 <HOSTNAME> systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 20 14:52:12 <HOSTNAME> systemd[1]: Failed to start Elasticsearch.
Nov 20 14:52:12 <HOSTNAME> systemd[1]: Unit elasticsearch.service entered failed state.
Nov 20 14:52:12 <HOSTNAME> systemd[1]: elasticsearch.service failed.

/var/log/elasticsearch/

[2020-11-20T14:51:53,369][WARN ][o.e.b.Natives            ] [elasticsearch_1] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-9221628356394379624/jna--1985354563/jna10392232823311871934.tmp: /tmp/elasticsearch-9221628356394379624/jna--1985354563/jna10392232823311871934.tmp: failed to map segment from shared object: Operation not permitted
        at java.lang.ClassLoader$NativeLibrary.load0(Native Method) ~[?:?]
        at java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2440) ~[?:?]
        at java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2497) ~[?:?]
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2694) ~[?:?]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2627) ~[?:?]
        at java.lang.Runtime.load0(Runtime.java:744) ~[?:?]
        at java.lang.System.load(System.java:1870) ~[?:?]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:?]
        at java.lang.Class.forName(Class.java:333) ~[?:?]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) [elasticsearch-cli-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.4.0.jar:7.4.0]
[2020-11-20T14:51:53,380][WARN ][o.e.b.Natives            ] [elasticsearch_1] cannot check if running as root because JNA is not available
[2020-11-20T14:51:53,380][WARN ][o.e.b.Natives            ] [elasticsearch_1] cannot install system call filter because JNA is not available
[2020-11-20T14:51:53,381][WARN ][o.e.b.Natives            ] [elasticsearch_1] cannot register console handler because JNA is not available
[2020-11-20T14:51:53,382][WARN ][o.e.b.Natives            ] [elasticsearch_1] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2020-11-20T14:51:53,382][WARN ][o.e.b.Natives            ] [elasticsearch_1] cannot getrlimit RLIMIT_AS because JNA is not available
[2020-11-20T14:51:53,382][WARN ][o.e.b.Natives            ] [elasticsearch_1] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2020-11-20T14:51:53,612][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch_1] using [1] data paths, mounts [[/opt (/dev/mapper/appvg-lv_opt)]], net usable_space [121.8gb], net total_space [147.5gb], types [ext4]
[2020-11-20T14:51:53,614][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch_1] heap size [5.1gb], compressed ordinary object pointers [true]
[2020-11-20T14:51:53,617][INFO ][o.e.n.Node               ] [elasticsearch_1] node name [elasticsearch_1], node ID [6OyoyA40S3-z6M7nAnCGIA], cluster name [<cluster name>]
[2020-11-20T14:51:53,617][INFO ][o.e.n.Node               ] [elasticsearch_1] version[7.4.0], pid[48882], build[default/rpm/22e1767283e61a198cb4db791ea66e3f11ab9910/2019-09-27T08:36:48.569419Z], OS[Linux/3.10.0-1127.19.1.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13/13+33]
[2020-11-20T14:51:53,618][INFO ][o.e.n.Node               ] [elasticsearch_1] JVM home [/usr/share/elasticsearch/jdk]
[2020-11-20T14:51:53,618][INFO ][o.e.n.Node               ] [elasticsearch_1] JVM arguments [-Xms5288m, -Xmx5288m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-9221628356394379624, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Xms5288m, -Xmx5288m, -Dio.netty.allocator.type=pooled, -XX:MaxDirectMemorySize=2772434944, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2020-11-20T14:51:55,967][INFO ][o.e.p.PluginsService     ] [elasticsearch_1] loaded module [aggs-matrix-stats]
[2020-11-20T14:51:55,968][INFO ][o.e.p.PluginsService     ] [elasticsearch_1] loaded module [analysis-common]
org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:242) [x-pack-monitoring-7.4.0.jar:7.4.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.0.jar:7.4.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.4.0.jar:7.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-11-20T14:52:11,962][ERROR][o.e.x.m.c.c.StatsCollector] [elasticsearch_1] collector [ccr_stats] failed to collect data
java.lang.NullPointerException: null
        at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:85) [x-pack-monitoring-7.4.0.jar:7.4.0]
        at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:242) [x-pack-monitoring-7.4.0.jar:7.4.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.0.jar:7.4.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.4.0.jar:7.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-11-20T14:52:11,963][ERROR][o.e.x.m.c.i.IndexStatsCollector] [elasticsearch_1] collector [index-stats] failed to collect data
java.lang.NullPointerException: null
        at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:85) [x-pack-monitoring-7.4.0.jar:7.4.0]
        at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:242) [x-pack-monitoring-7.4.0.jar:7.4.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.0.jar:7.4.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.4.0.jar:7.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-11-20T14:52:12,114][INFO ][o.e.t.TransportService   ] [elasticsearch_1] publish_address {<IP of localhost>:5300}, bound_addresses {0.0.0.0:5300}
[2020-11-20T14:52:12,123][INFO ][o.e.b.BootstrapChecks    ] [elasticsearch_1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-11-20T14:52:12,128][ERROR][o.e.b.Bootstrap          ] [elasticsearch_1] node validation exception
[1] bootstrap checks failed
[1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2020-11-20T14:52:12,132][INFO ][o.e.n.Node               ] [elasticsearch_1] stopping ...
[2020-11-20T14:52:12,155][INFO ][o.e.n.Node               ] [elasticsearch_1] stopped
[2020-11-20T14:52:12,156][INFO ][o.e.n.Node               ] [elasticsearch_1] closing ...
[2020-11-20T14:52:12,166][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch_1] fatal error in thread [Thread-3], exiting
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
        at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:34) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
        at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:33) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:66) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.close(SystemdPlugin.java:87) ~[?:?]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:104) ~[elasticsearch-core-7.4.0.jar:7.4.0]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:86) ~[elasticsearch-core-7.4.0.jar:7.4.0]
        at org.elasticsearch.node.Node.close(Node.java:881) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:104) ~[elasticsearch-core-7.4.0.jar:7.4.0]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:62) ~[elasticsearch-core-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:186) ~[elasticsearch-7.4.0.jar:7.4.0]

【问题讨论】:

    标签: java elasticsearch jna


    【解决方案1】:

    您的堆栈跟踪指向问题的根源:

    java.lang.UnsatisfiedLinkError中:/tmp/elasticsearch-9221628356394379624/jna--1985354563/jna10392232823311871934.tmp:/tmp/elasticsearch-9221628356394379624/jna--1985354563/jna10392232823311871934.tmp:未能映射从共享对象段:操作不允许

    link you provided 准确地描述了该问题:JNA 将其自己的本机库存根提取到您计算机上的临时目录中以便工作。当/tmp 目录没有此库的正确权限时,您将无法使用它。你可以see the loading code here

    关于您的问题,是的,可以在其他地方提取该库。一种简单的解决方案是自己为您的系统提取适当的jnidispatch 库。

    您可以设置任何位置并设置jna.boot.library.path 或使用系统路径设置jna.nosys=false。见javadoc for the Native class

    当 JNA 类被加载时,本地共享库 (jnidispatch) 也会被加载。尝试从 jna.boot.library.path(如果已定义)中定义的任何路径加载它,然后使用 System.loadLibrary(java.lang.String) 加载系统库路径,除非 jna.nosys=true。如果未找到,则将从类路径中提取相应的库(如果在 jar 文件中找到,则将其提取到临时目录中)并从那里加载,除非 jna.noclasspath=true。如果您的系统对文件的执行或加载有额外的安全限制(例如 SELinux),您可能应该将本机库安装在可访问的位置并相应地配置您的系统,而不是依赖 JNA 从其自己的 jar 中提取库文件。

    【讨论】:

    • 嗨丹尼尔,谢谢。我不是开发人员,所以我不明白我需要在哪里设置不同的路径。你能告诉我我需要在哪个文件中设置什么变量吗?谢谢,
    • @pratkul007 如果你有 Java 代码,你可以使用 System.setProperty("jna.boot.library.path", "\foo\bar") 来定位 dll。如果可以控制命令行开关,可以在命令行字符串中添加-Djna.boot.library.path=\foo\bar。见this answer
    • @pratkul007 在 Elasticsearch 中,您使用 -E 而不是在 Java 中使用 -DSee here
    • 感谢@Daniel 为我找到。只是想在哪个文件中我需要在 Elasticsearch 配置中添加这个变量。
    • 嗨丹尼尔,谢谢。我找到了下面的链接。我会尽力让你知道。谢谢。 stackoverflow.com/questions/47824643/…
    猜你喜欢
    • 2015-04-05
    • 1970-01-01
    • 2016-04-09
    • 1970-01-01
    • 1970-01-01
    • 2020-01-15
    • 2017-05-18
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多