每个工作进程的 Apache Storm 2 java 进程

Question

当我在运行风暴拓扑的机器上运行 ps aux | grep java 时，我可以看到 2 个 Java 进程正在运行。

ps -ef | grep java | grep "storm"
UID        PID  PPID  C STIME TTY          TIME CMD
test-3p-s+ 10857 10661  0 Apr12 ?        1-02:21:25 java -server -Ddaemon.name=supervisor -Dstorm.options= -Dstorm.home=/usr/share/apache-storm -Dstorm.log.dir=/usr/share/apache-storm/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/share/apache-storm/lib/log4j-slf4j-impl-2.1.jar:/usr/share/apache-storm/lib/reflectasm-1.10.1.jar:/usr/share/apache-storm/lib/kryo-3.0.3.jar:/usr/share/apache-storm/lib/storm-core-1.0.1.jar:/usr/share/apache-storm/lib/servlet-api-2.5.jar:/usr/share/apache-storm/lib/log4j-core-2.1.jar:/usr/share/apache-storm/lib/log4j-api-2.1.jar:/usr/share/apache-storm/lib/objenesis-2.1.jar:/usr/share/apache-storm/lib/disruptor-3.3.2.jar:/usr/share/apache-storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/share/apache-storm/lib/storm-rename-hack-1.0.1.jar:/usr/share/apache-storm/lib/asm-5.0.3.jar:/usr/share/apache-storm/lib/minlog-1.3.0.jar:/usr/share/apache-storm/lib/slf4j-api-1.7.7.jar:/usr/share/apache-storm/lib/clojure-1.7.0.jar:/usr/share/apache-storm/conf -Xmx2048m -Dlogfile.name=supervisor.log -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dlog4j.configurationFile=/usr/share/apache-storm/log4j2/cluster.xml org.apache.storm.daemon.supervisor
test-3p-s+ 16121 10857  0 Oct13 ?        00:11:31 java -cp /usr/share/apache-storm/lib/log4j-slf4j-impl-2.1.jar:/usr/share/apache-storm/lib/reflectasm-1.10.1.jar:/usr/share/apache-storm/lib/kryo-3.0.3.jar:/usr/share/apache-storm/lib/storm-core-1.0.1.jar:/usr/share/apache-storm/lib/servlet-api-2.5.jar:/usr/share/apache-storm/lib/log4j-core-2.1.jar:/usr/share/apache-storm/lib/log4j-api-2.1.jar:/usr/share/apache-storm/lib/objenesis-2.1.jar:/usr/share/apache-storm/lib/disruptor-3.3.2.jar:/usr/share/apache-storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/share/apache-storm/lib/storm-rename-hack-1.0.1.jar:/usr/share/apache-storm/lib/asm-5.0.3.jar:/usr/share/apache-storm/lib/minlog-1.3.0.jar:/usr/share/apache-storm/lib/slf4j-api-1.7.7.jar:/usr/share/apache-storm/lib/clojure-1.7.0.jar:/usr/share/apache-storm/conf:/var/lib/apache-storm/supervisor/stormdist/TestTopology-97-1665657576/stormjar.jar -Xmx64m -Dlogfile.name=worker.log -Dstorm.home=/usr/share/apache-storm -Dworkers.artifacts=/usr/share/apache-storm/logs/workers-artifacts -Dstorm.id=TestTopology-97-1665657576 -Dworker.id=3324b1de-e8ce-48e6-b065-3fd1f0659c20 -Dworker.port=6706 -Dstorm.log.dir=/usr/share/apache-storm/logs -Dlog4j.configurationFile=/usr/share/apache-storm/log4j2/worker.xml -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector org.apache.storm.LogWriter java -server -Xmx4g -Xms4g -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=16706 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:+UseG1GC -XX:+AggressiveOpts -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/apache-storm/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Xmx4g -Djava.library.path=/var/lib/apache-storm/supervisor/stormdist/TestTopology-97-1665657576/resources/Linux-amd64:/var/lib/apache-storm/supervisor/stormdist/TestTopology-97-1665657576/resources:/usr/local/lib:/opt/local/lib:/usr/lib -Dlogfile.name=worker.log -Dstorm.home=/usr/share/apache-storm -Dworkers.artifacts=/usr/share/apache-storm/logs/workers-artifacts -Dstorm.conf.file= -Dstorm.options= -Dstorm.log.dir=/usr/share/apache-storm/logs -Djava.io.tmpdir=/var/lib/apache-storm/workers/3324b1de-e8ce-48e6-b065-3fd1f0659c20/tmp -Dlogging.sensitivity=S3 -Dlog4j.configurationFile=/usr/share/apache-storm/log4j2/worker.xml -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector -Dstorm.id=TestTopology-97-1665657576 -Dworker.id=3324b1de-e8ce-48e6-b065-3fd1f0659c20 -Dworker.port=6706 -cp /usr/share/apache-storm/lib/log4j-slf4j-impl-2.1.jar:/usr/share/apache-storm/lib/reflectasm-1.10.1.jar:/usr/share/apache-storm/lib/kryo-3.0.3.jar:/usr/share/apache-storm/lib/storm-core-1.0.1.jar:/usr/share/apache-storm/lib/servlet-api-2.5.jar:/usr/share/apache-storm/lib/log4j-core-2.1.jar:/usr/share/apache-storm/lib/log4j-api-2.1.jar:/usr/share/apache-storm/lib/objenesis-2.1.jar:/usr/share/apache-storm/lib/disruptor-3.3.2.jar:/usr/share/apache-storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/share/apache-storm/lib/storm-rename-hack-1.0.1.jar:/usr/share/apache-storm/lib/asm-5.0.3.jar:/usr/share/apache-storm/lib/minlog-1.3.0.jar:/usr/share/apache-storm/lib/slf4j-api-1.7.7.jar:/usr/share/apache-storm/lib/clojure-1.7.0.jar:/usr/share/apache-storm/conf:/var/lib/apache-storm/supervisor/stormdist/TestTopology-97-1665657576/stormjar.jar org.apache.storm.daemon.worker TestTopology-97-1665657576 b64def27-088a-41ae-95a9-b5195c88bbfa 6706 3324b1de-e8ce-48e6-b065-3fd1f0659c20
test-3p-s+ 16158 16121 87 Oct13 ?        15-13:11:48 java -server -Xmx4g -Xms4g -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=16706 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:+UseG1GC -XX:+AggressiveOpts -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/apache-storm/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Xmx4g -Djava.library.path=/var/lib/apache-storm/supervisor/stormdist/TestTopology-97-1665657576/resources/Linux-amd64:/var/lib/apache-storm/supervisor/stormdist/TestTopology-97-1665657576/resources:/usr/local/lib:/opt/local/lib:/usr/lib -Dlogfile.name=worker.log -Dstorm.home=/usr/share/apache-storm -Dworkers.artifacts=/usr/share/apache-storm/logs/workers-artifacts -Dstorm.conf.file= -Dstorm.options= -Dstorm.log.dir=/usr/share/apache-storm/logs -Djava.io.tmpdir=/var/lib/apache-storm/workers/3324b1de-e8ce-48e6-b065-3fd1f0659c20/tmp -Dlogging.sensitivity=S3 -Dlog4j.configurationFile=/usr/share/apache-storm/log4j2/worker.xml -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector -Dstorm.id=TestTopology-97-1665657576 -Dworker.id=3324b1de-e8ce-48e6-b065-3fd1f0659c20 -Dworker.port=6706 -cp /usr/share/apache-storm/lib/log4j-slf4j-impl-2.1.jar:/usr/share/apache-storm/lib/reflectasm-1.10.1.jar:/usr/share/apache-storm/lib/kryo-3.0.3.jar:/usr/share/apache-storm/lib/storm-core-1.0.1.jar:/usr/share/apache-storm/lib/servlet-api-2.5.jar:/usr/share/apache-storm/lib/log4j-core-2.1.jar:/usr/share/apache-storm/lib/log4j-api-2.1.jar:/usr/share/apache-storm/lib/objenesis-2.1.jar:/usr/share/apache-storm/lib/disruptor-3.3.2.jar:/usr/share/apache-storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/share/apache-storm/lib/storm-rename-hack-1.0.1.jar:/usr/share/apache-storm/lib/asm-5.0.3.jar:/usr/share/apache-storm/lib/minlog-1.3.0.jar:/usr/share/apache-storm/lib/slf4j-api-1.7.7.jar:/usr/share/apache-storm/lib/clojure-1.7.0.jar:/usr/share/apache-storm/conf:/var/lib/apache-storm/supervisor/stormdist/TestTopology-97-1665657576/stormjar.jar org.apache.storm.daemon.worker TestTopology-97-1665657576 b64def27-088a-41ae-95a9-b5195c88bbfa 6706 3324b1de-e8ce-48e6-b065-3fd1f0659c20

似乎所有 3 个 java 进程都与父客户端相关，因为 pid 的 PPID 与前一个 java 进程的 pid 相同。但是有人可以解释一下：

1. 这些进程是如何产生的？ （使用java.lang.Process API？）
2. 谁在监视谁？
3. 为什么我们需要第二个 Java 进程？

Answer 1

这可能只是部分答案。在您的终端中，您基本上可以看到 3 个进程：

1. Supervisor - 这是负责协调LogWriters 和Workers. 的总体风暴进程，因此它基本上启动并杀死它们，并且它有自己的日志文件。您也可以配置要生成多少个工人（=插槽）
2. LogWriter：这似乎负责生成日志 - 但我真的不确定为什么需要一个单独的 JVM。
3. Worker - 这是执行拓扑的一个运算符（螺栓或喷口）。因此，这是由主管创建的，并且只要拓扑存在，它就会存在。您可以并行运行多个工作人员。

   我也不确定他们为什么要这样设计——所以请看看我单独的question。对于生成的确切时间和调用，您需要深入了解它们的实现——我无法在第一眼看到相关代码——但如果你能分享你的结果会很高兴。