【问题标题】:jps not listing tomcat process and jmap fails to create heap dumpjps 未列出 tomcat 进程和 jmap 无法创建堆转储
【发布时间】:2021-04-20 13:01:29
【问题描述】:

我正在运行由用户 cpappt 启动并使用 pid: 1682650 运行的 tomcat (9.0.39.0) 服务器。运行此服务器的机器不是裸机但是VMware 机器和JVM 来自OpenJ9

> ps -ef | grep Bootstrap
cpappt   1682650       1  0 Jan01 ?        01:09:58 /srv/jdk11/bin/java
  -Djava.util.logging.config.file=/srv/test/cpappt/tomcat/conf/logging.properties
  -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -server 
  -Xms32m -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError
  -Dcrewportalenvironment=test
  -Denvironment=test
  -Doracle.net.tns_admin=/CONFIG/global/ORA/HA
  -Dspring.profiles.active=test,notification-services-intern
  -Djdk.tls.ephemeralDHKeySize=2048
  -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
  -Dorg.apache.catalina.security.SecurityListener.UMASK=0022
  -Djava.awt.headless=true
  -Dcom.sun.management.jmxremote
  -Dfile.encoding=UTF-8
  -Dsun.jnu.encoding=UTF-8
  -Dcom.sun.management.jmxremote.port=7305
  -Dcom.sun.management.jmxremote.ssl=false
  -Dcom.sun.management.jmxremote.password.file=conf/jmxremote.password
  -Dignore.endorsed.dirs= -classpath /srv/test/cpappt/tomcat/bin/bootstrap.jar:/srv/test/cpappt/tomcat/bin/tomcat-juli.jar
  -Dcatalina.base=/srv/test/cpappt/tomcat
  -Dcatalina.home=/srv/test/cpappt/tomcat
  -Djava.io.tmpdir=/srv/test/cpappt/tomcat/temp org.apache.catalina.startup.Bootstrap start

我想提取一个 tomcat 的heap dump 来分析潜在的内存泄漏。

我尝试使用jps获取tomcat服务器的pid,可惜我只看到jps本身的pid

> jps -l
952152 jdk.jcmd/openj9.tools.attach.diagnostics.tools.Jps

我还以启动 tomcat 服务器的同一用户身份运行 jps 命令

> whoami 
cpappt

我也试过运行jps命令,加上tomcat启动脚本中指定的java.io.tmpdir,结果和之前一样(注:tmpdir本身是空的)。

jps -l -J-Djava.io.tmpdir=/srv/test/cpappt/tomcat/temp

(补充说明:如果我启动了一个小的“Foo 程序”,jps 正在向我显示该程序的pid。)

然后我尝试使用 pid 中的 ps 命令创建一个堆转储,但失败了:

> jmap -dump:live,format=b,file=/tmp/dump.hprof 1682650
unrecognized option -dump:live,format=b,file=/tmp/dump.hprof
jmap: obtain heap information about a Java process
 Usage:
    jmap <option>* <vmid>
        <vmid>: Attach API VM ID as shown in jps or other Attach API-based tools
        <vmid>s are read from stdin if none are supplied as arguments
    -histo: print statistics about classes on the heap, including number of objects and aggregate size
    -histo:live : Print only live objects
    -J: supply arguments to the Java VM running jmap
NOTE: this utility might significantly affect the performance of the target VM.
At least one option must be selected.

如果我只是尝试查看直方图,这也会失败:

> jmap -histo:live 1682650
Error getting data from 1682650: Exception connecting to 1682650

jcmd 也无法创建转储:

> jcmd 1682650 GC.heap_dump /tmp/heapdump
Error getting data from 1682650: Exception connecting to 1682650
...

我怀疑这两个问题(jpsjmap)是相关的...... 因此,如果有人知道可能是什么问题,甚至知道如何创建 堆转储,那就太好了。

来自 tomcat 服务器(和 java)的其他信息:

Server version number: 9.0.39.0
OS Name:               Linux
OS Version:            5.7.0-0.bpo.2-amd64
Architektur:           amd64
Java Home:             /srv/jdk-11.0.8+10_openj9
JVM Version:           11.0.8+10
JVM Hersteller:        Eclipse OpenJ9

> java -version
openjdk version "11.0.8" 2020-07-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.8+10)
Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.21.0, JRE 11 Linux amd64-64-Bit Compressed References 20200715_697 (JIT enabled, AOT enabled)
OpenJ9   - 34cf4c075
OMR      - 113e54219
JCL      - 95bb504fbb based on jdk-11.0.8+10)

> which java
/srv/jdk11/bin/java

> which jps
/srv/jdk11/bin/java

> which jmap 
/srv/jdk11/bin/jmap

更新:2021-01-09

我终于可以找到知道 jmx 用户密码的同事,所以使用jconsole 我能够触发heap dump,但我只能调用triggerClassicHeapDump()。 我将文件复制到我的本地机器并想用Eclipse MemoryAnalyzer 分析它(我还添加了Diagnostic Tool Framework for Java (DTFJ))。

但是当我尝试打开文件时,我现在收到一条错误消息...

Error opening heap dump 'heapdump.20210119.100934.2621412.0001.txt'. Check the error log for further details.
Error opening heap dump 'heapdump.20210119.100934.2621412.0001.txt'. Check the error log for further details.
Error opening heap dump 'heapdump.20210119.100934.2621412.0001.txt'
Unable to read dump .../heapdump.20210119.100934.2621412.0001.phd metafile .../heapdump.20210119.100934.2621412.0001.txt in DTFJ format DTFJ-PHD (java.io.IOException)
Unable to read dump .../heapdump.20210119.100934.2621412.0001.phd metafile .../heapdump.20210119.100934.2621412.0001.txt in DTFJ format DTFJ-PHD
Error parsing Javacore (java.io.IOException)
Error parsing Javacore
com.ibm.dtfj.javacore.parser.framework.scanner.ScannerException: Maximum line length (32768) exceeded. Input file corrupt or not a javacore. (com.ibm.dtfj.javacore.parser.framework.parser.ParserException)
com.ibm.dtfj.javacore.parser.framework.scanner.ScannerException: Maximum line length (32768) exceeded. Input file corrupt or not a javacore.
Maximum line length (32768) exceeded. Input file corrupt or not a javacore. (com.ibm.dtfj.javacore.parser.framework.scanner.ScannerException)
Maximum line length (32768) exceeded. Input file corrupt or not a javacore.
Unable to read dump .../heapdump.20210119.100934.2621412.0001.txt metafile null in DTFJ format DTFJ-Javacore (java.io.IOException)
Unable to read dump .../heapdump.20210119.100934.2621412.0001.txt metafile null in DTFJ format DTFJ-Javacore
Error parsing Javacore (java.io.IOException)
Error parsing Javacore
com.ibm.dtfj.javacore.parser.framework.scanner.ScannerException: Maximum line length (32768) exceeded. Input file corrupt or not a javacore. (com.ibm.dtfj.javacore.parser.framework.parser.ParserException)
com.ibm.dtfj.javacore.parser.framework.scanner.ScannerException: Maximum line length (32768) exceeded. Input file corrupt or not a javacore.
Maximum line length (32768) exceeded. Input file corrupt or not a javacore. (com.ibm.dtfj.javacore.parser.framework.scanner.ScannerException)
Maximum line length (32768) exceeded. Input file corrupt or not a javacore.

【问题讨论】:

    标签: java heap-dump jmap eclipse-memory-analyzer openj9


    【解决方案1】:

    我已经在tomcat的启动脚本中添加了-Xdump:heap:events=user(见Xdump Option Builder)并重启了服务器。

    > ps -ef | grep Bootstrap
    cpappt   1919301       1 99 14:38 pts/0    00:01:00 /srv/jdk11/bin/java 
     -Xms32m -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError
     -Xdump:heap:events=user
     ...
    

    现在我可以使用kill -3 &lt;pid&gt; 创建一个堆转储(查看superuser: what-does-kill-3-mean)。 在catalina.base 目录下创建了两个文件:

    • javacore.&lt;XXX&gt;.txt(原始线程转储 (-3 := SIGQUIT))和
    • heapdump.&lt;YYY&gt;.phd(嗯,实际的堆转储)

    最后我可以用Eclipse MAT打开heapdump.&lt;YYY&gt;.phd文件

    补充说明:我们将openj9 版本更新为当前版本(23.0)。

    > java -version 
    openjdk version "11.0.9" 2020-10-20
    OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.9+11)
    Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.23.0, JRE 11 Linux amd64-64-Bit Compressed References 20201022_810 (JIT enabled, AOT enabled)
    

    现在jps 也确实找到了 tomcat 服务器

    > jps -l
    1919301 org.apache.catalina.startup.Bootstrap
    1921897 jdk.jcmd/openj9.tools.attach.diagnostics.tools.Jps
    

    jcmd 也有效:

    > jcmd 1919301 Dump.heap dump
    Dump written to .../tomcat/dump
    

    【讨论】:

      【解决方案2】:

      还可以考虑使用 -Xdump:system:events=user,因为生成的核心文件包含更多信息 - 但是 IBM DTFJ 读取 OpenJ9 生成的核心文件存在问题。

      Memory Analyzer 还可以生成堆转储 - 请参阅文件 > 获取堆转储,但“附加 API”和“使用 Helper JVM 附加 API”可能需要一些配置。 Memory Analyzer 1.12 在获取堆转储方面有一些改进,因此请尝试最新版本。

      【讨论】:

        猜你喜欢
        • 1970-01-01
        • 1970-01-01
        • 1970-01-01
        • 1970-01-01
        • 1970-01-01
        • 1970-01-01
        • 2022-08-08
        • 2011-08-28
        • 1970-01-01
        相关资源
        最近更新 更多