java - How do I investigate the cause of a JVM crash?

Question

One day ago, after a few months of normal working, our java app starts to crash occasionally with the following error:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (safepoint.cpp:247), pid=2075, tid=140042095163136
#  guarantee(PageArmed == 0) failed: invariant
#
# JRE version: 6.0_23-b05
# Java VM: Java HotSpot(TM) 64-Bit Server VM (19.0-b09 mixed mode linux-amd64 compressed oops)
# An error report file with more information is saved as:
# /var/chat/jSocketer/build/hs_err_pid2075.log
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

I looked in hs_err_pid2075.log and saw that there was an active thread, that processed a network communication. However there wasn't any application or environment changes done in the last few months. Also there wasn't any load growth. What can I do to understand, what is the reason of crash? Are there any common steps to investigate a jvm crash?

UPD http://www.wuala.com/ubear/public

Answer 1

The crash is in the JVM, not in external native code. However, the operation it crashed on has been initiated by and external DLL.

This line in the hs_err_pid file explains the operation that crashed:

VM_Operation (0x00007f5e16e35450): GetAllStackTraces, mode: safepoint, requested by thread 0x0000000040796000

Now, thread 0x0000000040796000 is

0x0000000040796000 JavaThread "YJPAgent-Telemetry" daemon [_thread_blocked, id=2115, stack(0x00007f5e16d36000,0x00007f5e16e37000)]

which is a thread created by Yourkit. "GetAllStackTraces" is something that a profiler needs to call in order to do sampling. If you remove the profiler, the crash will not happen.

With this information It's not possible to say what causes the crash, but you can try the following: Remove all -XX VM parameters, -verbose:gc and the debugging VM parameters. They might interfere with the profiling interface of the JVM.

Update

Code that calls java.lang.Thread#getAllStackTraces() or java.lang.Thread#getStackTrace() may trigger the same crash