Critical Since a couple of weeks we are seeing intermittent crashes of our Jenkins master due to some Java code which should not have been reached.
The crash info looks like:
A fatal error has been detected by the Java Runtime Environment:
Internal Error (instanceKlass.cpp:1534), pid=2378, tid=239618880
Error: ShouldNotReachHere()
JRE version: Java(TM) SE Runtime Environment (7.0_45-b18) (build 1.7.0_45-b18)
Java VM: Java HotSpot(TM) Client VM (24.45-b08 mixed mode linux-x86 )
Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before retarting Java again
--------------- T H R E A D ---------------
Current thread (0xb6a86c00): JavaThread "C1 CompilerThread0" daemon [threadin_vm, id=2386, stack(0x0e404000,0x0e485000)]
Stack: [0x0e404000,0x0e485000], sp=0x0e483c70, free space=511k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x503a66] VMError::report_and_die()+0x1a6
V [libjvm.so+0x1db48a] report_should_not_reach_here(char const*, int)+0x4a
V [libjvm.so+0x2743be] instanceKlass::remove_dependent_nmethod(nmethod*)+0x3e
For a complete crash stack please see the attached log file.
It sounds like it has been caused by low memory. While top might indeed have shown that the whole memory was filled up on that machine, the java management console hasn't shown it.
I have already submitted crash report to Oracle a while ago but haven't gotten any reply yet. I think it's worth also filing a bug here in case someone knows about this problem or is also affected by this.
[JENKINS-20156] Jenkins (Java) crashes with Internal Error (instanceKlass.cpp:1534) [Error: ShouldNotReachHere()]
This looks to be caused by JRuby:
https://github.com/jruby/jruby/issues/1024
One fix is to add `-XX:-MethodFlushing` flag - I believe the original author also solved this by tuning memory, maybe increasing permgen means that there is less need to reclaim it?
I take my previous comment back - the only thing that worked for me was updating to the latest JDK8 - it appears to be a bug in JDK7
Unfortunately, this looks like a problem coming from JavaVM, and so I'd consider this a bug in JavaVM. While there might be something Jenkins can do to avoid hitting JavaVM hard, it's difficult to tell what we can do from the crash dump.
I'm forced to close this as "Cannot Reproduce". Hopefully Oracle would be able to look into this?