Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-20156

Jenkins (Java) crashes with Internal Error (instanceKlass.cpp:1534) [Error: ShouldNotReachHere()]

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Cannot Reproduce
    • Component/s: core
    • Labels:
    • Environment:
      Linux Ubuntu 12.04 64bit VM under ESX with latest OS updates including Java 7u45
    • Similar Issues:

      Description

      Since a couple of weeks we are seeing intermittent crashes of our Jenkins master due to some Java code which should not have been reached.

      The crash info looks like:

      A fatal error has been detected by the Java Runtime Environment:

      Internal Error (instanceKlass.cpp:1534), pid=2378, tid=239618880
      Error: ShouldNotReachHere()

      JRE version: Java(TM) SE Runtime Environment (7.0_45-b18) (build 1.7.0_45-b18)
      Java VM: Java HotSpot(TM) Client VM (24.45-b08 mixed mode linux-x86 )
      Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before retarting Java again

      --------------- T H R E A D ---------------

      Current thread (0xb6a86c00): JavaThread "C1 CompilerThread0" daemon [threadin_vm, id=2386, stack(0x0e404000,0x0e485000)]

      Stack: [0x0e404000,0x0e485000], sp=0x0e483c70, free space=511k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      V [libjvm.so+0x503a66] VMError::report_and_die()+0x1a6
      V [libjvm.so+0x1db48a] report_should_not_reach_here(char const*, int)+0x4a
      V [libjvm.so+0x2743be] instanceKlass::remove_dependent_nmethod(nmethod*)+0x3e

      For a complete crash stack please see the attached log file.

      It sounds like it has been caused by low memory. While top might indeed have shown that the whole memory was filled up on that machine, the java management console hasn't shown it.

      I have already submitted crash report to Oracle a while ago but haven't gotten any reply yet. I think it's worth also filing a bug here in case someone knows about this problem or is also affected by this.

        Attachments

          Activity

          Hide
          kohsuke Kohsuke Kawaguchi added a comment -

          Unfortunately, this looks like a problem coming from JavaVM, and so I'd consider this a bug in JavaVM. While there might be something Jenkins can do to avoid hitting JavaVM hard, it's difficult to tell what we can do from the crash dump.

          I'm forced to close this as "Cannot Reproduce". Hopefully Oracle would be able to look into this?

          Show
          kohsuke Kohsuke Kawaguchi added a comment - Unfortunately, this looks like a problem coming from JavaVM, and so I'd consider this a bug in JavaVM. While there might be something Jenkins can do to avoid hitting JavaVM hard, it's difficult to tell what we can do from the crash dump. I'm forced to close this as "Cannot Reproduce". Hopefully Oracle would be able to look into this?
          Hide
          dalvizu Dan Alvizu added a comment -

          This looks to be caused by JRuby:

          https://github.com/jruby/jruby/issues/1024

          One fix is to add `-XX:-MethodFlushing` flag - I believe the original author also solved this by tuning memory, maybe increasing permgen means that there is less need to reclaim it?

          Show
          dalvizu Dan Alvizu added a comment - This looks to be caused by JRuby: https://github.com/jruby/jruby/issues/1024 One fix is to add `-XX:-MethodFlushing` flag - I believe the original author also solved this by tuning memory, maybe increasing permgen means that there is less need to reclaim it?
          Hide
          dalvizu Dan Alvizu added a comment -

          I take my previous comment back - the only thing that worked for me was updating to the latest JDK8 - it appears to be a bug in JDK7

          Show
          dalvizu Dan Alvizu added a comment - I take my previous comment back - the only thing that worked for me was updating to the latest JDK8 - it appears to be a bug in JDK7

            People

            Assignee:
            kohsuke Kohsuke Kawaguchi
            Reporter:
            whimboo Henrik Skupin
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: