Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21422

Jenkins crashing due to out of memory when rebuilding jobs

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • build-pipeline-plugin, core
    • None
    • linux

      After an upgrade to version 1.546 Jenkins became very unstable crashing 2-4 times per day. From what I can tell it is because of missing junitResult.xml files when jenkins tries to load the test trend.

      Attached is a snippet from the log where you can see the FileNotFoundException for the junitResult.xml file followed by a StackOverFlow error and then an out of memory error.

      Also attached is a screen shot from a heap analysis showing that the request handler thread building the test trend is holding a considerable amount of the heap. Please note that this particular heap analysis is from a different crash than the log snippet. I did not analyze the heap on the crash shown in the log, but I am sure it would show the test trend for the same job that throw the StackOverFlow.

          [JENKINS-21422] Jenkins crashing due to out of memory when rebuilding jobs

          John Carlile created issue -
          John Carlile made changes -
          Description Original: After an upgrade to version 1.546 Jenkins became very unstable crashing 2-4 times per day. From what I can tell it is because of missing junitResult.xml files when jenkins tries to load the test trend.

          Attached is a snippet from the log where you can see the FileNotFoundException for the junitResult.xml file followed by a StackOverFlow error and then an out of memory error.

          Also attached is a screen shot from the heap analysis showing that the request handler thread building the test trend is holding a considerable amount of the heap.
          New: After an upgrade to version 1.546 Jenkins became very unstable crashing 2-4 times per day. From what I can tell it is because of missing junitResult.xml files when jenkins tries to load the test trend.

          Attached is a snippet from the log where you can see the FileNotFoundException for the junitResult.xml file followed by a StackOverFlow error and then an out of memory error.

          Also attached is a screen shot from a heap analysis showing that the request handler thread building the test trend is holding a considerable amount of the heap. Please note that this particular heap analysis is from a different crash than the log snippet. I did not analyze the heap on the crash shown in the log, but I am sure it would show the test trend for the same job that throw the StackOverFlow.

          John Carlile added a comment - - edited

          Attached is a different screen shot from the heap dump analysis that provides a little more context. The heap dump is 7GB making it difficult to share.

          John Carlile added a comment - - edited Attached is a different screen shot from the heap dump analysis that provides a little more context. The heap dump is 7GB making it difficult to share.
          John Carlile made changes -
          Attachment New: heap-dump-capture.PNG [ 25026 ]

          John Carlile added a comment - - edited

          I have been trying to get to the bottom of this, but not having much luck yet. The one thing that is consistent with all occurrences of this crash is there is a series of FileNotFoundExceptions logged for one of the junitResult.xml files in a build (we only keep about a week's worth of builds). Immediately following the FileNotFound is the StackOverflow, which ultimately leads to OOM. To make matters more confusing I do see the junitResult.xml on disk that is shown in the FileNotFoundException and once Jenkins is restarted the job seems fine.

          I spent a little time spelunking in the jenkins code. Also wrote myself a little junit test to see if I could reproduce. No luck there yet. Any ideas on how to narrow down the root cause would be helpful.

          John Carlile added a comment - - edited I have been trying to get to the bottom of this, but not having much luck yet. The one thing that is consistent with all occurrences of this crash is there is a series of FileNotFoundExceptions logged for one of the junitResult.xml files in a build (we only keep about a week's worth of builds). Immediately following the FileNotFound is the StackOverflow, which ultimately leads to OOM. To make matters more confusing I do see the junitResult.xml on disk that is shown in the FileNotFoundException and once Jenkins is restarted the job seems fine. I spent a little time spelunking in the jenkins code. Also wrote myself a little junit test to see if I could reproduce. No luck there yet. Any ideas on how to narrow down the root cause would be helpful.
          John Carlile made changes -
          Summary Original: Jenkins crashing with StackOverFlow error New: Jenkins crashing due to out of memory

          John Carlile added a comment - - edited

          I just updated the description of this. After some more investigation it seems like the StackOverFlow may be a byproduct of jenkins running out of heap.

          I was able to produce this. Looks like it is related to rebuilding a job from the build-pipeline view. Here are the steps to reproduce:

          1) run a job
          2) from build-pipeline-plugin view hit rebuild button

          Result: OOM

          Note this did not happen on 1.536. Must be some incompatibility in the newer version.

          We have a 1.3.x variety of the build pipeline plugin

          John Carlile added a comment - - edited I just updated the description of this. After some more investigation it seems like the StackOverFlow may be a byproduct of jenkins running out of heap. I was able to produce this. Looks like it is related to rebuilding a job from the build-pipeline view. Here are the steps to reproduce: 1) run a job 2) from build-pipeline-plugin view hit rebuild button Result: OOM Note this did not happen on 1.536. Must be some incompatibility in the newer version. We have a 1.3.x variety of the build pipeline plugin
          John Carlile made changes -
          Component/s New: build-pipeline [ 15962 ]
          Affects Version/s New: current [ 10162 ]
          John Carlile made changes -
          Summary Original: Jenkins crashing due to out of memory New: Jenkins crashing due to out of memory when rebuilding jobs

          Ladislav Toldy added a comment - - edited

          Same bug on 1.532.2
          Maven Project Plugin 2.0.3
          Build Pipeline Plugin 1.3.3

          Happens while viewing maven jobs(rendering test trend graph) via build-pipeline view.

          Result: OOM and restart is needed

          Ladislav Toldy added a comment - - edited Same bug on 1.532.2 Maven Project Plugin 2.0.3 Build Pipeline Plugin 1.3.3 Happens while viewing maven jobs(rendering test trend graph) via build-pipeline view. Result: OOM and restart is needed

            dalvizu Dan Alvizu
            jocarli John Carlile
            Votes:
            2 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: