Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61251

Out of memory issues during builds

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Cannot Reproduce
    • Component/s: core
    • Labels:
    • Environment:
    • Similar Issues:

      Description

      Hi,

      we've started to get an Out of memory issues on our Jenkins instance last week. There was no notable change in the code (1 month ago, we've installed warnings-ng for pylint reports and Cobertura), but the problems started last week.

       

      Right now, we are seeing odd behavior with our builds and such that something is probably leaking memory and then GC utilizes a CPU for 100%. This is like one build looks like:
       
      Start: !1.png|thumbnail|
      During: !2.png|thumbnail|
      During: !3.png|thumbnail|
      After some time: !4.png|thumbnail|

      My guess is that the memory should be released, but something is holding it. I was trying to go through a your help page and tried to analyze a Heap Dump. The notable is Java.Util.HashMap class has almost all allocated bytes.

      I'm providing a 2 heap dumps, one during a build link (~2.93GB) and one after the Jenkins instance is unresponsible due to the Out Of Memory link (~3.57GB).

      They suggest that the problem is org.codehaus.groovy.ast.ClassNode, but to say the truth, I'm not a Java developer so I'm not getting what can be an issue.

      Also, I'm providing a /var/log/jenkins/jenkins.log which contains lots of warnings, which may help: link (~30MB)

      I've already spent a great of time debugging what is happening, but I can't get my head around.

      Could you take a look or point where the problem should be?

      Thanks
      Andrej

       

        Attachments

        1. 1.png
          1.png
          244 kB
        2. 2.png
          2.png
          239 kB
        3. 3.png
          3.png
          243 kB
        4. 4.png
          4.png
          232 kB
        5. fileDescriptors.jpg
          fileDescriptors.jpg
          169 kB
        6. threadCount.jpg
          threadCount.jpg
          161 kB

          Activity

          astaruch Andrej Staruch created issue -
          astaruch Andrej Staruch made changes -
          Field Original Value New Value
          Attachment fileDescriptors.jpg [ 50508 ]
          astaruch Andrej Staruch made changes -
          Attachment threadCount.jpg [ 50509 ]
          Hide
          astaruch Andrej Staruch added a comment -

          The problem is in Java. Our server's Java vas upgraded from jdk-8 to jdk-11. After downgrading it back to jdk-8, thing went back to normal.

          Java 11 has some issues regarding open files I guess. We switched back to jdk-8 around 13:00 and this was the only change. We were then able to run our builds without any hassle. We can on our monitory graphs that probably Jenkins, or some of our other plugin, has a problem with file descriptors?

          Show
          astaruch Andrej Staruch added a comment - The problem is in Java. Our server's Java vas upgraded from jdk-8 to jdk-11. After downgrading it back to jdk-8, thing went back to normal. Java 11 has some issues regarding open files I guess. We switched back to jdk-8 around 13:00 and this was the only change. We were then able to run our builds without any hassle. We can on our monitory graphs that probably Jenkins, or some of our other plugin, has a problem with file descriptors?
          Hide
          danielbeck Daniel Beck added a comment -

          open files

          If you use the "Resource Root URL" feature, that has a known problem with unclosed open files in 2.204.x: JENKINS-61121. That will be addressed in LTS next month in 2.222.1.

          Show
          danielbeck Daniel Beck added a comment - open files If you use the "Resource Root URL" feature, that has a known problem with unclosed open files in 2.204.x: JENKINS-61121 . That will be addressed in LTS next month in 2.222.1.
          Hide
          danielbeck Daniel Beck added a comment -

          Closing for now, first comment indicates it's an obsolete problem.

          Show
          danielbeck Daniel Beck added a comment - Closing for now, first comment indicates it's an obsolete problem.
          danielbeck Daniel Beck made changes -
          Resolution Cannot Reproduce [ 5 ]
          Status Open [ 1 ] Closed [ 6 ]
          Hide
          astaruch Andrej Staruch added a comment -

          Just FYI: we don't use "Resource Root URL" feature. So there is some other problem with java11, but I was not able to narrow it down which plugin was doing it.

          Show
          astaruch Andrej Staruch added a comment - Just FYI: we don't use "Resource Root URL" feature. So there is some other problem with java11, but I was not able to narrow it down which plugin was doing it.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            astaruch Andrej Staruch
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: