Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61251

Out of memory issues during builds

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Cannot Reproduce
    • Component/s: core
    • Labels:
    • Environment:
    • Similar Issues:

      Description

      Hi,

      we've started to get an Out of memory issues on our Jenkins instance last week. There was no notable change in the code (1 month ago, we've installed warnings-ng for pylint reports and Cobertura), but the problems started last week.

       

      Right now, we are seeing odd behavior with our builds and such that something is probably leaking memory and then GC utilizes a CPU for 100%. This is like one build looks like:
       
      Start: !1.png|thumbnail|
      During: !2.png|thumbnail|
      During: !3.png|thumbnail|
      After some time: !4.png|thumbnail|

      My guess is that the memory should be released, but something is holding it. I was trying to go through a your help page and tried to analyze a Heap Dump. The notable is Java.Util.HashMap class has almost all allocated bytes.

      I'm providing a 2 heap dumps, one during a build link (~2.93GB) and one after the Jenkins instance is unresponsible due to the Out Of Memory link (~3.57GB).

      They suggest that the problem is org.codehaus.groovy.ast.ClassNode, but to say the truth, I'm not a Java developer so I'm not getting what can be an issue.

      Also, I'm providing a /var/log/jenkins/jenkins.log which contains lots of warnings, which may help: link (~30MB)

      I've already spent a great of time debugging what is happening, but I can't get my head around.

      Could you take a look or point where the problem should be?

      Thanks
      Andrej

       

        Attachments

        1. 1.png
          1.png
          244 kB
        2. 2.png
          2.png
          239 kB
        3. 3.png
          3.png
          243 kB
        4. 4.png
          4.png
          232 kB
        5. fileDescriptors.jpg
          fileDescriptors.jpg
          169 kB
        6. threadCount.jpg
          threadCount.jpg
          161 kB

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            astaruch Andrej Staruch
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: