Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43619

Memory leak in hudson.ClassicPluginStrategy$AntClassLoader2 and org.eclipse.jetty.webapp.WebAppClassLoader

    • Icon: Bug Bug
    • Resolution: Won't Fix
    • Icon: Critical Critical
    • core

      We have tests constantly running on 6 jenkins slaves, every week or two jenkins becomes unstable because of memory issues.

       

      I have enabled GC logging, and using to gceasy.io I can clearly see that memory usage is increasing over time (included graph).

       

      Also took some jmap heap dumps which shows increasing number of "org.jruby.RubyObject" and "hudson.model.FreeStyleBuild" instances. (screenshots attached, one of them is taken after 3 days of start and another one after 7 days)

      Also included screenshots of common path to accumulation point (maybe it helps). Not sure we can upload jmap dumps (I assume they can contain sensitive information, e.g passwords, ssh private keys).

       

      Our jobs are configured to keep only 100 last builds.

       

       

          [JENKINS-43619] Memory leak in hudson.ClassicPluginStrategy$AntClassLoader2 and org.eclipse.jetty.webapp.WebAppClassLoader

          Oleg Nenashev added a comment -

          FreeStyle builds may be fine, it needs full heap dump check.
          Regarding jruby, I would suspect the Ruby plugin, which does not cleanup the contributed buildEnvironments after the build completion. But this object should be deallocated along with the build.

          Would it be possible to get more information about references to FreestyleBuild objects? The provided ones are related to the Pending builds (ref from the executor is active), hence they should be retained in the memory. It is more interesting to understand if there are any runaway references to deleted builds.

          Also, how many jobs do you have on the instance?

          Oleg Nenashev added a comment - FreeStyle builds may be fine, it needs full heap dump check. Regarding jruby, I would suspect the Ruby plugin, which does not cleanup the contributed buildEnvironments after the build completion. But this object should be deallocated along with the build. Would it be possible to get more information about references to FreestyleBuild objects? The provided ones are related to the Pending builds (ref from the executor is active), hence they should be retained in the memory. It is more interesting to understand if there are any runaway references to deleted builds. Also, how many jobs do you have on the instance?

          We have ~50 jobs in total and up to 15 (most of the time it is ~10) are running in parallel constantly (once job finishes it is started again).

           

          Regarding providing more information about references and etc, I am happy to provide more information, but I'm not a Java developer and my concern is that I'm posting wrong or not detailed enough screenshots, to avoid wasting each other's time maybe it we should have a screen sharing session.

           

          If you prefer not to have screen sharing session then I have an additional question:

          If I'm clicking FreeStyleBuild object in Eclipse Memory Analyzer and select "List objects of incoming references" then get new page, but I'm don't know which object should I expand on that page?

          Tarvi Pillessaar added a comment - We have ~50 jobs in total and up to 15 (most of the time it is ~10) are running in parallel constantly (once job finishes it is started again).   Regarding providing more information about references and etc, I am happy to provide more information, but I'm not a Java developer and my concern is that I'm posting wrong or not detailed enough screenshots, to avoid wasting each other's time maybe it we should have a screen sharing session.   If you prefer not to have screen sharing session then I have an additional question: If I'm clicking FreeStyleBuild object in Eclipse Memory Analyzer and select "List objects of incoming references" then get new page, but I'm don't know which object should I expand on that page?

          Oleg Nenashev added a comment -

          For such scale you definitely have too many objects.

          > If you prefer not to have screen sharing session then I have an additional question:

          Sorry, I just have no time for that in next few weeks. Maybe somebody else would be interested to do that though it is not a common procedure in open-source projects.

          I would start from gathering more information about your instance, e.g. installed plugins and versions. You can grab this and other info using this plugin: https://wiki.jenkins-ci.org/display/JENKINS/Support+Core+Plugin

          Oleg Nenashev added a comment - For such scale you definitely have too many objects. > If you prefer not to have screen sharing session then I have an additional question: Sorry, I just have no time for that in next few weeks. Maybe somebody else would be interested to do that though it is not a common procedure in open-source projects. I would start from gathering more information about your instance, e.g. installed plugins and versions. You can grab this and other info using this plugin: https://wiki.jenkins-ci.org/display/JENKINS/Support+Core+Plugin

          Closing this issue, it is not relevant for me any longer.

          Tarvi Pillessaar added a comment - Closing this issue, it is not relevant for me any longer.

            Unassigned Unassigned
            tarvi Tarvi Pillessaar
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: