Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46514

ClassLoader leak in remoting with job-dsl plugin

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Cannot Reproduce
    • Component/s: job-dsl-plugin, remoting
    • Labels:
      None
    • Environment:
      Jenkins 2.67
      job-dsl 1.64
    • Similar Issues:

      Description

      We have the same issue as JENKINS-30832 and we suspected a classloader leak, that is why we did some heap dumps. What we found is that after running the seed job (which processes N groovy scripts) M times, there were N*M GroovyClassLoaders retained. YourKit Java Profiler showed us that each of them loaded the classes (mostly closures) from one of our N groovy scripts.

      We also traced what is retaining the classloaders and we ended up at the unexportLog field of hudson.remoting.ExportTable. We didn't fully understand why was it retaining the classloaders, see the screenshot attached (the majority of LinkedList nodes was cut out). ul_rsp is one of the groovy scripts processed, the adress 172.17.0.1 corresponds to the node the seed job was running on.

      As a workaround hudson.remoting.ExportTable.unexportLogSize was set to 0 to prevent adding entries to unexportLog. Checking the heap dump revealed that the same amount of GroovyClassLoaders are present but held via weak/soft references only, making them eligible for garbage collection.

      I should also mention that the classloaders were leaked even after we got rid of the mixins in the seed job.

        Attachments

          Activity

          apetres Petres Andras created issue -
          daspilker Daniel Spilker made changes -
          Field Original Value New Value
          Assignee Daniel Spilker [ daspilker ] Oleg Nenashev [ oleg_nenashev ]
          apetres Petres Andras made changes -
          Description We have the same issue as JENKINS-30832 and we suspected a classloader leak, that is why we did some heap dumps. What we found is that after running the seed job (which processes N groovy scripts) M times, there were N*M GroovyClassLoaders retained. YourKit Java Profiler showed us that each of them loaded the classes (mostly closures) from one of our N groovy scripts.

          We also traced what is retaining the classloaders and we ended up at the _unexportLog_ field of _hudson.remoting.ExportTable_. We didn't fully understand why was it retaining the classloaders, see the screenshot attached (the majority of _LinkedList_ nodes was cut out). _ul_rsp_ is one of the groovy scripts processed, the adress _172.17.0.1_ corresponds to the node the seed job was running on.

          As a workaround hudson.remoting._ExportTable.unexportLogSize_ was set to 0 to prevent adding entries to _unexportLog_. Checking the heap dump revealed that the same amount of GroovyClassLoaders are present but held via weak/soft references only, making them eligible for garbage collection.

          I should also mention that the classloaders were leaked even after we got rid of the mixins in out seed job.
          We have the same issue as JENKINS-30832 and we suspected a classloader leak, that is why we did some heap dumps. What we found is that after running the seed job (which processes N groovy scripts) M times, there were N*M GroovyClassLoaders retained. YourKit Java Profiler showed us that each of them loaded the classes (mostly closures) from one of our N groovy scripts.

          We also traced what is retaining the classloaders and we ended up at the _unexportLog_ field of _hudson.remoting.ExportTable_. We didn't fully understand why was it retaining the classloaders, see the screenshot attached (the majority of _LinkedList_ nodes was cut out). _ul_rsp_ is one of the groovy scripts processed, the adress _172.17.0.1_ corresponds to the node the seed job was running on.

          As a workaround hudson.remoting._ExportTable.unexportLogSize_ was set to 0 to prevent adding entries to _unexportLog_. Checking the heap dump revealed that the same amount of GroovyClassLoaders are present but held via weak/soft references only, making them eligible for garbage collection.

          I should also mention that the classloaders were leaked even after we got rid of the mixins in the seed job.
          apetres Petres Andras made changes -
          Attachment ExportTable.dump.txt [ 39574 ]
          daspilker Daniel Spilker made changes -
          Assignee Oleg Nenashev [ oleg_nenashev ] Daniel Spilker [ daspilker ]
          Resolution Cannot Reproduce [ 5 ]
          Status Open [ 1 ] Resolved [ 5 ]
          daspilker Daniel Spilker made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            Assignee:
            daspilker Daniel Spilker
            Reporter:
            apetres Petres Andras
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: