-
Bug
-
Resolution: Cannot Reproduce
-
Minor
-
None
-
Jenkins 2.67
job-dsl 1.64
We have the same issue as JENKINS-30832 and we suspected a classloader leak, that is why we did some heap dumps. What we found is that after running the seed job (which processes N groovy scripts) M times, there were N*M GroovyClassLoaders retained. YourKit Java Profiler showed us that each of them loaded the classes (mostly closures) from one of our N groovy scripts.
We also traced what is retaining the classloaders and we ended up at the unexportLog field of hudson.remoting.ExportTable. We didn't fully understand why was it retaining the classloaders, see the screenshot attached (the majority of LinkedList nodes was cut out). ul_rsp is one of the groovy scripts processed, the adress 172.17.0.1 corresponds to the node the seed job was running on.
As a workaround hudson.remoting.ExportTable.unexportLogSize was set to 0 to prevent adding entries to unexportLog. Checking the heap dump revealed that the same amount of GroovyClassLoaders are present but held via weak/soft references only, making them eligible for garbage collection.
I should also mention that the classloaders were leaked even after we got rid of the mixins in the seed job.
[JENKINS-46514] ClassLoader leak in remoting with job-dsl plugin
Petres Andras
created issue -
Daniel Spilker
made changes -
| Assignee | Original: Daniel Spilker [ daspilker ] | New: Oleg Nenashev [ oleg_nenashev ] |
Petres Andras
made changes -
| Description |
Original:
We have the same issue as JENKINS-30832 and we suspected a classloader leak, that is why we did some heap dumps. What we found is that after running the seed job (which processes N groovy scripts) M times, there were N*M GroovyClassLoaders retained. YourKit Java Profiler showed us that each of them loaded the classes (mostly closures) from one of our N groovy scripts. We also traced what is retaining the classloaders and we ended up at the _unexportLog_ field of _hudson.remoting.ExportTable_. We didn't fully understand why was it retaining the classloaders, see the screenshot attached (the majority of _LinkedList_ nodes was cut out). _ul_rsp_ is one of the groovy scripts processed, the adress _172.17.0.1_ corresponds to the node the seed job was running on. As a workaround hudson.remoting._ExportTable.unexportLogSize_ was set to 0 to prevent adding entries to _unexportLog_. Checking the heap dump revealed that the same amount of GroovyClassLoaders are present but held via weak/soft references only, making them eligible for garbage collection. I should also mention that the classloaders were leaked even after we got rid of the mixins in out seed job. |
New:
We have the same issue as JENKINS-30832 and we suspected a classloader leak, that is why we did some heap dumps. What we found is that after running the seed job (which processes N groovy scripts) M times, there were N*M GroovyClassLoaders retained. YourKit Java Profiler showed us that each of them loaded the classes (mostly closures) from one of our N groovy scripts. We also traced what is retaining the classloaders and we ended up at the _unexportLog_ field of _hudson.remoting.ExportTable_. We didn't fully understand why was it retaining the classloaders, see the screenshot attached (the majority of _LinkedList_ nodes was cut out). _ul_rsp_ is one of the groovy scripts processed, the adress _172.17.0.1_ corresponds to the node the seed job was running on. As a workaround hudson.remoting._ExportTable.unexportLogSize_ was set to 0 to prevent adding entries to _unexportLog_. Checking the heap dump revealed that the same amount of GroovyClassLoaders are present but held via weak/soft references only, making them eligible for garbage collection. I should also mention that the classloaders were leaked even after we got rid of the mixins in the seed job. |
Daniel Spilker
added a comment - oleg_nenashev The Job DSL engine uses remoting to access the DSL script file in the workspace through FilePath. The class in question is ScriptRequestGenerator. Unfortunately the class is overly complex because the Groovy class loader is an URLClassLoader and I needed to create a special URL handler for files in the workspace...
Daniel Spilker
added a comment - oleg_nenashev The Job DSL engine uses remoting to access the DSL script file in the workspace through FilePath . The class in question is ScriptRequestGenerator . Unfortunately the class is overly complex because the Groovy class loader is an URLClassLoader and I needed to create a special URL handler for files in the workspace... It could be solved via JOB_NAME@tmp folder on Master like Jenkins Pipeline does for Shared Libraries and load() operations, but I am not exactly sure it is a good practice
Daniel Spilker
added a comment - oleg_nenashev Job DSL does something similar for JARs, ScriptRequestGenerator
But it can also access any file from the workspace. That's another use of remoting: JenkinsJobManagement.
But IMHO that should not lead to the GroocyClassLoader being send over remoting. The classes in question should be loaded by the plugin class loader or above. Can we debug that somehow? E.g. can we see which objects are transferred?
Daniel Spilker
added a comment - oleg_nenashev Job DSL does something similar for JARs, ScriptRequestGenerator
But it can also access any file from the workspace. That's another use of remoting: JenkinsJobManagement .
But IMHO that should not lead to the GroocyClassLoader being send over remoting. The classes in question should be loaded by the plugin class loader or above. Can we debug that somehow? E.g. can we see which objects are transferred? Petres Andras
added a comment - oleg_nenashev to me it seems that ExportTable entries may help, since it stores data about objects sent over remote, right?
Petres Andras
added a comment - oleg_nenashev to me it seems that ExportTable entries may help, since it stores data about objects sent over remote, right? > Can we debug that somehow? E.g. can we see which objects are transferred?
> to me it seems that ExportTable entries may help, since it stores data about objects sent over remote, right?
It is. You can dump Export Table at any moment, but some extra export table logging (and even "exportTable.log" would be useful).
Petres Andras
made changes -
| Attachment | New: ExportTable.dump.txt [ 39574 ] |
It is not exactly clear to me why the object with the custom classloader is being sent over the channel. Likely you do remote calls within your Groovy script. I do not see the scripts attached, so I cannot say for sure. While Remoting behaves correctly by creating "createdAt" object with the Groovy classloader in such case, it is something not really desired in this case.
So far I also wonder why it comes from the Channel Pinger thread. So far no giid explanation for that. I need your Java settings in order to check if there is a custom classloader defined there.
Unexport Log Size is a valid workaround.
IMHO you firstly need to investigate your scripts and check if they invoke remote calls directly. If no, it is something to be investigated in the JobDSL engine or its plugin implementations.