-
Bug
-
Resolution: Cannot Reproduce
-
Minor
-
None
-
Jenkins 2.67
job-dsl 1.64
We have the same issue as JENKINS-30832 and we suspected a classloader leak, that is why we did some heap dumps. What we found is that after running the seed job (which processes N groovy scripts) M times, there were N*M GroovyClassLoaders retained. YourKit Java Profiler showed us that each of them loaded the classes (mostly closures) from one of our N groovy scripts.
We also traced what is retaining the classloaders and we ended up at the unexportLog field of hudson.remoting.ExportTable. We didn't fully understand why was it retaining the classloaders, see the screenshot attached (the majority of LinkedList nodes was cut out). ul_rsp is one of the groovy scripts processed, the adress 172.17.0.1 corresponds to the node the seed job was running on.
As a workaround hudson.remoting.ExportTable.unexportLogSize was set to 0 to prevent adding entries to unexportLog. Checking the heap dump revealed that the same amount of GroovyClassLoaders are present but held via weak/soft references only, making them eligible for garbage collection.
I should also mention that the classloaders were leaked even after we got rid of the mixins in the seed job.
[JENKINS-46514] ClassLoader leak in remoting with job-dsl plugin
Daniel Spilker
added a comment - oleg_nenashev Job DSL does something similar for JARs, ScriptRequestGenerator
But it can also access any file from the workspace. That's another use of remoting: JenkinsJobManagement.
But IMHO that should not lead to the GroocyClassLoader being send over remoting. The classes in question should be loaded by the plugin class loader or above. Can we debug that somehow? E.g. can we see which objects are transferred?
Daniel Spilker
added a comment - oleg_nenashev Job DSL does something similar for JARs, ScriptRequestGenerator
But it can also access any file from the workspace. That's another use of remoting: JenkinsJobManagement .
But IMHO that should not lead to the GroocyClassLoader being send over remoting. The classes in question should be loaded by the plugin class loader or above. Can we debug that somehow? E.g. can we see which objects are transferred? 
Petres Andras
added a comment - oleg_nenashev to me it seems that ExportTable entries may help, since it stores data about objects sent over remote, right?
Petres Andras
added a comment - oleg_nenashev to me it seems that ExportTable entries may help, since it stores data about objects sent over remote, right? > Can we debug that somehow? E.g. can we see which objects are transferred?
> to me it seems that ExportTable entries may help, since it stores data about objects sent over remote, right?
It is. You can dump Export Table at any moment, but some extra export table logging (and even "exportTable.log" would be useful).
Petres Andras
added a comment - I did the dump using ExportTable.dump() and attached it.
What other kind of "extra export table logging" would you need?
Petres Andras
added a comment - I did the dump using ExportTable.dump() and attached it.
What other kind of "extra export table logging" would you need? Daniel Spilker
added a comment - Hm, the dump does not contain any Groovy or Job DSL classes. I'm not sure how to debug this.
Daniel Spilker
added a comment - Hm, the dump does not contain any Groovy or Job DSL classes. I'm not sure how to debug this. If this is a real case, I am pretty sure it will break with JEP-200 on Jenkins 2.102+
apetres have you already upgraded just in case?
Petres Andras
added a comment - I upgraded to 2.106 and did some dumps, no ExportTable or remoting-related stuff is retaining the classloaders anymore.
Petres Andras
added a comment - I upgraded to 2.106 and did some dumps, no ExportTable or remoting-related stuff is retaining the classloaders anymore. Daniel Spilker
added a comment - apetres Can we close the issue? Petres Andras
added a comment - Sure! 
It could be solved via JOB_NAME@tmp folder on Master like Jenkins Pipeline does for Shared Libraries and load() operations, but I am not exactly sure it is a good practice