Status: Resolved (View Workflow)
Jenkins LTS 2.277.2
gerrit-trigger version 2.34.0 (latest)
Since upgrading a large number (100s) of Jenkins instances to LTS v2.277.2, and the associated plugin updates required for the UI changes, I have observed a number of these instances are leak memory, and in one case it only lasts about 36 hours before exhausting the heap (in that case it is 6GB).
a heapdump (e.g. jmap -dump:live,file=jenkins-20210603-0939.hprof) showed that there were a large number (1000s) of hudson.remoting.Channel classes retained in a list by the class org.jvnet.hudson.test.ChannelShutdownListener. Doing a grep in the installed plugins folder, the only plugin that referenced this class was the gerrit-trigger plugin:
plugins$ grep -iRH "org.jvnet.hudson.test.ChannelShutdownListener" *
Binary file gerrit-trigger/WEB-INF/lib/jenkins-test-harness-2.56.jar matches
I found it curious that a test class was being included in a production plugin, so I do not know why this jenkins-test-harness is bundled into the plugin.
Uninstalling (or disabling) the gerrit-trigger plugin (and in some cases the sonar-gerrit plugin as well, which has gerrit-trigger as a dependency) and restarting caused this ChannelShutdownListener to no longer be loaded (as evidenced by the output of jmap -histo:live <pid> | grep org.jvnet.hudson.test.ChannelShutdownListener), and the memory profile has remained flat over the past 24 hours.
I have attached a screenshot of the leak suspect from MAT https://www.eclipse.org/mat/
The jenkins-test-harness is pulled in here in the pom: https://github.com/jenkinsci/gerrit-trigger-plugin/blob/master/pom.xml#L280-L283, which was brought in with this commit in February: https://github.com/jenkinsci/gerrit-trigger-plugin/commit/04fd6aff727c0104344c91c72dda372c23cac4ba
The release notes suggest that 2.34.0 was the release that brought in in the change as part of https://issues.jenkins.io/browse/JENKINS-53250
I don't understand the interaction of this test-harness library and the running system, but it seems to be injecting itself in and causing the leak. If it's just a test harness it should probably be a dev dependency and not included in the main plugin, perhaps it came in by mistake?
Note: It does not appear to matter whether the gerrit plugin in used or not (and in the worst case I mentioned before, it wasn't used at all), so we are working to uninstall it where it isn't needed.
- is caused by
JENKINS-53250 Gerrit Trigger plugin compatibility with Jenkins As Code plugin (JEP-201)
- Fixed but Unreleased
- is related to
JENKINS-67938 Plugin should not bundle jenkins-test-harness as it causes a memory leak
- relates to
JENKINS-65650 java.lang.NoClassDefFoundError: org/junit/rules/ExternalResource on builds after upgrade to 2.277.4
- Fixed but Unreleased
JENKINS-66060 Cavisson NS-ND Integration Performance Publisher bundles Jenkins test harness, leading to java.lang.NoClassDefFoundError and memory leaks
Just chiming in to say that we're seeing the same thing on 2.291.
JENKINS-66060 is a similar bug that affects the Cavisson NS-ND Integration Performance Publisher plugin.
The test harness was removed from hpi in https://github.com/jenkinsci/gerrit-trigger-plugin/commit/63407f4c4a314043b39b21c5cc340bb9a8200e7b. Resoled in alpha.
Can someone please verify this is addressed? Thanks!
rsandell olivergondza My team was also experiencing this issue, we upgrade to 2.35.0-beta-1 and the memory leak appears to be resolved. Thanks for addressing
rsandell, Can you please cut a stable release? This memory leak is affecting numerous users.
The unexpected presence of this test harness has been mentioned in https://issues.jenkins.io/browse/JENKINS-65650 .