Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65771

Memory leak with gerrit-trigger plugin installed

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • gerrit-trigger-plugin
    • None
    • Jenkins LTS 2.277.2
      gerrit-trigger version 2.34.0 (latest)
    • 2.35.0-beta-1

      Since upgrading a large number (100s) of Jenkins instances to LTS v2.277.2, and the associated plugin updates required for the UI changes, I have observed a number of these instances are leak memory, and in one case it only lasts about 36 hours before exhausting the heap (in that case it is 6GB).

      a heapdump (e.g. jmap -dump:live,file=jenkins-20210603-0939.hprof) showed that there were a large number (1000s) of hudson.remoting.Channel classes retained in a list by the class org.jvnet.hudson.test.ChannelShutdownListener. Doing a grep in the installed plugins folder, the only plugin that referenced this class was the gerrit-trigger plugin:

      plugins$ grep -iRH "org.jvnet.hudson.test.ChannelShutdownListener" *
      Binary file gerrit-trigger/WEB-INF/lib/jenkins-test-harness-2.56.jar matches
       
      I found it curious that a test class was being included in a production plugin, so I do not know why this jenkins-test-harness is bundled into the plugin.

      Uninstalling (or disabling) the gerrit-trigger plugin (and in some cases the sonar-gerrit plugin as well, which has gerrit-trigger as a dependency) and restarting caused this ChannelShutdownListener to no longer be loaded (as evidenced by the output of jmap -histo:live <pid> | grep org.jvnet.hudson.test.ChannelShutdownListener), and the memory profile has remained flat over the past 24 hours.

      I have attached a screenshot of the leak suspect from MAT https://www.eclipse.org/mat/
       
      The jenkins-test-harness is pulled in here in the pom: https://github.com/jenkinsci/gerrit-trigger-plugin/blob/master/pom.xml#L280-L283, which was brought in with this commit in February: https://github.com/jenkinsci/gerrit-trigger-plugin/commit/04fd6aff727c0104344c91c72dda372c23cac4ba

      The release notes suggest that 2.34.0 was the release that brought in in the change as part of https://issues.jenkins.io/browse/JENKINS-53250

      I don't understand the interaction of this test-harness library and the running system, but it seems to be injecting itself in and causing the leak. If it's just a test harness it should probably be a dev dependency and not included in the main plugin, perhaps it came in by mistake?
       
      Note: It does not appear to matter whether the gerrit plugin in used or not (and in the worst case I mentioned before, it wasn't used at all), so we are working to uninstall it where it isn't needed.

          [JENKINS-65771] Memory leak with gerrit-trigger plugin installed

          Andy Taylor added a comment -

          The unexpected presence of this test harness has been mentioned in https://issues.jenkins.io/browse/JENKINS-65650 .

          Andy Taylor added a comment - The unexpected presence of this test harness has been mentioned in https://issues.jenkins.io/browse/JENKINS-65650  .

          Just chiming in to say that we're seeing the same thing on 2.291.

          Radoslaw Moszczynski added a comment - Just chiming in to say that we're seeing the same thing on 2.291.

          Basil Crow added a comment -

          JENKINS-66060 is a similar bug that affects the Cavisson NS-ND Integration Performance Publisher plugin.

          Basil Crow added a comment - JENKINS-66060 is a similar bug that affects the Cavisson NS-ND Integration Performance Publisher plugin.

          The test harness was removed from hpi in https://github.com/jenkinsci/gerrit-trigger-plugin/commit/63407f4c4a314043b39b21c5cc340bb9a8200e7b. Resoled in alpha.

          Can someone please verify this is addressed? Thanks!

          Oliver Gondža added a comment - The test harness was removed from hpi in https://github.com/jenkinsci/gerrit-trigger-plugin/commit/63407f4c4a314043b39b21c5cc340bb9a8200e7b . Resoled in alpha. Can someone please verify this is addressed? Thanks!

          James Forsee added a comment -

          rsandell olivergondza My team was also experiencing this issue, we upgrade to 2.35.0-beta-1 and the memory leak appears to be resolved. Thanks for addressing

          James Forsee added a comment - rsandell olivergondza  My team was also experiencing this issue, we upgrade to 2.35.0-beta-1 and the memory leak appears to be resolved. Thanks for addressing

          Oliver Gondža added a comment - rsandell , Can you please cut a stable release? This memory leak is affecting numerous users. Fix: https://issues.jenkins.io/browse/JENKINS-65771?focusedCommentId=411431&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-411431

            rsandell rsandell
            fraz3alpha Andy Taylor
            Votes:
            3 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: