Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65771

Memory leak with gerrit-trigger plugin installed

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: gerrit-trigger-plugin
    • Labels:
      None
    • Environment:
      Jenkins LTS 2.277.2
      gerrit-trigger version 2.34.0 (latest)
    • Similar Issues:
    • Released As:
      2.35.0-beta-1

      Description

      Since upgrading a large number (100s) of Jenkins instances to LTS v2.277.2, and the associated plugin updates required for the UI changes, I have observed a number of these instances are leak memory, and in one case it only lasts about 36 hours before exhausting the heap (in that case it is 6GB).

      a heapdump (e.g. jmap -dump:live,file=jenkins-20210603-0939.hprof) showed that there were a large number (1000s) of hudson.remoting.Channel classes retained in a list by the class org.jvnet.hudson.test.ChannelShutdownListener. Doing a grep in the installed plugins folder, the only plugin that referenced this class was the gerrit-trigger plugin:

      plugins$ grep -iRH "org.jvnet.hudson.test.ChannelShutdownListener" *
      Binary file gerrit-trigger/WEB-INF/lib/jenkins-test-harness-2.56.jar matches
       
      I found it curious that a test class was being included in a production plugin, so I do not know why this jenkins-test-harness is bundled into the plugin.

      Uninstalling (or disabling) the gerrit-trigger plugin (and in some cases the sonar-gerrit plugin as well, which has gerrit-trigger as a dependency) and restarting caused this ChannelShutdownListener to no longer be loaded (as evidenced by the output of jmap -histo:live <pid> | grep org.jvnet.hudson.test.ChannelShutdownListener), and the memory profile has remained flat over the past 24 hours.

      I have attached a screenshot of the leak suspect from MAT https://www.eclipse.org/mat/
       
      The jenkins-test-harness is pulled in here in the pom: https://github.com/jenkinsci/gerrit-trigger-plugin/blob/master/pom.xml#L280-L283, which was brought in with this commit in February: https://github.com/jenkinsci/gerrit-trigger-plugin/commit/04fd6aff727c0104344c91c72dda372c23cac4ba

      The release notes suggest that 2.34.0 was the release that brought in in the change as part of https://issues.jenkins.io/browse/JENKINS-53250

      I don't understand the interaction of this test-harness library and the running system, but it seems to be injecting itself in and causing the leak. If it's just a test harness it should probably be a dev dependency and not included in the main plugin, perhaps it came in by mistake?
       
      Note: It does not appear to matter whether the gerrit plugin in used or not (and in the worst case I mentioned before, it wasn't used at all), so we are working to uninstall it where it isn't needed.

        Attachments

          Issue Links

            Activity

            Hide
            fraz3alpha Andy Taylor added a comment -

            The unexpected presence of this test harness has been mentioned in https://issues.jenkins.io/browse/JENKINS-65650 .

            Show
            fraz3alpha Andy Taylor added a comment - The unexpected presence of this test harness has been mentioned in https://issues.jenkins.io/browse/JENKINS-65650  .
            Hide
            rmoszczy Radoslaw Moszczynski added a comment -

            Just chiming in to say that we're seeing the same thing on 2.291.

            Show
            rmoszczy Radoslaw Moszczynski added a comment - Just chiming in to say that we're seeing the same thing on 2.291.
            Hide
            basil Basil Crow added a comment -

            JENKINS-66060 is a similar bug that affects the Cavisson NS-ND Integration Performance Publisher plugin.

            Show
            basil Basil Crow added a comment - JENKINS-66060 is a similar bug that affects the Cavisson NS-ND Integration Performance Publisher plugin.
            Hide
            olivergondza Oliver Gondža added a comment -

            The test harness was removed from hpi in https://github.com/jenkinsci/gerrit-trigger-plugin/commit/63407f4c4a314043b39b21c5cc340bb9a8200e7b. Resoled in alpha.

            Can someone please verify this is addressed? Thanks!

            Show
            olivergondza Oliver Gondža added a comment - The test harness was removed from hpi in https://github.com/jenkinsci/gerrit-trigger-plugin/commit/63407f4c4a314043b39b21c5cc340bb9a8200e7b . Resoled in alpha. Can someone please verify this is addressed? Thanks!
            Hide
            jamesforsee James Forsee added a comment -

            rsandell Oliver Gondža My team was also experiencing this issue, we upgrade to 2.35.0-beta-1 and the memory leak appears to be resolved. Thanks for addressing

            Show
            jamesforsee James Forsee added a comment - rsandell Oliver Gondža  My team was also experiencing this issue, we upgrade to 2.35.0-beta-1 and the memory leak appears to be resolved. Thanks for addressing
            Hide
            olivergondza Oliver Gondža added a comment -
            Show
            olivergondza Oliver Gondža added a comment - rsandell , Can you please cut a stable release? This memory leak is affecting numerous users. Fix: https://issues.jenkins.io/browse/JENKINS-65771?focusedCommentId=411431&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-411431

              People

              Assignee:
              rsandell rsandell
              Reporter:
              fraz3alpha Andy Taylor
              Votes:
              3 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: