Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-71970

Memory leak due to channel listeners that are never cleared

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • workflow-api-plugin
    • Tested with Jenkins 2.375.4 and 2.414.1.
      workflow-api-plugin version: 1267.vd9b_a_ddd9eb_47

    Description

      Nodes have started experiencing memory leak issues after updating to workflow-api-plugin 1267.vd9b_a_ddd9eb_47. E.g. running a sh step in a pipeline will reserve memory which is never cleared. Eventually node's JVM will run out of memory as it gets filled up apparently with BufferedBuildListener$Replacements which each reserve 65 KB buffer. See screenshot.png and screenshot2.png.

      Stopping the job or performing a GC will not clear the reserved memory. Only solution currently is to disconnect the node from Jenkins and reconnect it to clear up its memory.

       

      Possible root cause

      https://github.com/jenkinsci/workflow-api-plugin/releases/tag/1248.v4b_91043341d2 --> https://github.com/jenkinsci/workflow-api-plugin/pull/294 - this change might have introduced a problem where a listener is added to the Channel but never removed.

       

      Reproduction steps

      Start a loop job that opens up a shell:

      timestamps {
        node("nodeName") {
          int i = 0
          while (true) {
            sh "echo ${i}"
            i++
            // Optionally use jmap to see the BufferedBuildListener count growing:
            //sh "jmap -histo <remoting-jar-pid>|egrep '(BufferedBuildListener|[[]B)'"
            //sleep time: 1, unit: "SECONDS"     
          }
        }
      } 

      Once run is started, see [jenkinsUrl]/manage/computer/[nodeName]/dumpExportTable → The dump will start showing an increasing number of object=hudson.CloseProofOutputStream@ABC123 type=hudson.CloseProofOutputStream interfaces=[java.io.OutputStream] - these will not get cleared until node is either rebooted or disconnected+connected from Jenkins.

       

      Affected component: tested only with workflow-api-plugin 1267.vd9b_a_ddd9eb_47 version. But could be already in version 1248.v4b_91043341d2 if analyzed root cause is correct.

       

      Workaround: Do a disconnect+connect to a node, this will clear up its memory.

       

      Possible similar issue: https://issues.jenkins.io/browse/JENKINS-70388

      Attachments

        Issue Links

          Activity

            People

              jglick Jesse Glick
              mattik Matti
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: