Nodes have started experiencing memory leak issues after updating to workflow-api-plugin 1267.vd9b_a_ddd9eb_47. E.g. running a sh step in a pipeline will reserve memory which is never cleared. Eventually node's JVM will run out of memory as it gets filled up apparently with BufferedBuildListener$Replacements which each reserve 65 KB buffer. See screenshot.png and screenshot2.png.

Stopping the job or performing a GC will not clear the reserved memory. Only solution currently is to disconnect the node from Jenkins and reconnect it to clear up its memory.

Possible root cause

https://github.com/jenkinsci/workflow-api-plugin/releases/tag/1248.v4b_91043341d2 --> https://github.com/jenkinsci/workflow-api-plugin/pull/294 - this change might have introduced a problem where a listener is added to the Channel but never removed.

Reproduction steps

Start a loop job that opens up a shell:

timestamps {
  node("nodeName") {
    int i = 0
    while (true) {
      sh "echo ${i}"
      i++
      // Optionally use jmap to see the BufferedBuildListener count growing:
      //sh "jmap -histo <remoting-jar-pid>|egrep '(BufferedBuildListener|[[]B)'"
      //sleep time: 1, unit: "SECONDS"     
    }
  }
} 

Once run is started, see [jenkinsUrl]/manage/computer/[nodeName]/dumpExportTable → The dump will start showing an increasing number of object=hudson.CloseProofOutputStream@ABC123 type=hudson.CloseProofOutputStream interfaces=[java.io.OutputStream] - these will not get cleared until node is either rebooted or disconnected+connected from Jenkins.

Affected component: tested only with workflow-api-plugin 1267.vd9b_a_ddd9eb_47 version. But could be already in version 1248.v4b_91043341d2 if analyzed root cause is correct.

Workaround: Do a disconnect+connect to a node, this will clear up its memory.

Possible similar issue: https://issues.jenkins.io/browse/JENKINS-70388