Tested with Jenkins 2.375.4 and 2.414.1.
workflow-api-plugin version: 1267.vd9b_a_ddd9eb_47
Nodes have started experiencing memory leak issues after updating to workflow-api-plugin 1267.vd9b_a_ddd9eb_47. E.g. running a sh step in a pipeline will reserve memory which is never cleared. Eventually node's JVM will run out of memory as it gets filled up apparently with BufferedBuildListener$Replacements which each reserve 65 KB buffer. See screenshot.png and screenshot2.png.
Stopping the job or performing a GC will not clear the reserved memory. Only solution currently is to disconnect the node from Jenkins and reconnect it to clear up its memory.
Possible root cause
https://github.com/jenkinsci/workflow-api-plugin/releases/tag/1248.v4b_91043341d2 --> https://github.com/jenkinsci/workflow-api-plugin/pull/294 - this change might have introduced a problem where a listener is added to the Channel but never removed.
Start a loop job that opens up a shell:
Once run is started, see [jenkinsUrl]/manage/computer/[nodeName]/dumpExportTable → The dump will start showing an increasing number of object=hudson.CloseProofOutputStream@ABC123 type=hudson.CloseProofOutputStream interfaces=[java.io.OutputStream] - these will not get cleared until node is either rebooted or disconnected+connected from Jenkins.
Affected component: tested only with workflow-api-plugin 1267.vd9b_a_ddd9eb_47 version. But could be already in version 1248.v4b_91043341d2 if analyzed root cause is correct.
Workaround: Do a disconnect+connect to a node, this will clear up its memory.
Possible similar issue: https://issues.jenkins.io/browse/JENKINS-70388