• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Component/s: workflow-api-plugin
• Labels:

  • leak
  • regression
• Environment:

  Tested with Jenkins 2.375.4 and 2.414.1.
  workflow-api-plugin version: 1267.vd9b_a_ddd9eb_47

1. 

   image-2023-09-11-12-54-28-766.png

   112 kB

   2023-09-11 10:54

2. 

   screenshot2.png

   238 kB

   2023-09-06 07:31

3. 

   screenshot.png

   41 kB

   2023-09-06 07:27

[JENKINS-71970] Memory leak due to channel listeners that are never cleared

Matti created issue - 2023-09-06 07:54

Matti made changes - 2023-09-06 08:47

Description

Original: Nodes have started experiencing memory leak issues after updating to workflow-api-plugin 1267.vd9b_a_ddd9eb_47. E.g. running a sh step in a pipeline will reserve memory which is never cleared. Eventually node's JVM will run out of memory as it gets filled up apparently with BufferedBuildListener$Replacements which each reserve 65 KB buffer. See screenshot.png and screenshot2.png. Stopping the job or performing a GC will +not+ clear the reserved memory. Only solution currently is to disconnect the node from Jenkins and reconnect it to clear up its memory.   *Possible root cause* [https://github.com/jenkinsci/workflow-api-plugin/releases/tag/1248.v4b_91043341d2] --> [https://github.com/jenkinsci/workflow-api-plugin/pull/294] - this change might have introduced a problem where a listener is added to the Channel but never removed.   *Reproduction steps* Start a loop job that opens up a shell: timestamps \{   node("nodeName") {     int i = 0     while (true) {       sh "echo ${i}"       i++       // Optionally use jmap to see the BufferedBuildListener count growing:       //sh "jmap -histo <remoting-jar-pid>|egrep '(BufferedBuildListener|[[]B)'"       //sleep time: 1, unit: "SECONDS"         }   } } Once run is started, see {_}[{_}_jenkinsUrl]/manage/computer/[nodeName]/dumpExportTable_ → The dump will start showing an increasing number of _object=hudson.CloseProofOutputStream@ABC123 type=hudson.CloseProofOutputStream interfaces=[java.io.OutputStream]_ - these will not get cleared until node is either rebooted or disconnected+connected from Jenkins.   *Affected component:* tested only with {color:#1d1c1d}workflow-api-plugin{color} {color:#1d1c1d}1267.vd9b_a_ddd9eb_47 version. But could be already in version 1248.v4b_91043341d2 if analyzed root cause is correct.{color}   {color:#1d1c1d}*Workaround:* Do a disconnect+connect to a node, this will clear up its memory.{color}

New: Nodes have started experiencing memory leak issues after updating to workflow-api-plugin 1267.vd9b_a_ddd9eb_47. E.g. running a sh step in a pipeline will reserve memory which is never cleared. Eventually node's JVM will run out of memory as it gets filled up apparently with BufferedBuildListener$Replacements which each reserve 65 KB buffer. See screenshot.png and screenshot2.png. Stopping the job or performing a GC will +not+ clear the reserved memory. Only solution currently is to disconnect the node from Jenkins and reconnect it to clear up its memory.   *Possible root cause* [https://github.com/jenkinsci/workflow-api-plugin/releases/tag/1248.v4b_91043341d2] --> [https://github.com/jenkinsci/workflow-api-plugin/pull/294] - this change might have introduced a problem where a listener is added to the Channel but never removed.   *Reproduction steps* Start a loop job that opens up a shell: {code:java} timestamps {   node("nodeName") {     int i = 0     while (true) {       sh "echo ${i}"       i++       // Optionally use jmap to see the BufferedBuildListener count growing:       //sh "jmap -histo <remoting-jar-pid>|egrep '(BufferedBuildListener|[[]B)'"       //sleep time: 1, unit: "SECONDS"         }   } } {code} Once run is started, see _[\{_}_jenkinsUrl]/manage/computer/[nodeName]/dumpExportTable_ → The dump will start showing an increasing number of _object=hudson.CloseProofOutputStream@ABC123 type=hudson.CloseProofOutputStream interfaces=[java.io.OutputStream]_ - these will not get cleared until node is either rebooted or disconnected+connected from Jenkins.   *Affected component:* tested only with {color:#1d1c1d}workflow-api-plugin{color} {color:#1d1c1d}1267.vd9b_a_ddd9eb_47 version. But could be already in version 1248.v4b_91043341d2 if analyzed root cause is correct.{color}   {color:#1d1c1d}*Workaround:* Do a disconnect+connect to a node, this will clear up its memory.{color}

Matti made changes - 2023-09-06 08:48

Description

Original: Nodes have started experiencing memory leak issues after updating to workflow-api-plugin 1267.vd9b_a_ddd9eb_47. E.g. running a sh step in a pipeline will reserve memory which is never cleared. Eventually node's JVM will run out of memory as it gets filled up apparently with BufferedBuildListener$Replacements which each reserve 65 KB buffer. See screenshot.png and screenshot2.png. Stopping the job or performing a GC will +not+ clear the reserved memory. Only solution currently is to disconnect the node from Jenkins and reconnect it to clear up its memory.   *Possible root cause* [https://github.com/jenkinsci/workflow-api-plugin/releases/tag/1248.v4b_91043341d2] --> [https://github.com/jenkinsci/workflow-api-plugin/pull/294] - this change might have introduced a problem where a listener is added to the Channel but never removed.   *Reproduction steps* Start a loop job that opens up a shell: {code:java} timestamps {   node("nodeName") {     int i = 0     while (true) {       sh "echo ${i}"       i++       // Optionally use jmap to see the BufferedBuildListener count growing:       //sh "jmap -histo <remoting-jar-pid>|egrep '(BufferedBuildListener|[[]B)'"       //sleep time: 1, unit: "SECONDS"         }   } } {code} Once run is started, see _[\{_}_jenkinsUrl]/manage/computer/[nodeName]/dumpExportTable_ → The dump will start showing an increasing number of _object=hudson.CloseProofOutputStream@ABC123 type=hudson.CloseProofOutputStream interfaces=[java.io.OutputStream]_ - these will not get cleared until node is either rebooted or disconnected+connected from Jenkins.   *Affected component:* tested only with {color:#1d1c1d}workflow-api-plugin{color} {color:#1d1c1d}1267.vd9b_a_ddd9eb_47 version. But could be already in version 1248.v4b_91043341d2 if analyzed root cause is correct.{color}   {color:#1d1c1d}*Workaround:* Do a disconnect+connect to a node, this will clear up its memory.{color}

New: Nodes have started experiencing memory leak issues after updating to workflow-api-plugin 1267.vd9b_a_ddd9eb_47. E.g. running a sh step in a pipeline will reserve memory which is never cleared. Eventually node's JVM will run out of memory as it gets filled up apparently with BufferedBuildListener$Replacements which each reserve 65 KB buffer. See screenshot.png and screenshot2.png. Stopping the job or performing a GC will +not+ clear the reserved memory. Only solution currently is to disconnect the node from Jenkins and reconnect it to clear up its memory.   *Possible root cause* [https://github.com/jenkinsci/workflow-api-plugin/releases/tag/1248.v4b_91043341d2] --> [https://github.com/jenkinsci/workflow-api-plugin/pull/294] - this change might have introduced a problem where a listener is added to the Channel but never removed.   *Reproduction steps* Start a loop job that opens up a shell: {code:java} timestamps {   node("nodeName") {     int i = 0     while (true) {       sh "echo ${i}"       i++       // Optionally use jmap to see the BufferedBuildListener count growing:       //sh "jmap -histo <remoting-jar-pid>|egrep '(BufferedBuildListener|[[]B)'"       //sleep time: 1, unit: "SECONDS"         }   } } {code} Once run is started, see _[\\{_}_jenkinsUrl]/manage/computer/[nodeName]/dumpExportTable_ → The dump will start showing an increasing number of _object=hudson.CloseProofOutputStream@ABC123 type=hudson.CloseProofOutputStream interfaces=[java.io.OutputStream]_ - these will not get cleared until node is either rebooted or disconnected+connected from Jenkins.   *Affected component:* tested only with {color:#1d1c1d}workflow-api-plugin{color} {color:#1d1c1d}1267.vd9b_a_ddd9eb_47 version. But could be already in version 1248.v4b_91043341d2 if analyzed root cause is correct.{color}   {color:#1d1c1d}*Workaround:* Do a disconnect+connect to a node, this will clear up its memory.{color}   {color:#1d1c1d}Possible similar issue: https://issues.jenkins.io/browse/JENKINS-70388{color}

Matti made changes - 2023-09-06 08:59

Description

Original: Nodes have started experiencing memory leak issues after updating to workflow-api-plugin 1267.vd9b_a_ddd9eb_47. E.g. running a sh step in a pipeline will reserve memory which is never cleared. Eventually node's JVM will run out of memory as it gets filled up apparently with BufferedBuildListener$Replacements which each reserve 65 KB buffer. See screenshot.png and screenshot2.png. Stopping the job or performing a GC will +not+ clear the reserved memory. Only solution currently is to disconnect the node from Jenkins and reconnect it to clear up its memory.   *Possible root cause* [https://github.com/jenkinsci/workflow-api-plugin/releases/tag/1248.v4b_91043341d2] --> [https://github.com/jenkinsci/workflow-api-plugin/pull/294] - this change might have introduced a problem where a listener is added to the Channel but never removed.   *Reproduction steps* Start a loop job that opens up a shell: {code:java} timestamps {   node("nodeName") {     int i = 0     while (true) {       sh "echo ${i}"       i++       // Optionally use jmap to see the BufferedBuildListener count growing:       //sh "jmap -histo <remoting-jar-pid>|egrep '(BufferedBuildListener|[[]B)'"       //sleep time: 1, unit: "SECONDS"         }   } } {code} Once run is started, see _[\\{_}_jenkinsUrl]/manage/computer/[nodeName]/dumpExportTable_ → The dump will start showing an increasing number of _object=hudson.CloseProofOutputStream@ABC123 type=hudson.CloseProofOutputStream interfaces=[java.io.OutputStream]_ - these will not get cleared until node is either rebooted or disconnected+connected from Jenkins.   *Affected component:* tested only with {color:#1d1c1d}workflow-api-plugin{color} {color:#1d1c1d}1267.vd9b_a_ddd9eb_47 version. But could be already in version 1248.v4b_91043341d2 if analyzed root cause is correct.{color}   {color:#1d1c1d}*Workaround:* Do a disconnect+connect to a node, this will clear up its memory.{color}   {color:#1d1c1d}Possible similar issue: https://issues.jenkins.io/browse/JENKINS-70388{color}

New: Nodes have started experiencing memory leak issues after updating to workflow-api-plugin 1267.vd9b_a_ddd9eb_47. E.g. running a sh step in a pipeline will reserve memory which is never cleared. Eventually node's JVM will run out of memory as it gets filled up apparently with BufferedBuildListener$Replacements which each reserve 65 KB buffer. See screenshot.png and screenshot2.png. Stopping the job or performing a GC will +not+ clear the reserved memory. Only solution currently is to disconnect the node from Jenkins and reconnect it to clear up its memory.   *Possible root cause* [https://github.com/jenkinsci/workflow-api-plugin/releases/tag/1248.v4b_91043341d2] --> [https://github.com/jenkinsci/workflow-api-plugin/pull/294] - this change might have introduced a problem where a listener is added to the Channel but never removed.   *Reproduction steps* Start a loop job that opens up a shell: {code:java} timestamps {   node("nodeName") {     int i = 0     while (true) {       sh "echo ${i}"       i++       // Optionally use jmap to see the BufferedBuildListener count growing:       //sh "jmap -histo <remoting-jar-pid>|egrep '(BufferedBuildListener|[[]B)'"       //sleep time: 1, unit: "SECONDS"         }   } } {code} Once run is started, see _[jenkinsUrl]/manage/computer/[nodeName]/dumpExportTable_ → The dump will start showing an increasing number of _object=hudson.CloseProofOutputStream@ABC123 type=hudson.CloseProofOutputStream interfaces=[java.io.OutputStream]_ - these will not get cleared until node is either rebooted or disconnected+connected from Jenkins.   *Affected component:* tested only with {color:#1d1c1d}workflow-api-plugin{color} {color:#1d1c1d}1267.vd9b_a_ddd9eb_47 version. But could be already in version 1248.v4b_91043341d2 if analyzed root cause is correct.{color}   {color:#1d1c1d}*Workaround:* Do a disconnect+connect to a node, this will clear up its memory.{color}   {color:#1d1c1d}Possible similar issue: https://issues.jenkins.io/browse/JENKINS-70388{color}

Florent made changes - 2023-09-11 10:54

Attachment

New: image-2023-09-11-12-53-59-067.png [ 61123 ]

Florent made changes - 2023-09-11 10:54

Attachment

New: image-2023-09-11-12-54-28-766.png [ 61124 ]

Florent made changes - 2023-09-11 10:54

Attachment

Original: image-2023-09-11-12-53-59-067.png [ 61123 ]

Collapse comment: Florent added a comment - 2023-09-11 10:55

Florent added a comment - 2023-09-11 10:55

We have same issue on Jenkins 2.414.1 after upgrading plugin to 1647

Expand comment: Florent added a comment - 2023-09-11 10:55

Florent added a comment - 2023-09-11 10:55 We have same issue on Jenkins 2.414.1 after upgrading plugin to 1647

Jesse Glick made changes - 2023-09-11 15:41

Assignee

New: Jesse Glick [ jglick ]

Jesse Glick made changes - 2023-09-11 15:42

Assignee

Original: Jesse Glick [ jglick ]

New: rsandell [ rsandell ]

Load more newer events