Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-26048

Jenkins no longer cleaning up child processes when build stopped - as of 1.587

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • core, (1)
      maven-plugin
    • CentOS 6, JRE 7 or 8
      Windows Server 2008 R2, JRE 7 or 8

      Similar to bug 22641, Jenkins no longer cleaning up child processes when a job is stopped.

      To reproduce : Start any job with a surefire execution, once the tests start running, stop the job through the jenkins interface. The surefire process continues to run.

      Just like bug 22641, this behavior started in 1.553, was resolved in 1.565.2, then started again in 1.587.

      Rating this as critical because processes build up eventually causing issues with running out of memory or hitting the nproc limit.

          [JENKINS-26048] Jenkins no longer cleaning up child processes when build stopped - as of 1.587

          Don Bogardus added a comment - - edited

          Verified this is still happening in latest jenkins version on CentOS 6. 

          For the test I'm running java tests with the failsafe plugin, which launches multiple jvms for concurrent execution and many phantomjs processes. 

          1.586 - When build is stopped, all child processes are stopped

          1.587 - When build is stopped, all child processes keep running

          2.66.1 - When build is stopped, all child processes keep running

           

          Don Bogardus added a comment - - edited Verified this is still happening in latest jenkins version on CentOS 6.  For the test I'm running java tests with the failsafe plugin, which launches multiple jvms for concurrent execution and many phantomjs processes.  1.586 - When build is stopped, all child processes are stopped 1.587 - When build is stopped, all child processes keep running 2.66.1 - When build is stopped, all child processes keep running  

          Daniel Beck added a comment -

          dbogardus Are you able to reproduce this issue with a simpler environment? What are the complete steps to reproduce from scratch?

          When you write 2.66.1, do you mean 2.46.1?

          Daniel Beck added a comment - dbogardus Are you able to reproduce this issue with a simpler environment? What are the complete steps to reproduce from scratch? When you write 2.66.1, do you mean 2.46.1?

          Don Bogardus added a comment - - edited

          danielbeck , I used the latest weekly version - 2.66.1 . (It's just what came down from yum)

          I am able to reproduce it with the simple java/mvn/surefire scenario created by rddesmond :

          https://issues.jenkins-ci.org/browse/JENKINS-28968?focusedCommentId=230600&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-230600

          With this small example, 1.586 cleans up the surefire process, and 2.66.1 does not. 

           

           

          Don Bogardus added a comment - - edited danielbeck , I used the latest weekly version - 2.66.1 . (It's just what came down from yum) I am able to reproduce it with the simple java/mvn/surefire scenario created by rddesmond : https://issues.jenkins-ci.org/browse/JENKINS-28968?focusedCommentId=230600&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-230600 With this small example, 1.586 cleans up the surefire process, and 2.66.1 does not.     

          Oleg Nenashev added a comment -

          dbogardus I propose to move the discussion back to the JENKINS-28968 thread. Maven Project plugin is a very specific case in terms of the architecture, so I would rather vote for handling it separately until we confirm there is a generic issue in the core. So far I am not conviced

          Oleg Nenashev added a comment - dbogardus  I propose to move the discussion back to the  JENKINS-28968 thread. Maven Project plugin is a very specific case in terms of the architecture, so I would rather vote for handling it separately until we confirm there is a generic issue in the core. So far I am not conviced

          Don Bogardus added a comment -

          oleg_nenashev Acknowledged. I did try a few things to reproduce the bug outside of the maven/surefire scenario but was unable to. 

          Don Bogardus added a comment - oleg_nenashev Acknowledged. I did try a few things to reproduce the bug outside of the maven/surefire scenario but was unable to. 

          Mor L added a comment -

          Hi,

          We're facing a similar issue - but we're using gradle and not using maven or surefire.

          We use pipeline and call gradle using the bat step (windows 64 bit slaves, master is on windows too).

          This happens on 2 separate projects in which gradle spawns child processes for testing using various other tools. Sometimes 'gradle test' hangs due to bugs in the test code/scenarios and when we stop the build - there are always java processes left up which we have to kill manually.

          Verified that ProcessTreeKiller is not disabled.

          Using Jenkins 2.73.3, pipeline suite 2.5.

          Mor L added a comment - Hi, We're facing a similar issue - but we're using gradle and not using maven or surefire. We use pipeline and call gradle using the bat step (windows 64 bit slaves, master is on windows too). This happens on 2 separate projects in which gradle spawns child processes for testing using various other tools. Sometimes 'gradle test' hangs due to bugs in the test code/scenarios and when we stop the build - there are always java processes left up which we have to kill manually. Verified that ProcessTreeKiller is not disabled. Using Jenkins 2.73.3, pipeline suite 2.5.

          Oleg Nenashev added a comment -

          Apparently the 32bit OS support was partially broken in WinP: https://github.com/kohsuke/winp/issues/48
          Does anybody see the issue on 64bit systems?

          Oleg Nenashev added a comment - Apparently the 32bit OS support was partially broken in WinP: https://github.com/kohsuke/winp/issues/48 Does anybody see the issue on 64bit systems?

          Oleg Nenashev added a comment -

          I will have no time to work on it anytime soon, please see https://groups.google.com/d/msg/jenkinsci-dev/uc6NsMoCFQI/AIO4WG1UCwAJ for the context. I will unassign it so that somebody else can work on it

          Oleg Nenashev added a comment - I will have no time to work on it anytime soon, please see https://groups.google.com/d/msg/jenkinsci-dev/uc6NsMoCFQI/AIO4WG1UCwAJ for the context. I will unassign it so that somebody else can work on it

          Piotr Zolnacz added a comment -

          Hi, 

          I'm going to fix this bug, this is quite critical for my daily work with jenkins. 

          oleg_nenashev it seems like it doesn't appear in 64 bit slaves.

          Piotr Zolnacz added a comment - Hi,  I'm going to fix this bug, this is quite critical for my daily work with jenkins.  oleg_nenashev it seems like it doesn't appear in 64 bit slaves.

          Is there a solution in sight? Or is it solved by running a slave at 64 bit?

          Johannes Schmieder added a comment - Is there a solution in sight? Or is it solved by running a slave at 64 bit?

            pi0tras Piotr Zolnacz
            dbogardus Don Bogardus
            Votes:
            39 Vote for this issue
            Watchers:
            39 Start watching this issue

              Created:
              Updated: