• Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • None
    • Linux
      Jenkins 2.60.2
      The latest Pipeline plug-ins as of 2017-07-09

      After one of the recent updates to pipeline plug-ins the process tree killer always kills all processes even if the BUILD_ID (as described in https://wiki.jenkins.io/display/JENKINS/ProcessTreeKiller) is changed. This is a serious regression because some of our build jobs stopped working.

      The following pipeline script can be used to reproduce the problem:

      node('master') {
        sh '''BUILD_ID="dont-kill-me bla/blub" sleep 60 &'''
      }
      
      

      The sleep process is killed immediately when the job finishes. The problem only occurs in Pipeline jobs, freestyle jobs still work as expected, i.e. the sleep process continues running for 60 seconds.

          [JENKINS-46089] ProcessTreeKiller broken in pipeline jobs

          Sam Van Oort added a comment -

          Hi Thorsten,
          This is Pipeline working as designed – what you're describing would permit the Pipeline to leak processes potentially due to how it's launching them with the shell step. IIUC what you're saying, the fact that it doesn't kill the process in FreeStyle projects would be a bug with Freestyle.

          Sam Van Oort added a comment - Hi Thorsten, This is Pipeline working as designed – what you're describing would permit the Pipeline to leak processes potentially due to how it's launching them with the shell step. IIUC what you're saying, the fact that it doesn't kill the process in FreeStyle projects would be a bug with Freestyle.

          Hi svanoort,

          this is either a bug for FreeStyle projects and therefore also a bug in the documentation, because the documentation page of the ProcessTreeKiller @ https://wiki.jenkins.io/display/JENKINS/ProcessTreeKiller clearly states what you should do when you want to keep a Daemon process alive. Or this is working as intended for FreeStyle projects but is broken for Pipeline builds.

          Besides there are very good reasons for keeping a process running. E.g. when you are using Gradle for your build environment, Gradle starts a Daemon that make subsequent builds faster. As Jenkins kills all Daemons at the end of each build, it clearly breaks a feature of Gradle. So there have to be a possibility to keep created processes alive.

          oleg_nenashev or jglick maybe one of you can comment on this.

          Thanks in advance!

          Georg Friedrich added a comment - Hi svanoort , this is either a bug for FreeStyle projects and therefore also a bug in the documentation, because the documentation page of the ProcessTreeKiller @ https://wiki.jenkins.io/display/JENKINS/ProcessTreeKiller clearly states what you should do when you want to keep a Daemon process alive. Or this is working as intended for FreeStyle projects but is broken for Pipeline builds. Besides there are very good reasons for keeping a process running. E.g. when you are using Gradle for your build environment, Gradle starts a Daemon that make subsequent builds faster. As Jenkins kills all Daemons at the end of each build, it clearly breaks a feature of Gradle. So there have to be a possibility to keep created processes alive. oleg_nenashev or jglick maybe one of you can comment on this. Thanks in advance!

          Oleg Nenashev added a comment -

          Generally I agree with your diagnosis gfriedrich. The feature is documented in a way that it makes impression that it should work in Pipeline. Just because there was no Pipeline in April 2011 when the page was updated last time...

          "dont-kill-me" always seemed to be a hack to me. If there are processes which should not be terminated by builds, they should be ideally to be started in another way (e.g. by service, etc.). In the case of Gradlethe daemon could be started automatically outside the Jenkins job (e.g. by VM initialization logic or by a Gradle tool installer).

          • If Durable Task logic does not support ProcessTreeKiller extension points, this is a bug.
          • If it just does not support "dont-kill-me", I think it would be enough to document it.

          Oleg Nenashev added a comment - Generally I agree with your diagnosis gfriedrich . The feature is documented in a way that it makes impression that it should work in Pipeline. Just because there was no Pipeline in April 2011 when the page was updated last time... "dont-kill-me" always seemed to be a hack to me. If there are processes which should not be terminated by builds, they should be ideally to be started in another way (e.g. by service, etc.). In the case of Gradlethe daemon could be started automatically outside the Jenkins job (e.g. by VM initialization logic or by a Gradle tool installer). If Durable Task logic does not support ProcessTreeKiller extension points, this is a bug. If it just does not support "dont-kill-me", I think it would be enough to document it.

          It would be a real shame if this very useful functionality wasn't available in Pipeline jobs. We use it to deploy the full application stack after a build which can then be used for testing and playing around. Since we have many branches it's not feasible to set up environments outside of Jenkins and somehow deploy the build results.

          Thorsten Meinl added a comment - It would be a real shame if this very useful functionality wasn't available in Pipeline jobs. We use it to deploy the full application stack after a build which can then be used for testing and playing around. Since we have many branches it's not feasible to set up environments outside of Jenkins and somehow deploy the build results.

          I think so too, oleg_nenashev. Overriding the BUILD_ID environment variable looks very hacky.

          Unfortunately there is no possibility for Gradle to run the daemons initially. And of course the daemon feature of Gradle will only make sense for "always-on" build nodes, so using it in combination with docker agents is useless.

          That said: Would it be possible (from an architectural point of view) to disable the ProcessTreeKiller for specific builds instead of using this BUILD_ID hack? I think there are enough reasons here to give users the possibility to do so.

          Georg Friedrich added a comment - I think so too, oleg_nenashev . Overriding the BUILD_ID environment variable looks very hacky. Unfortunately there is no possibility for Gradle to run the daemons initially. And of course the daemon feature of Gradle will only make sense for "always-on" build nodes, so using it in combination with docker agents is useless. That said: Would it be possible (from an architectural point of view) to disable the ProcessTreeKiller for specific builds instead of using this BUILD_ID hack? I think there are enough reasons here to give users the possibility to do so.

          Ah, I just found that there already IS the ability to stop Jenkins from killing processes by using the "veto kill process"-feature. You should have mentioned this oleg_nenashev

          As far as I can see this is/was bugged for slave nodes but just got fixed 2 days ago. Unfortunately it missed the 2.119 release but I guess it will be available with 2.120 then.

          That's really great news, so I will wait patiently for the 2.120 and will create an extension for not killing the Gradle daemons.

          Georg Friedrich added a comment - Ah, I just found that there already IS the ability to stop Jenkins from killing processes by using the "veto kill process"-feature. You should have mentioned this oleg_nenashev As far as I can see this is/was bugged for slave nodes but just got fixed 2 days ago. Unfortunately it missed the 2.119 release but I guess it will be available with 2.120 then. That's really great news, so I will wait patiently for the 2.120 and will create an extension for not killing the Gradle daemons.

          Jesse Glick added a comment -

          Try

          sh '''
          JENKINS_SERVER_COOKIE=ignore ./gradlew whatever
          '''
          

          Maybe it works, maybe not, just a thought.

          Jesse Glick added a comment - Try sh ''' JENKINS_SERVER_COOKIE=ignore ./gradlew whatever ''' Maybe it works, maybe not, just a thought.

            Unassigned Unassigned
            sithmein Thorsten Meinl
            Votes:
            1 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: