Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53315

Timeout step should support a closure to execute prior to killing body

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Currently, the timeout step simply kills whatever processes were launched during execution of its body, and then throws an exception.  This makes it difficult to perform any automated debugging on these processes, since they are killed by the time the user finds out that they are hung (or slow).   It would be nice to be able to get some information about the state of affairs before things are killed, and maybe even perform safe shutdown steps prior to kill.

      Currently: 

      try {
        timeout(time: 1, unit: 'HOURS') {
           sh "java IntermittentlySlowProcess"
        }
      } catch (t) {
          //It's too late to, for example, send a "kill -3" to the slow/hung java process
      }
      

      What I'd propose

      (and I'm willing to try to make a PR if this seems reasonable):

      timeout(time: 1, unit: 'HOURS', beforeKill: {
         sh "killall -3 java" //for example
      }) {
         sh "java IntermittentlySlowProcess"
      }
      

      The new  "beforeKill" closure can be used for clean shutdown of complex tasks, analysis of problems, etc.

      One workaround may be to wrap whatever you are running and trap signals, but that's ugly and error-prone (and will likely cause zombies)

      Thoughts welcome.

       

       

        Attachments

          Activity

          akom Alexander Komarov created issue -
          akom Alexander Komarov made changes -
          Field Original Value New Value
          Description Currently, the timeout step simply kills whatever processes were launched during execution of its body, and then throws an exception.  This makes it *difficult to perform any automated debugging* on these processes, since they are killed by the time the user finds out that they are hung (or slow). 

          You can argue that this should be done outside of Jenkins, but, too often, Jenkins is the only place this is visible due to frequency of execution or differences in the environment.
          h3. Currently:

           
          {code:java}
          try {
            timeout(time: 1, unit: 'HOURS') {
               sh "java IntermittentlySlowProcess"
            }
          } catch (t) {
              //It's too late to, for example, send a "kill -3" to the slow/hung java process
          }
          {code}
           
          h3. What I'd propose

          (and I'm willing to try to make a PR if this seems reasonable):

           
          {code:java}
          timeout(time: 1, unit: 'HOURS', beforeKill: {
             sh "killall -3 java" //for example
          }) {
             sh "java IntermittentlySlowProcess"
          }
          {code}
           

           

          "beforeKill" can be used for clean shutdown of complex tasks, analysis of problems, etc.

          Thoughts welcome.
          Currently, the timeout step simply kills whatever processes were launched during execution of its body, and then throws an exception.  This makes it *difficult to perform any automated debugging* on these processes, since they are killed by the time the user finds out that they are hung (or slow). 

          You can argue that this should be done outside of Jenkins, but, too often, Jenkins is the only place this is visible due to frequency of execution or differences in the environment.
          h3. Currently: 
          {code:java}
          try {
            timeout(time: 1, unit: 'HOURS') {
               sh "java IntermittentlySlowProcess"
            }
          } catch (t) {
              //It's too late to, for example, send a "kill -3" to the slow/hung java process
          }
          {code}
           
          h3. What I'd propose

          (and I'm willing to try to make a PR if this seems reasonable):
          {code:java}
          timeout(time: 1, unit: 'HOURS', beforeKill: {
             sh "killall -3 java" //for example
          }) {
             sh "java IntermittentlySlowProcess"
          }
          {code}
          The new  "beforeKill" closure can be used for clean shutdown of complex tasks, analysis of problems, etc.

          Thoughts welcome.
          akom Alexander Komarov made changes -
          Description Currently, the timeout step simply kills whatever processes were launched during execution of its body, and then throws an exception.  This makes it *difficult to perform any automated debugging* on these processes, since they are killed by the time the user finds out that they are hung (or slow). 

          You can argue that this should be done outside of Jenkins, but, too often, Jenkins is the only place this is visible due to frequency of execution or differences in the environment.
          h3. Currently: 
          {code:java}
          try {
            timeout(time: 1, unit: 'HOURS') {
               sh "java IntermittentlySlowProcess"
            }
          } catch (t) {
              //It's too late to, for example, send a "kill -3" to the slow/hung java process
          }
          {code}
           
          h3. What I'd propose

          (and I'm willing to try to make a PR if this seems reasonable):
          {code:java}
          timeout(time: 1, unit: 'HOURS', beforeKill: {
             sh "killall -3 java" //for example
          }) {
             sh "java IntermittentlySlowProcess"
          }
          {code}
          The new  "beforeKill" closure can be used for clean shutdown of complex tasks, analysis of problems, etc.

          Thoughts welcome.
          Currently, the timeout step simply kills whatever processes were launched during execution of its body, and then throws an exception.  This makes it *difficult to perform any automated debugging* on these processes, since they are killed by the time the user finds out that they are hung (or slow). 

          You can argue that this should be done outside of Jenkins, but, too often, Jenkins is the only place this is visible due to frequency of execution or differences in the environment.
          h3. Currently: 
          {code:java}
          try {
            timeout(time: 1, unit: 'HOURS') {
               sh "java IntermittentlySlowProcess"
            }
          } catch (t) {
              //It's too late to, for example, send a "kill -3" to the slow/hung java process
          }
          {code}
          h3. What I'd propose

          (and I'm willing to try to make a PR if this seems reasonable):
          {code:java}
          timeout(time: 1, unit: 'HOURS', beforeKill: {
             sh "killall -3 java" //for example
          }) {
             sh "java IntermittentlySlowProcess"
          }
          {code}
          The new  "beforeKill" closure can be used for clean shutdown of complex tasks, analysis of problems, etc.

          Thoughts welcome.
          akom Alexander Komarov made changes -
          Description Currently, the timeout step simply kills whatever processes were launched during execution of its body, and then throws an exception.  This makes it *difficult to perform any automated debugging* on these processes, since they are killed by the time the user finds out that they are hung (or slow). 

          You can argue that this should be done outside of Jenkins, but, too often, Jenkins is the only place this is visible due to frequency of execution or differences in the environment.
          h3. Currently: 
          {code:java}
          try {
            timeout(time: 1, unit: 'HOURS') {
               sh "java IntermittentlySlowProcess"
            }
          } catch (t) {
              //It's too late to, for example, send a "kill -3" to the slow/hung java process
          }
          {code}
          h3. What I'd propose

          (and I'm willing to try to make a PR if this seems reasonable):
          {code:java}
          timeout(time: 1, unit: 'HOURS', beforeKill: {
             sh "killall -3 java" //for example
          }) {
             sh "java IntermittentlySlowProcess"
          }
          {code}
          The new  "beforeKill" closure can be used for clean shutdown of complex tasks, analysis of problems, etc.

          Thoughts welcome.
          Currently, the timeout step simply kills whatever processes were launched during execution of its body, and then throws an exception.  This makes it *difficult to perform any automated debugging* on these processes, since they are killed by the time the user finds out that they are hung (or slow).   It would be nice to be able to get some information about the state of affairs before things are killed, and maybe even perform safe shutdown steps prior to kill.
          h3. Currently: 
          {code:java}
          try {
            timeout(time: 1, unit: 'HOURS') {
               sh "java IntermittentlySlowProcess"
            }
          } catch (t) {
              //It's too late to, for example, send a "kill -3" to the slow/hung java process
          }
          {code}
          h3. What I'd propose

          (and I'm willing to try to make a PR if this seems reasonable):
          {code:java}
          timeout(time: 1, unit: 'HOURS', beforeKill: {
             sh "killall -3 java" //for example
          }) {
             sh "java IntermittentlySlowProcess"
          }
          {code}
          The new  "beforeKill" closure can be used for clean shutdown of complex tasks, analysis of problems, etc.

          One workaround may be to wrap whatever you are running and trap signals, but that's ugly and error-prone (and will likely cause zombies)

          Thoughts welcome.

           

           
          Hide
          akom Alexander Komarov added a comment -

          Jesse Glick can I get some pointers on how to implement this?

          I've tried using workflow-cps-plugin (ParallelStep) for guidance, but I don't seem to be able to figure it out.  It's trivial to implement a step with a body, but how does a step take a second body as a parameter and then properly invoke it? 

          I've tried making the parameter a Closure, a CpsClosure(2), etc.  None of these options seem right.  Do I need to implement a custom context and something similar to newBodyInvoker(BodyReference)? Then how do I convert my Closure to a BodyReference?

          Anyway, any help would be appreciated (or a reason why this is a dumb idea).

          Show
          akom Alexander Komarov added a comment - Jesse Glick can I get some pointers on how to implement this? I've tried using workflow-cps-plugin (ParallelStep) for guidance, but I don't seem to be able to figure it out.  It's trivial to implement a step with a body, but how does a step take a second body as a parameter and then properly invoke it?  I've tried making the parameter a Closure, a CpsClosure(2), etc.  None of these options seem right.  Do I need to implement a custom context and something similar to  newBodyInvoker(BodyReference) ? Then how do I convert my Closure to a BodyReference? Anyway, any help would be appreciated (or a reason why this is a dumb idea).
          Hide
          jglick Jesse Glick added a comment -

          Before you dive into things which are somewhere between ridiculously hard and perhaps impossible—why are you doing this? If you just want a thread dump from a Java program before it gets killed, you can do that with a few lines of Bash, with no changes to your Pipeline script or special Jenkins knowledge. trap TERM and kill -QUIT the Java PID before killing it for good.

          Show
          jglick Jesse Glick added a comment - Before you dive into things which are somewhere between ridiculously hard and perhaps impossible—why are you doing this? If you just want a thread dump from a Java program before it gets killed, you can do that with a few lines of Bash, with no changes to your Pipeline script or special Jenkins knowledge. trap TERM and kill -QUIT the Java PID before killing it for good.
          Hide
          akom Alexander Komarov added a comment -

          Jesse Glick thanks for the feedback.  I have thought of using a trap wrapper but it just seemed like an additional layer of hackery and possible future debugging effort (it's also not cross-platform).  Perhaps I can wrap it in a groovy script or java instead, or something along these lines.

          Since you say that it's too hard to do what I planned, I'll hold off.  Feel free to close, although I'm curious if anyone else is interested in the same functionality (doesn't seem like it so far).

          Show
          akom Alexander Komarov added a comment - Jesse Glick thanks for the feedback.  I have thought of using a trap wrapper but it just seemed like an additional layer of hackery and possible future debugging effort (it's also not cross-platform).  Perhaps I can wrap it in a groovy script or java instead, or something along these lines. Since you say that it's too hard to do what I planned, I'll hold off.  Feel free to close, although I'm curious if anyone else is interested in the same functionality (doesn't seem like it so far).
          Hide
          akom Alexander Komarov added a comment -

          Also, it appears to be impossible to trap the SIGTERM that Jenkins sends using bash (the trap works, but the subprocess, eg gradle, is simultaneously killed anyway).

          Show
          akom Alexander Komarov added a comment - Also, it appears to be impossible to trap the SIGTERM that Jenkins sends using bash (the trap works, but the subprocess, eg gradle, is simultaneously killed anyway).
          Hide
          jglick Jesse Glick added a comment -

          Well, it is cross-platform to the extent that the Bourne shell is. I do not know offhand what the Powershell equivalent would be.

          Not sure why you would be having an issue with the subprocess being killed. Try unsetting the JENKINS_SERVER_COOKIE environment variable and see if that helps. It should not be necessary because of this, but it has been a long time since I wrote this.

          Show
          jglick Jesse Glick added a comment - Well, it is cross-platform to the extent that the Bourne shell is. I do not know offhand what the Powershell equivalent would be. Not sure why you would be having an issue with the subprocess being killed. Try unsetting the JENKINS_SERVER_COOKIE environment variable and see if that helps. It should not be necessary because of this , but it has been a long time since I wrote this.
          Hide
          akom Alexander Komarov added a comment - - edited

          It appears to be the way Bash works.  The very first example here shows that although it is possible to trap the signal and prevent the bash shell from exiting, it is not possible to prevent the currently executing shell command (sleep in this case) from also receiving the signal, hence the while loop.  

          I was able to create a groovy wrapper script using the approach suggested here and that works correctly (both options in this answer work).

          UPDATE: the trick with bash is to background the main process... 

          Show
          akom Alexander Komarov added a comment - - edited It appears to be the way Bash works.  The very first example here shows that although it is possible to trap the signal and prevent the bash shell from exiting, it is not possible to prevent the currently executing shell command (sleep in this case) from also receiving the signal, hence the while loop.   I was able to create a groovy wrapper script using the approach  suggested here  and that works correctly (both options in this answer work). UPDATE: the trick with bash is to background the main process ... 
          Hide
          akom Alexander Komarov added a comment - - edited

          Summary of findings:

          I was able to work around this using a few different methods: <-- Link

          • A shell script that traps and cleans up before kill (not cross-platform)
          • A groovy script that traps and cleans up before kill 
          • A pure pipeline DSL script that uses parallel closure to do the same without an external script (at the cost of uglier output)
          Show
          akom Alexander Komarov added a comment - - edited Summary of findings: I was able to work around this using a few different methods : <-- Link A shell script that traps and cleans up before kill (not cross-platform) A groovy script that traps and cleans up before kill  A pure pipeline DSL script that uses parallel closure to do the same without an external script (at the cost of uglier output)
          akom Alexander Komarov made changes -
          Resolution Won't Do [ 10001 ]
          Status Open [ 1 ] Closed [ 6 ]
          Hide
          jglick Jesse Glick added a comment -

          So if I follow correctly, Jenkins was behaving correctly to the extent that its behavior followed what Ctrl-C in a shell would do (send SIGTERM to subprocesses), and you were able to write a Bourne shell script with a little difficulty that intercepted this signal as sent to the subprocess and did something else before running a clean shutdown? If so, would you mind posting your script (or, better, filing it on stackoverflow.com)?

          I should have mentioned earlier that Java itself has a friendlier solution to this problem: shutdown hooks. A hook could do whatever you liked, including getting all stack traces or dumping threads including more sophisticated information such as locks. With such a hook inserted into your program (if you indeed control it, or can write in-process plugins for it), there is no need for any special wrapper script.

          Show
          jglick Jesse Glick added a comment - So if I follow correctly, Jenkins was behaving correctly to the extent that its behavior followed what Ctrl-C in a shell would do (send SIGTERM to subprocesses), and you were able to write a Bourne shell script with a little difficulty that intercepted this signal as sent to the subprocess and did something else before running a clean shutdown? If so, would you mind posting your script (or, better, filing it on stackoverflow.com)? I should have mentioned earlier that Java itself has a friendlier solution to this problem: shutdown hooks . A hook could do whatever you liked, including getting all stack traces or dumping threads including more sophisticated information such as locks . With such a hook inserted into your program (if you indeed control it, or can write in-process plugins for it), there is no need for any special wrapper script.
          Hide
          akom Alexander Komarov added a comment - - edited

          I'll try to make the link in my previous comment more visible.  I have 4 solutions now, with this one being the one I'm using:

          https://github.com/akomakom/jenkins-scripts/blob/master/pipeline-timeout-prekill/pipeline-timeout-prekill-pure-dsl-failfast.groovy

          The only thing I would wish to change would be to silence or reduce the waitUntil output... 

          Show
          akom Alexander Komarov added a comment - - edited I'll try to make the link in my previous comment more visible.  I have 4 solutions now, with this one being the one I'm using: https://github.com/akomakom/jenkins-scripts/blob/master/pipeline-timeout-prekill/pipeline-timeout-prekill-pure-dsl-failfast.groovy The only thing I would wish to change would be to silence or reduce the  waitUntil output... 
          Hide
          jglick Jesse Glick added a comment -

          To be clear, I strongly recommend solving this at the native process level rather than with Pipeline tricks.

          Show
          jglick Jesse Glick added a comment - To be clear, I strongly recommend solving this at the native process level rather than with Pipeline tricks.
          Hide
          akom Alexander Komarov added a comment -

          Why, specifically?  Is the DSL behavior expected to change?

          They all have downsides over the built-in that this ticket originally requests:

          • bash script won't work in Windows.  
          • groovy script fixes that, but both make quoting for a complex sub-process invocation very hard to understand (eg 'bash -c "sleep 5"') because they have to accept the commands as parameters.
          • pipeline tricks don't have either limitation, but they do add unnecessary verbosity to the output.
          Show
          akom Alexander Komarov added a comment - Why, specifically?  Is the DSL behavior expected to change? They all have downsides over the built-in that this ticket originally requests: bash script won't work in Windows.   groovy script fixes that, but both make quoting for a complex sub-process invocation very hard to understand (eg 'bash -c "sleep 5"') because they have to accept the commands as parameters. pipeline tricks don't have either limitation, but they do add unnecessary verbosity to the output.
          Hide
          jglick Jesse Glick added a comment -

          Bourne shell scripts will indeed not be available in most Windows flavors, but I assume an analogous feature is easy to write in Powershell. Or, again, depending on the Java program in question, you can intercept SIGTERM in a cross-platform manner using Java APIs. In fact, at least on Linux, you can dispense with the timeout step altogether!

          sh 'timeout 11 timeout -sQUIT 10 java -jar …/jdk5/demo/jfc/SwingSet2/SwingSet2.jar'
          

          As to why you should always prefer native solutions to Pipeline script tricks, this is a subject for innumerable conference talks. Just trust me that you are better off keeping the Pipeline script to the bare minimum necessary to orchestrate actions on the Jenkins service, boiling your actual build down to a sh './run' if at all possible.

          Show
          jglick Jesse Glick added a comment - Bourne shell scripts will indeed not be available in most Windows flavors, but I assume an analogous feature is easy to write in Powershell. Or, again, depending on the Java program in question, you can intercept SIGTERM in a cross-platform manner using Java APIs. In fact, at least on Linux, you can dispense with the timeout step altogether! sh 'timeout 11 timeout -sQUIT 10 java -jar …/jdk5/demo/jfc/SwingSet2/SwingSet2.jar' As to why you should always prefer native solutions to Pipeline script tricks, this is a subject for innumerable conference talks. Just trust me that you are better off keeping the Pipeline script to the bare minimum necessary to orchestrate actions on the Jenkins service, boiling your actual build down to a sh './run' if at all possible.
          Hide
          akom Alexander Komarov added a comment -

          Jesse Glick at your suggestion I switched from using pipeline tricks to the bash script (which I tested only locally): https://github.com/akomakom/jenkins-scripts/blob/master/pipeline-timeout-prekill/pipeline-timeout-prekill.sh

          It does not work in Jenkins.  It works fine in local testing (pkill --signal SIGTERM xxxx or Ctrl-C) but not in a Jenkins timeout step - the trap never runs, termination is instant.

          This leads me to believe that Jenkins actually sends a SIGTERM to the whole process tree right away.

          In other words, there is still no good solution aside for pipeline tricks.

          Show
          akom Alexander Komarov added a comment - Jesse Glick at your suggestion I switched from using pipeline tricks to the bash script (which I tested only locally): https://github.com/akomakom/jenkins-scripts/blob/master/pipeline-timeout-prekill/pipeline-timeout-prekill.sh It does not work in Jenkins .  It works fine in local testing (pkill --signal SIGTERM xxxx or Ctrl-C) but not in a Jenkins timeout step - the trap never runs, termination is instant. This leads me to believe that Jenkins actually sends a SIGTERM to the whole process tree right away. In other words, there is still no good solution aside for pipeline tricks.
          Hide
          akom Alexander Komarov added a comment -

          Interestingly, when I use the linux "timeout" that you mention instead of the timeout step, traps work fine... So the linux solution is a combination of the shell script and timeout:

          timeout 10 bash pipeline-timeout-prekill.sh './gradlew ....'

          Which is a lot of parts...  but OK - now I'll just need to find an equivalent for windows (we do cross-platform testing on many platforms, that's why I wanted to solve this in pipeline code).

          Show
          akom Alexander Komarov added a comment - Interestingly, when I use the linux "timeout" that you mention instead of the timeout step, traps work fine... So the linux solution is a combination of the shell script and timeout: timeout 10 bash pipeline-timeout-prekill.sh  './gradlew ....' Which is a lot of parts...  but OK - now I'll just need to find an equivalent for windows (we do cross-platform testing on many platforms, that's why I wanted to solve this in pipeline code).
          Hide
          jglick Jesse Glick added a comment -

          As I said before, yes Jenkins might be sending SIGTERM to the whole process tree, not only the entry script. This might be worked around (untested) via

          JENKINS_SERVER_COOKIE=suppress java -jar …
          

          since it uses this environment variable to identify some processes. I have forgotten the details at this point. (JENKINS-28182)

          The “good solutions” are

          • Java shutdown hooks
          • using /usr/bin/timeout
          Show
          jglick Jesse Glick added a comment - As I said before, yes Jenkins might be sending SIGTERM to the whole process tree, not only the entry script. This might be worked around (untested) via JENKINS_SERVER_COOKIE=suppress java -jar … since it uses this environment variable to identify some processes. I have forgotten the details at this point. ( JENKINS-28182 ) The “good solutions” are Java shutdown hooks using /usr/bin/timeout
          Hide
          jglick Jesse Glick added a comment -

          And no you do not need a separate pipeline-timeout-prekill.sh if you are using /usr/bin/timeout. Look at my example again. That one-liner sends SIGQUIT after ten seconds, then waits one more second for the thread dump to appear, and sends a SIGTERM.

          Show
          jglick Jesse Glick added a comment - And no you do not need a separate pipeline-timeout-prekill.sh if you are using /usr/bin/timeout . Look at my example again. That one-liner sends SIGQUIT after ten seconds, then waits one more second for the thread dump to appear, and sends a SIGTERM .
          Hide
          akom Alexander Komarov added a comment -

          Thanks, I did see that, but a "kill -3" on the main gradle process isn't going to help me - I need to get the thread dump from the hung tests which are running a separate process launched by gradle.  My current approach is to run "jstack" on every java process on the slave (we have a single executor policy)

          I also just tried setting JENKINS_SERVER_COOKIE on the child process and that doesn't help with killing subprocesses, but I haven't had time to research further.  /usr/bin/timeout does work in combination with my trap script.

          Show
          akom Alexander Komarov added a comment - Thanks, I did see that, but a "kill -3" on the main gradle process isn't going to help me - I need to get the thread dump from the hung tests which are running a separate process launched by gradle.  My current approach is to run "jstack" on every java process on the slave (we have a single executor policy) I also just tried setting JENKINS_SERVER_COOKIE on the child process and that doesn't help with killing subprocesses, but I haven't had time to research further.  /usr/bin/timeout does work in combination with my trap script.
          Hide
          jglick Jesse Glick added a comment -

          get the thread dump from the hung tests which are running a separate process launched by gradle

          If that is all you wanted, you may be barking up the wrong tree. The JUnit Timeout rule, for example, applies a per-test-case timeout (which is likely to be more robust and easier to manage that a per-build timeout) and automatically displays a thread dump for hung tests.

          Show
          jglick Jesse Glick added a comment - get the thread dump from the hung tests which are running a separate process launched by gradle If that is all you wanted, you may be barking up the wrong tree. The JUnit Timeout rule , for example, applies a per-test-case timeout (which is likely to be more robust and easier to manage that a per-build timeout) and automatically displays a thread dump for hung tests.
          Hide
          akom Alexander Komarov added a comment -

          Sadly that's not all I'm after. In the case of complex integration tests, the tests may be hanging due to a hung external process (ie a server) started as part of the test. That is the process I need the stacktrace for, not the test. Otherwise, Timeout rule would work great.

          Show
          akom Alexander Komarov added a comment - Sadly that's not all I'm after. In the case of complex integration tests, the tests may be hanging due to a hung external process (ie a server) started as part of the test. That is the process I need the stacktrace for, not the test. Otherwise, Timeout rule would work great.
          Hide
          jglick Jesse Glick added a comment -

          In the case of JUnit, you would probably want to write a simple TestRule (if one does not already exist) which is responsible for launching the external process, sending it SIGQUIT if the test fails, and finally sending in SIGTERM in all cases.


          To your original proposal, if you really have to do this inside Pipeline script for some reason, try something like

          timeout(time: 1, unit: 'HOURS') {
            def ok = false
            try {
              sh 'java IntermittentlySlowProcess'
              ok = true
            } finally {
              if (!ok) {
                sh 'killall -3 java'
              }
            }
          }
          
          Show
          jglick Jesse Glick added a comment - In the case of JUnit, you would probably want to write a simple TestRule (if one does not already exist) which is responsible for launching the external process, sending it SIGQUIT if the test fails, and finally sending in SIGTERM in all cases. To your original proposal, if you really have to do this inside Pipeline script for some reason, try something like timeout(time: 1, unit: 'HOURS' ) { def ok = false try { sh 'java IntermittentlySlowProcess' ok = true } finally { if (!ok) { sh 'killall -3 java' } } }
          Hide
          akom Alexander Komarov added a comment - - edited

          Thanks Jesse Glick for the code snippet, but it doesn't appear to work (Java is killed before finally executes).  Here is a complete job to reproduce:

          node('linux') {
                  writeFile file: 'Hang.java', text: '''
          public class Hang {
              public static void main(String[] args) throws InterruptedException {
                  System.out.println("Start java sleep");
                  Thread.sleep(60000);
                  System.out.println("Finished java sleep");
              }
          }
              '''
                  sh 'javac Hang.java'
                  timeout(time: 2, unit: 'SECONDS') {
                      try {
                          sh 'java Hang'
                      } finally {
                          echo "Timeout handler"
                          sh 'ps aux | grep java'
                          echo "Timeout handler done"
                      }
                  }
              }
          

          Output:

          + javac Hang.java
          [Pipeline] timeout
          Timeout set to expire in 2 sec
          [Pipeline] {
          [Pipeline] sh
          + java Hang
          Start java sleep
          Cancelling nested steps due to timeout
          Sending interrupt signal to process
          script returned exit code 143
          [Pipeline] echo
          Timeout handler

          Now, the reasons I'm bringing this up again are:

          1. I'd like to use the inactivity timeout mode.  If I have to use a native wrapper script instead of the pipeline timeout{} step, then I'd have to reinvent inactivity implementation.
          2. I noticed that build-timeout-plugin now supports my exact use case (in Freestyle jobs).  

           

           UPDATE: I implemented "inactivity" mode in my executable groovy script equivalent of the Jenkins "timeout" as a workaround.

          Show
          akom Alexander Komarov added a comment - - edited Thanks Jesse Glick for the code snippet, but it doesn't appear to work ( Java is killed before finally executes ).  Here is a complete job to reproduce: node( 'linux' ) { writeFile file: 'Hang.java' , text: ''' public class Hang { public static void main( String [] args) throws InterruptedException { System .out.println( "Start java sleep" ); Thread .sleep(60000); System .out.println( "Finished java sleep" ); } } ''' sh 'javac Hang.java' timeout(time: 2, unit: 'SECONDS' ) { try { sh 'java Hang' } finally { echo "Timeout handler" sh 'ps aux | grep java' echo "Timeout handler done" } } } Output: + javac Hang.java [Pipeline] timeout Timeout set to expire in 2 sec [Pipeline] { [Pipeline] sh + java Hang Start java sleep Cancelling nested steps due to timeout Sending interrupt signal to process script returned exit code 143 [Pipeline] echo Timeout handler Now, the reasons I'm bringing this up again are: I'd like to use the  inactivity timeout mode.  If I have to use a native wrapper script instead of the pipeline  timeout{} step, then I'd have to reinvent inactivity implementation. I noticed that  build-timeout-plugin  now supports my exact use case (in Freestyle jobs).       UPDATE : I implemented "inactivity" mode in my executable groovy script equivalent  of the Jenkins "timeout" as a  workaround .
          Hide
          jglick Jesse Glick added a comment -

          Java is killed before finally executes

          Well, true.

          This is better handled either with a native wrapper, or with something like https://docs.oracle.com/javase/8/docs/api/java/lang/Runtime.html#addShutdownHook-java.lang.Thread- in the case of the JVM.

          Show
          jglick Jesse Glick added a comment - Java is killed before finally executes Well, true. This is better handled either with a native wrapper, or with something like https://docs.oracle.com/javase/8/docs/api/java/lang/Runtime.html#addShutdownHook-java.lang.Thread- in the case of the JVM.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            akom Alexander Komarov
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: