Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60434

"Prepare for shutdown" should continue executing already running pipelines to completion

    XMLWordPrintable

Details

    Description

      Based on dnusbaum's comment from JENKINS-34256:

      A fix for this issue was just released in Pipeline: Groovy Plugin version 2.78. I think there is/was some confusion as to the expected behavior (myself included!), so let me try to clarify: When Jenkins prepares for shutdown, all running Pipelines are paused, and this is the intended behavior. The unintended behavior was that if you canceled shutdown, Pipelines remained paused. This has been fixed in 2.78; Pipelines will now resume execution if shutdown is canceled. Before 2.78, you had to manually pause and unpause each Pipeline to get it to resume execution, or restart Jenkins. Additionally, preparing Jenkins for shutdown and canceling shutdown now each cause a message to be printed to Pipeline build logs indicating that the Pipeline is being paused or resumed due to shutdown so that it is easier to understand what is happening.

      Based on comments here and elsewhere, I think some users would prefer a variant of "Prepare for shutdown" in which Pipelines continue executing to completion, the same as other types of jobs like Freestyle. If that is something you want, please open a new ticket, describing your use case and the desired behavior.

      [...]

      If there is some other aspect of this issue that you would like to see addressed, or a different behavior you would prefer, please open a new ticket describing your particular use case.

      My use case is to make restarting Jenkins master to allow upgrading Jenkins core or updating Jenkins plugins easier, because now I need to do the following:

      1. wait until no pipelines are running anymore
        • which can get difficult in bigger Jenkins environments during the day i.e. normal working hours (due to steady commits triggering pipelines), but also in case there are longer lasting test suites that e.g. are triggered all around the clock
      2. click "prepare for shutdown"
      3. ... (continue normal work like upgrading/updating)

      Attachments

        Issue Links

          Activity

            canuck1987 Tim Brown added a comment - - edited

            Hi,

            This is not a direct solution, but hopefully will help someone. It appears that as well as Online/Offline and Disconnected, Jenkins nodes also have a Suspended state. This seems to mean that they will run their current tasks and not take on new ones. So I wrote two script console snippets to suspend and resume all nodes (can be easily extended to updated work on a subset by using `findAll` and setting a predicate, instead of using `each`):

            Suspend (Pause) all nodes

            Jenkins.instance.getNodes().each{ node ->
                def computer = node.toComputer()
                computer.setAcceptingTasks(false)
                println("${computer.getName()} accepting tasks: ${computer.isAcceptingTasks()}")
            }
            // Prevent this from dumping the list of nodes
            return null
            

            Resume (unpause) all nodes

            Jenkins.instance.getNodes().each{ node ->
                def computer = node.toComputer()
                computer.setAcceptingTasks(true)
                println("${computer.getName()} accepting tasks: ${computer.isAcceptingTasks()}")
            }
            // Prevent this from dumping the list of nodes
            return null
            

            Note: if this works for this use-case it should be trivial to add as an option the Management page (e.g. a toggle button to Suspend Nodes, resume Nodes).

            I think this will be useful (for us at least) in cases where we want to pause all (or a subset) of Physical nodes, e.g. for a driver update, but leave Jenkins itself running.

            We have a problem with the Restart functionality that we want to Prepare for shutdown, but then shut down when we are ready (not when safeRestart is ready).
            I have a snippet which (I think) will return the list of Jenkins Pipeline jobs that are blocking the shutdown (not Paused/SUSPENDED - not sure what the difference is but `.isPaused()` seemed to return false despite the Pipeline console stating `Pausing (Preparing for shutdown)`).

            import org.jenkinsci.plugins.workflow.job.WorkflowRun
            
            Jenkins.instance.getView('All').getBuilds().findAll()
            {
              // Get list of job that have have started (not in queue) but have not finished.
              it.getResult().equals(null)
            }.findAll() {
              // Return list of runs workflow runs that block the (safe) restart.
              it instanceof WorkflowRun && it.getExecution().blocksRestart()
            }
            

            (feel free to extend for non-Pipeline jobs).

            canuck1987 Tim Brown added a comment - - edited Hi, This is not a direct solution, but hopefully will help someone. It appears that as well as Online/Offline and Disconnected, Jenkins nodes also have a Suspended state. This seems to mean that they will run their current tasks and not take on new ones. So I wrote two script console snippets to suspend and resume all nodes (can be easily extended to updated work on a subset by using `findAll` and setting a predicate, instead of using `each`): Suspend (Pause) all nodes Jenkins.instance.getNodes().each{ node -> def computer = node.toComputer() computer.setAcceptingTasks( false ) println( "${computer.getName()} accepting tasks: ${computer.isAcceptingTasks()}" ) } // Prevent this from dumping the list of nodes return null Resume (unpause) all nodes Jenkins.instance.getNodes().each{ node -> def computer = node.toComputer() computer.setAcceptingTasks( true ) println( "${computer.getName()} accepting tasks: ${computer.isAcceptingTasks()}" ) } // Prevent this from dumping the list of nodes return null Note: if this works for this use-case it should be trivial to add as an option the Management page (e.g. a toggle button to Suspend Nodes, resume Nodes). I think this will be useful (for us at least) in cases where we want to pause all (or a subset) of Physical nodes, e.g. for a driver update, but leave Jenkins itself running. We have a problem with the Restart functionality that we want to Prepare for shutdown, but then shut down when we are ready (not when safeRestart is ready). I have a snippet which (I think) will return the list of Jenkins Pipeline jobs that are blocking the shutdown (not Paused/SUSPENDED - not sure what the difference is but `.isPaused()` seemed to return false despite the Pipeline console stating `Pausing (Preparing for shutdown)`). import org.jenkinsci.plugins.workflow.job.WorkflowRun Jenkins.instance.getView( 'All' ).getBuilds().findAll() { // Get list of job that have have started (not in queue) but have not finished. it.getResult().equals( null ) }.findAll() { // Return list of runs workflow runs that block the (safe) restart. it instanceof WorkflowRun && it.getExecution().blocksRestart() } (feel free to extend for non-Pipeline jobs).
            taylor01 David Taylor added a comment -

            Any updates on this? /safeRestart does not work correctly and it hasn't for at least 3-4 years. 

            I thought it's supposed to prevent any new jobs from starting, but allow currently running pipelines to complete. Instead it pauses currently running pipelines, which prevents them from ever finishing. I think it allows freestyle jobs to complete, but pipelines get paused when they move to the next stage.

            taylor01 David Taylor added a comment - Any updates on this? /safeRestart does not work correctly and it hasn't for at least 3-4 years.  I thought it's supposed to prevent any new jobs from starting, but allow currently running pipelines to complete. Instead it pauses currently running pipelines, which prevents them from ever finishing. I think it allows freestyle jobs to complete, but pipelines get paused when they move to the next stage.
            paybas Pay Bas added a comment -

            Yeah this has been plaguing me forever. The "Prepare for Shutdown" feature not allowing already running pipelines to continue/finish is a major p.i.t.a.

            I need a way to put Jenkins in maintenance mode at 17:00 to prevent new builds from being started, but let current ones finish so I have a clean Jenkins at 18:00 when I actually start the maintenance.

            paybas Pay Bas added a comment - Yeah this has been plaguing me forever. The "Prepare for Shutdown" feature not allowing already running pipelines to continue/finish is a major p.i.t.a. I need a way to put Jenkins in maintenance mode at 17:00 to prevent new builds from being started, but let current ones finish so I have a clean Jenkins at 18:00 when I actually start the maintenance.
            ivanrossier Ivan Rossier added a comment -

            Totally agree with last remark: "Prepare for shutdown' was really useful for maintenance  purpose.

            ivanrossier Ivan Rossier added a comment - Totally agree with last remark: "Prepare for shutdown' was really useful for maintenance  purpose.
            kastningj Jen added a comment -

            Yes, 100% agree! Please make fixing the "Prepare for shutdown" a priority!

            kastningj Jen added a comment - Yes, 100% agree! Please make fixing the "Prepare for shutdown" a priority!

            People

              Unassigned Unassigned
              reinholdfuereder Reinhold Füreder
              Votes:
              30 Vote for this issue
              Watchers:
              37 Start watching this issue

              Dates

                Created:
                Updated: