Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38316

Entering, then cancelling, quiet mode causes builds to hang

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major
    • workflow-cps-plugin
    • None
    • jenkins:latest docker image

      This problem can be reproduced with a simple pipeline job that does nothing but echo and sleep in a loop.

      1) Start a run of the pipeline job, the job produces output in the Console Output as I expect
      2) Put Jenkins in "Preparing for Shutdown" mode. The jobs stop producing output. That's somewhat surprising, as runs that were started before entering Shutdown Mode are usually allowed to complete.
      3) Cancel "Shutdown Mode." The job is still "stuck" and doesn't produce any output. The job also cannot be canceled except by a "hard kill," i.e. going through the "Forcibly terminate running steps" and "Forcibly kill entire build" links

      The behavior in #2 is surprising, but not really a problem for me. However, the fact that entering and leaving shutdown mode completely hoses running jobs is really bad. This is compounded by an init.groovy script we use that places Jenkins in shutdown mode at startup.

          [JENKINS-38316] Entering, then cancelling, quiet mode causes builds to hang

          jglick I've tested pause & resume workaround and it seems to work.

          Doing the following leads to a hanging pipeline:

          • Trigger pipeline build.
          • Put Jenkins into Quiet Down mode.
          • Cancel Quiet Down mode.

          But doing the following makes the build resume and complete as expected:

          • Trigger pipeline build.
          • Pause pipeline build.
          • Put Jenkins into Quiet Down mode.
          • Cancel Quiet Down mode.
          • Resume pipeline build.

          I've had to write a (horrible) script that pauses all pipeline builds before triggering Quiet Down mode (we do this when we back up the master configuration files) and then resumes them all again afterwards.

          Is a fix for this on the cards soon? If you give me some pointers I might be able to help with the implementation.

          Paul Brimicombe added a comment - jglick I've tested pause & resume workaround and it seems to work. Doing the following leads to a hanging pipeline: Trigger pipeline build. Put Jenkins into Quiet Down mode. Cancel Quiet Down mode. But doing the following makes the build resume and complete as expected: Trigger pipeline build. Pause pipeline build. Put Jenkins into Quiet Down mode. Cancel Quiet Down mode. Resume pipeline build. I've had to write a (horrible) script that pauses all pipeline builds before triggering Quiet Down mode (we do this when we back up the master configuration files) and then resumes them all again afterwards. Is a fix for this on the cards soon? If you give me some pointers I might be able to help with the implementation.

          Jesse Glick added a comment -

          Not sure what the fix would look like. There is no event fired when quiet mode is cancelled AFAIK, so you would need to set up some kind of polling using Timer. At any rate, sounds like it should be easy enough to write a functional test for.

          Jesse Glick added a comment - Not sure what the fix would look like. There is no event fired when quiet mode is cancelled AFAIK, so you would need to set up some kind of polling using Timer . At any rate, sounds like it should be easy enough to write a functional test for.

          Given that pipelines are now a central feature of Jenkins would we be able to add an event for cancelling quiet mode (if one doesn't exist)? I feel like there might be other processes that could be interested in this and it would make for a much cleaner solution.

          Paul Brimicombe added a comment - Given that pipelines are now a central feature of Jenkins would we be able to add an event for cancelling quiet mode (if one doesn't exist)? I feel like there might be other processes that could be interested in this and it would make for a much cleaner solution.

          mcrooney added a comment -

          Seeing this on Jenkins 2.73.2 as well. Very odd that simple pipeline jobs totally hang just by entering quiet mode. It seems to cause them to hang in an unrecoverable way. We use "Quiet Down" to restart for upgrades to Jenkins and plugins in a graceful way, but very ironic and problematic that it is behaving very ungracefully by causing any in progress pipelines to get stuck.

          mcrooney added a comment - Seeing this on Jenkins 2.73.2 as well. Very odd that simple pipeline jobs totally hang just by entering quiet mode. It seems to cause them to hang in an unrecoverable way. We use "Quiet Down" to restart for upgrades to Jenkins and plugins in a graceful way, but very ironic and problematic that it is behaving very ungracefully by causing any in progress pipelines to get stuck.

          Also a problem in Jenkins: 2.89.4, Pipeline: 2.5 

          Yasen Terziivanov added a comment - Also a problem in Jenkins: 2.89.4, Pipeline: 2.5 

          ad #2 pipeline stops/freezes instead of expected and documented (e.g. also in UI "Manage Jenkins > Prepare for Shutdown") finishing of job:

          • this is surprising and unexpected considering also the documentation, but of course makes sense as described by Jesse.
          • however, would it make sense to change that to e.g. finish execution of currently active stage like commented in JENKINS-49365 – and somehow signal that in the UI similar to current "(part)" suffix in "build executor status" to allow for gentle shutdown at well-defined points? And therefore, as mentioned in JENKINS-49365, also when docker.image(...).inside(...) is used...

          ad #3 pipeline does not wake up after canceling the shutdown mode:

          • yes, I can confirm: "pause" (also) after cancelling followed by "resume" works and would be an IMHO acceptable workaround – although I really would not mind if that would be "just" fixed, naively assuming that it is not that difficult to do?

          While (resumable, i.e. non-docker.image(...).inside(...)-dependent) pipelines resume fine after a restart, there is a remainder in "build executor status" that (after the pipeline has finished) requires the workaround to abort it (the finished pipeline) by pressing the 'x' that in turn then asks "Are you sure you want to abort null?":

          Reinhold Füreder added a comment - ad #2 pipeline stops/freezes instead of expected and documented (e.g. also in UI "Manage Jenkins > Prepare for Shutdown") finishing of job: this is surprising and unexpected considering also the documentation, but of course makes sense as described by Jesse. however, would it make sense to change that to e.g. finish execution of currently active stage like commented in JENKINS-49365 – and somehow signal that in the UI similar to current "(part)" suffix in "build executor status" to allow for gentle shutdown at well-defined points? And therefore, as mentioned in JENKINS-49365 , also when docker.image(...).inside(...) is used... ad #3 pipeline does not wake up after canceling the shutdown mode: yes, I can confirm: "pause" (also) after cancelling followed by "resume" works and would be an IMHO acceptable workaround – although I really would not mind if that would be "just" fixed, naively assuming that it is not that difficult to do? While (resumable, i.e. non- docker.image(...).inside(...) -dependent) pipelines resume fine after a restart, there is a remainder in "build executor status" that (after the pipeline has finished) requires the workaround to abort it (the finished pipeline) by pressing the 'x' that in turn then asks "Are you sure you want to abort null?":

          Sam Van Oort added a comment -

          Closing in preference to the pre-existing issue that has the same result (sigh, still not resolved yet).

          Sam Van Oort added a comment - Closing in preference to the pre-existing issue that has the same result (sigh, still not resolved yet).

          David Taylor added a comment -

          Any updates would be very appreciated. This bug is causing a lot of problems for our build system

          David Taylor added a comment - Any updates would be very appreciated. This bug is causing a lot of problems for our build system

          Rick Liu added a comment -

          This still happened to Jenkins v*2.138.4* LTS

          Rick Liu added a comment - This still happened to Jenkins v*2.138.4* LTS

          Jesse Glick added a comment -

          (incorrect resolution)

          Jesse Glick added a comment - (incorrect resolution)

            Unassigned Unassigned
            srwalter Steven Walter
            Votes:
            20 Vote for this issue
            Watchers:
            26 Start watching this issue

              Created:
              Updated:
              Resolved: