Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34256

Preparing Jenkins For Shutdown Hangs Running Pipelines

XMLWordPrintable

    • workflow-cps 2.78

      Start a couple long-running pipelines with

      node {
      sleep 100
      }

      Queue up a few more jobs. Go to "manage jenkins" and "prepare for shutdown."

      Now pipeline jobs that would finish and unenqueue never finish and have to manually be killed (which does work). Freestyle jobs complete normally. Queued jobs aren't run, so that part of prepare-for-shutdown works.

      Even stranger: upon killing and restarting with Ctrl+C, we get this lovely conundrum:

      Those pipeline builds won't show up in the build queue on the main screen.

      Checks to do:

      • Regression in core?
      • Regression in pipeline?
      • does /safeRestart or /restart trigger it?

        1. Screen Shot 2016-04-14 at 3.45.11 PM.png
          Screen Shot 2016-04-14 at 3.45.11 PM.png
          209 kB

          [JENKINS-34256] Preparing Jenkins For Shutdown Hangs Running Pipelines

          Sam Van Oort created issue -

          Spike Washburn added a comment -

          Sam will investigate if this is a 2.0 regression, which would increase priority.

          Spike Washburn added a comment - Sam will investigate if this is a 2.0 regression, which would increase priority.
          Sam Van Oort made changes -
          Description Original: Start a couple long-running pipelines with

          node {
             sleep 100
          }

          Queue up a few more jobs. Go to "manage jenkins" and "prepare for shutdown."

          Now pipeline jobs that would finish and unenqueue *never* finish and have to manually be killed (which does work). Freestyle jobs complete normally. Queued jobs aren't run, so that part of prepare-for-shutdown works.

          Even stranger: upon killing and restarting with Ctrl+C, we get this lovely conundrum:
          !Screen Shot 2016-04-14 at 3.45.11 PM.png|thumbnail!

          Those pipeline builds won't show up in the build queue on the main screen.

          		New: Start a couple long-running pipelines with

          node {
             sleep 100
          }

          Queue up a few more jobs. Go to "manage jenkins" and "prepare for shutdown."

          Now pipeline jobs that would finish and unenqueue *never* finish and have to manually be killed (which does work). Freestyle jobs complete normally. Queued jobs aren't run, so that part of prepare-for-shutdown works.

          Even stranger: upon killing and restarting with Ctrl+C, we get this lovely conundrum:
          !Screen Shot 2016-04-14 at 3.45.11 PM.png|thumbnail!

          Those pipeline builds won't show up in the build queue on the main screen.

          Checks to do:
          - Regression in core?
          - Regression in pipeline?
          - does /safeRestart or /restart trigger it?

          Sam Van Oort added a comment -

          Alright, I'm stumped now. I can't reproduce the hung pipelines no matter what I try now, even though it was consistent before. The issues with the weird flashing pipeline results afterward is probably due to the standing issue with queues, so... I'm going to close this as "cannot reproduce" for now, unless it recurs.

          Sam Van Oort added a comment - Alright, I'm stumped now. I can't reproduce the hung pipelines no matter what I try now, even though it was consistent before. The issues with the weird flashing pipeline results afterward is probably due to the standing issue with queues, so... I'm going to close this as "cannot reproduce" for now, unless it recurs.

          Sam Van Oort added a comment -

          Can't reproduce the hung jobs for whatever reason now, even though it was consistent before. The flashing and bizarre UI results with pipeline probably restate to the standing queue bugs, so I'm closing this unless it recurs.

          Sam Van Oort added a comment - Can't reproduce the hung jobs for whatever reason now, even though it was consistent before. The flashing and bizarre UI results with pipeline probably restate to the standing queue bugs, so I'm closing this unless it recurs.
          Sam Van Oort made changes -
          Assignee Original: Jesse Glick [ jglick ]
          Resolution New: Cannot Reproduce [ 5 ]
          Status Original: Open [ 1 ] New: Closed [ 6 ]

          Daniel Ruoso added a comment - - edited

          I can reproduce this issue... If I have several jobs like that scheduled, they show up as "part of ..." in the queue. The jobs get stuck waiting to be scheduled, but since jenkins is marked for shutdown they never get scheduled.

          The status of the node, however, never goes back to "idle", so it's never clear when is it safe to shutdown the master.

          Daniel Ruoso added a comment - - edited I can reproduce this issue... If I have several jobs like that scheduled, they show up as "part of ..." in the queue. The jobs get stuck waiting to be scheduled, but since jenkins is marked for shutdown they never get scheduled. The status of the node, however, never goes back to "idle", so it's never clear when is it safe to shutdown the master.
          Daniel Ruoso made changes -
          Resolution Original: Cannot Reproduce [ 5 ]
          Status Original: Closed [ 6 ] New: Reopened [ 4 ]

          Daniel Ruoso added a comment -

          I suspect the difference is related to the fact that the jobs show up in the queue as "part of ..." as opposed of just the job name...

          When I look at the steps of the pipeline, it shows up as:

          "
          Start of Pipeline Success
          Allocate node : Start
          "

          and the "allocate node: start" has the status as 'running', which seems to make the node behave as if it was not idle.

          Daniel Ruoso added a comment - I suspect the difference is related to the fact that the jobs show up in the queue as "part of ..." as opposed of just the job name... When I look at the steps of the pipeline, it shows up as: " Start of Pipeline Success Allocate node : Start " and the "allocate node: start" has the status as 'running', which seems to make the node behave as if it was not idle.
          Daniel Ruoso made changes -
          Summary Original: Preparing Jenkins 2 For Shutdown Hangs Running Pipelines New: Preparing Jenkins For Shutdown Hangs Running Pipelines

            dnusbaum Devin Nusbaum
            svanoort Sam Van Oort
            Votes:
            44 Vote for this issue
            Watchers:
            70 Start watching this issue

              Created:
              Updated:
              Resolved: