Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-28183

Hard killed job's stage blocks stage in following jobs

    XMLWordPrintable

Details

    Description

      Build #480 of my job hung in Stage 'Imaging' and I had to do a hard kill using BUILD_URL/doDelete

      Now all builds of my job hang on:

      Running: Imaging
      Entering stage Imaging
      Waiting for builds [480]

      Restarting Jenkins does not help.

      Attachments

        Issue Links

          depends on

          New Feature - A new feature of the product, which has yet to be developed. JENKINS-26107 Allow stage to operate as a labelled block

          • Major - Major loss of function.
          • Closed
          is blocking

          New Feature - A new feature of the product, which has yet to be developed. JENKINS-25550 Hard kill

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-29754 stage with concurrency :1 can hang forever.

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-36479 Locked resources not freed up by Pipeline job hard kill

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-34021 NPE: Failed to abort CpsFlowExecutiontask.DurableTaskStep$Execution.stop(DurableTaskStep.java:140)

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-34517 Pipeline is unable to locate global libraries when build attempts to continue on restart

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. JENKINS-33761 Ability to disable Pipeline durability and "resume" build.

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. JENKINS-32691 Hard to find "Click here to forcibly terminate running steps"

          • Blocker - Blocks development and/or testing work, production could not run.
          • Resolved
          links to

          Web Link CloudBees Internal OSS-762

          Activity

            Ascending order - Click to sort in descending order
            View 21 older comments
            abayer Andrew Bayer added a comment -

            Once JENKINS-26107 is released, stage concurrency will be deprecated. So this won't actually get fixed - instead, the recommendation will be to use lockable-resources, which is getting a fix for at least some of this scenario over at JENKINS-36479.

            abayer Andrew Bayer added a comment - Once JENKINS-26107 is released, stage concurrency will be deprecated. So this won't actually get fixed - instead, the recommendation will be to use lockable-resources , which is getting a fix for at least some of this scenario over at JENKINS-36479 .
            recampbell Ryan Campbell added a comment -

            As per the discussion above, this issue will not be fixed. The concurrency option of the stage step has been deprecated. Instead, users are advised to use the lock step of the Lockable Resource plugin.

            recampbell Ryan Campbell added a comment - As per the discussion above, this issue will not be fixed. The concurrency option of the stage step has been deprecated. Instead, users are advised to use the lock step of the Lockable Resource plugin .
            sag47 Sam Gleske added a comment -

            For those who encounter this issue and DON'T want to restart your Jenkins instance. This can be cleaned up via script console.

            import jenkins.model.Jenkins
import org.jenkinsci.plugins.workflow.job.WorkflowRun
import org.jenkinsci.plugins.workflow.support.steps.StageStepExecution

jobByFullName = 'folder/job'
jobBuildNumber = '3'

//kill it
Jenkins j = Jenkins.instance
WorkflowRun b = j.getItemByFullName(jobByFullName).getBuild(jobBuildNumber)
b.doKill()
StageStepExecution.exit(b)
            sag47 Sam Gleske added a comment - For those who encounter this issue and DON'T want to restart your Jenkins instance. This can be cleaned up via script console. import jenkins.model.Jenkins import org.jenkinsci.plugins.workflow.job.WorkflowRun import org.jenkinsci.plugins.workflow.support.steps.StageStepExecution jobByFullName = 'folder/job' jobBuildNumber = '3' //kill it Jenkins j = Jenkins.instance WorkflowRun b = j.getItemByFullName(jobByFullName).getBuild(jobBuildNumber) b.doKill() StageStepExecution.exit(b)
            jgrant216 Jeff G added a comment -

            sag47, thank you for that script.

            Latest pipeline plugins on current LTS and I had a pair of multi-branch pipeline jobs stuck between the master and slave assignment.  Even a restart caused the jobs to resume from where they had stalled, but still not continue and still not respond to abort requests (no force kill showed up either).  The script you provided allowed me to kill those and the subsequent builds worked correctly.

            jgrant216 Jeff G added a comment - sag47 , thank you for that script. Latest pipeline plugins on current LTS and I had a pair of multi-branch pipeline jobs stuck between the master and slave assignment.  Even a restart caused the jobs to resume from where they had stalled, but still not continue and still not respond to abort requests (no force kill showed up either).  The script you provided allowed me to kill those and the subsequent builds worked correctly.
            sag47 Sam Gleske added a comment -

            jgrant216, glad it helped. I have other kill-all-*.groovy scripts which make it even easier. Refer to https://github.com/samrocketman/jenkins-script-console-scripts

            sag47 Sam Gleske added a comment - jgrant216 , glad it helped. I have other kill-all-*.groovy scripts which make it even easier. Refer to https://github.com/samrocketman/jenkins-script-console-scripts

            People

              jglick Jesse Glick
              anshuarya Anshu Arya
              Votes:
              7 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: