Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-28183

Hard killed job's stage blocks stage in following jobs

      Build #480 of my job hung in Stage 'Imaging' and I had to do a hard kill using BUILD_URL/doDelete

      Now all builds of my job hang on:

      Running: Imaging
      Entering stage Imaging
      Waiting for builds [480]

      Restarting Jenkins does not help.

          [JENKINS-28183] Hard killed job's stage blocks stage in following jobs

          Sorin Sbarnea added a comment -

          Please have a look at https://issues.jenkins-ci.org/browse/JENKINS-34021 which describes the NPE issue. We already encountered this twice in the last month and I am sure that nobody did some filesystem changes before these bug was triggered.

          Sorin Sbarnea added a comment - Please have a look at https://issues.jenkins-ci.org/browse/JENKINS-34021 which describes the NPE issue. We already encountered this twice in the last month and I am sure that nobody did some filesystem changes before these bug was triggered.

          Ok. Thanks for reporting a separate issue.

          Antonio Muñiz added a comment - Ok. Thanks for reporting a separate issue.

          Had the same issue.
          Worked around it like this:

          • aborted the waiting build (forcibly)
          • renamed the blocking build's folder on the file system
          • saw that ```$JENKINS_HOME/org.jenkinsci.plugins.workflow.support.steps.StageStep.xml``` wasn't cleared because already a new build had started
          • aborted the new build, too
          • found the StageStep xml file empty and was able to restart the job.

          Dorian Daumiller added a comment - Had the same issue. Worked around it like this: aborted the waiting build (forcibly) renamed the blocking build's folder on the file system saw that ```$JENKINS_HOME/org.jenkinsci.plugins.workflow.support.steps.StageStep.xml``` wasn't cleared because already a new build had started aborted the new build, too found the StageStep xml file empty and was able to restart the job.

          Tavin Cole added a comment - - edited

          instead of renaming folders, deleting the StageStep.xml file also works (I did this with jenkins shut down)

          Tavin Cole added a comment - - edited instead of renaming folders, deleting the StageStep.xml file also works (I did this with jenkins shut down)

          Jesse Glick added a comment -

          The NullPointerException has since been fixed IIRC.

          Unlikely to be fixed since concurrency of stage is slated for deprecation. Use lock instead, and we will work on JENKINS-36479.

          Jesse Glick added a comment - The NullPointerException has since been fixed IIRC. Unlikely to be fixed since concurrency of stage is slated for deprecation. Use lock instead, and we will work on JENKINS-36479 .

          Andrew Bayer added a comment -

          Once JENKINS-26107 is released, stage concurrency will be deprecated. So this won't actually get fixed - instead, the recommendation will be to use lockable-resources, which is getting a fix for at least some of this scenario over at JENKINS-36479.

          Andrew Bayer added a comment - Once JENKINS-26107 is released, stage concurrency will be deprecated. So this won't actually get fixed - instead, the recommendation will be to use lockable-resources , which is getting a fix for at least some of this scenario over at JENKINS-36479 .

          Ryan Campbell added a comment -

          As per the discussion above, this issue will not be fixed. The concurrency option of the stage step has been deprecated. Instead, users are advised to use the lock step of the Lockable Resource plugin.

          Ryan Campbell added a comment - As per the discussion above, this issue will not be fixed. The concurrency option of the stage step has been deprecated. Instead, users are advised to use the lock step of the Lockable Resource plugin .

          Sam Gleske added a comment -

          For those who encounter this issue and DON'T want to restart your Jenkins instance. This can be cleaned up via script console.

          import jenkins.model.Jenkins
          import org.jenkinsci.plugins.workflow.job.WorkflowRun
          import org.jenkinsci.plugins.workflow.support.steps.StageStepExecution
          
          jobByFullName = 'folder/job'
          jobBuildNumber = '3'
          
          //kill it
          Jenkins j = Jenkins.instance
          WorkflowRun b = j.getItemByFullName(jobByFullName).getBuild(jobBuildNumber)
          b.doKill()
          StageStepExecution.exit(b)
          

          Sam Gleske added a comment - For those who encounter this issue and DON'T want to restart your Jenkins instance. This can be cleaned up via script console. import jenkins.model.Jenkins import org.jenkinsci.plugins.workflow.job.WorkflowRun import org.jenkinsci.plugins.workflow.support.steps.StageStepExecution jobByFullName = 'folder/job' jobBuildNumber = '3' //kill it Jenkins j = Jenkins.instance WorkflowRun b = j.getItemByFullName(jobByFullName).getBuild(jobBuildNumber) b.doKill() StageStepExecution.exit(b)

          Jeff G added a comment -

          sag47, thank you for that script.

          Latest pipeline plugins on current LTS and I had a pair of multi-branch pipeline jobs stuck between the master and slave assignment.  Even a restart caused the jobs to resume from where they had stalled, but still not continue and still not respond to abort requests (no force kill showed up either).  The script you provided allowed me to kill those and the subsequent builds worked correctly.

          Jeff G added a comment - sag47 , thank you for that script. Latest pipeline plugins on current LTS and I had a pair of multi-branch pipeline jobs stuck between the master and slave assignment.  Even a restart caused the jobs to resume from where they had stalled, but still not continue and still not respond to abort requests (no force kill showed up either).  The script you provided allowed me to kill those and the subsequent builds worked correctly.

          Sam Gleske added a comment -

          jgrant216, glad it helped. I have other kill-all-*.groovy scripts which make it even easier. Refer to https://github.com/samrocketman/jenkins-script-console-scripts

          Sam Gleske added a comment - jgrant216 , glad it helped. I have other kill-all-*.groovy scripts which make it even easier. Refer to https://github.com/samrocketman/jenkins-script-console-scripts

            jglick Jesse Glick
            anshuarya Anshu Arya
            Votes:
            7 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated:
              Resolved: