Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40936

Milestone should be able to abort older builds when reaching the same milestone

      When using a deployment strategy with an user input option older builds are cancelled as soon as the user acknowledges the input. That's working pretty well, however:
      Frequent commits will clock up the executors all waiting on the same input if the user does not acknowledge 'quick' enough. We might be able to optimise this and free executors by automatically aborting old builds which are all sitting in the same milestone.

      build #1

      • milestone 1
      • input
        -milestone 2

      build #2 (newer commit)

      • milestone 1
      • input
      • milestone 2

      When build #2 reaches the input it's effectively in milestone 1, where #1 is currently sitting as well. It could be very good if at this point we would be able to abort #1 as we have a new build in the same milestone.

      I was not able to find a solution achieving above behaviour, but if that's otherwise possible, I would be very happy to implement it, although I believe it would be a good addition to excellent milestone pattern readily available.

          [JENKINS-40936] Milestone should be able to abort older builds when reaching the same milestone

          Patrick Wolf added a comment -

          Depending on how many commits and runs are stacked up the lock step from the lockable resources plugin could help here:

          milestone()
          lock(resource: 'my-pipeline', inversePrecedence: true) {
            milestone()
            input "proceed?"
          }
          

          In this case all of the builds will still stack up at the lock but they will get aborted much quicker. Assume you have 6 commits:

          Build 1 will enter the lock and wait at the input. All other builds after that will wait at the lock.

          When Build 1 passes the input the newest build waiting at the lock (say Build 5) enters the lock, passes the new milestone and waits at the input. As soon as Build 5 passes the new milestone all other builds (Builds 2,3,4) are aborted while Build 5 waits at the input

          This doesn't eliminate the queue but it can dramatically reduce it if you have several builds waiting.

          Patrick Wolf added a comment - Depending on how many commits and runs are stacked up the lock step from the lockable resources plugin could help here: milestone() lock(resource: 'my-pipeline' , inversePrecedence: true ) { milestone() input "proceed?" } In this case all of the builds will still stack up at the lock but they will get aborted much quicker. Assume you have 6 commits: Build 1 will enter the lock and wait at the input . All other builds after that will wait at the lock . When Build 1 passes the input the newest build waiting at the lock (say Build 5) enters the lock , passes the new milestone and waits at the input . As soon as Build 5 passes the new milestone all other builds (Builds 2,3,4) are aborted while Build 5 waits at the input This doesn't eliminate the queue but it can dramatically reduce it if you have several builds waiting.

          jeroen_muis added a comment -

          Hi Patrick,

          Thanks for the suggestion, but unfortunately it won't help enough as the number of testers against the number of commits simply doesn't stack up, and they won't be testing the 'latest and greatest'.

          Best regards,
          Jeroen

          jeroen_muis added a comment - Hi Patrick, Thanks for the suggestion, but unfortunately it won't help enough as the number of testers against the number of commits simply doesn't stack up, and they won't be testing the 'latest and greatest'. Best regards, Jeroen

          Jason Wen added a comment -

          +1 on this improvement. I am going to file same ER and find this issue has already been filed.

          The use case that Jeroen explains is exactly same use case I have. I think this could be a typical use case to create a deployment pipeline involving manual approval of promoting deployment from QA to staging and from staging to production environment.

          The use case is a newer build reach the same milestone that an old build is waiting at for manual confirmation. In this case, the new build has done all the things the old build did, conceptually it can be regarded as 'passed' the old build, so the old build should be aborted.

          I would suggest adding an optional parameter to support this behavior, e.g
          abortSameMilestone: If the value is true the older builds waiting at a milestone will be aborted when a new build reach the same milestone. The default value is false.

          I am also happy to implement this ER if it is approved.

          Jason Wen added a comment - +1 on this improvement. I am going to file same ER and find this issue has already been filed. The use case that Jeroen explains is exactly same use case I have. I think this could be a typical use case to create a deployment pipeline involving manual approval of promoting deployment from QA to staging and from staging to production environment. The use case is a newer build reach the same milestone that an old build is waiting at for manual confirmation. In this case, the new build has done all the things the old build did, conceptually it can be regarded as 'passed' the old build, so the old build should be aborted. I would suggest adding an optional parameter to support this behavior, e.g abortSameMilestone: If the value is true the older builds waiting at a milestone will be aborted when a new build reach the same milestone. The default value is false. I am also happy to implement this ER if it is approved.

          Jesse Glick added a comment -

          Frequent commits will clock up the executors all waiting on the same input

          Do not use input inside node (in most cases doing so is a bad idea!) and there is no real problem: there will be a bunch of old builds “running”, but there is no cost to them—Jenkins is idle. As soon as someone approves the newest build (or one of the newer ones), the previous ones will get aborted automatically.

          The downside to eagerly aborting older builds as soon as you get to the prompt in a newer build is that it does not guarantee progress can be made: someone looks at a build pending input, starts to do some sanity checks, and then it is canceled in favor of a new one before they can approve, and the cycle repeats. This may or may not be a problem, depending on how long manual testing takes, and how frequently fresh builds appear.

          Jesse Glick added a comment - Frequent commits will clock up the executors all waiting on the same input Do not use input inside node (in most cases doing so is a bad idea!) and there is no real problem: there will be a bunch of old builds “running”, but there is no cost to them—Jenkins is idle. As soon as someone approves the newest build (or one of the newer ones), the previous ones will get aborted automatically. The downside to eagerly aborting older builds as soon as you get to the prompt in a newer build is that it does not guarantee progress can be made: someone looks at a build pending input, starts to do some sanity checks, and then it is canceled in favor of a new one before they can approve, and the cycle repeats. This may or may not be a problem, depending on how long manual testing takes, and how frequently fresh builds appear.

          This issue would be really appreciate. I tried to find a workaround without success...

          It looks like the issue is also that an input step cannot be aborted by a milestone trigger.

          In facts, i tried, for testing purpose, to put before my input step this :

           

          milestone(BUILD_NUMBER) 

           

          But the previous build waiting in input step is never being cancelled.

          Mathieu Durand added a comment - This issue would be really appreciate. I tried to find a workaround without success... It looks like the issue is also that an input step cannot be aborted by a milestone trigger. In facts, i tried, for testing purpose, to put before my input step this :   milestone(BUILD_NUMBER)    But the previous build waiting in input step is never being cancelled.

          Rupert Madden-Abbott added a comment - - edited

          If you only ever want the latest build to be considered at the input step, then could you just do this:

          milestone()
          milestone()
          input "proceed?"
          

          The first build will stop at the input step.

          The second build will pass both milestones, aborting the first build, and then stop at the input step.

          Edit: Okay no this doesn't work and I misunderstood how this step works.

          In case it helps other, this doesn't work because the first and second build still pass both milestones in order. When the second build passes the second milestone, the first build has already passed it. Therefore, the second build has not passed a milestone that the first one has not passed, and so the first build is not aborted.

          Rupert Madden-Abbott added a comment - - edited If you only ever want the latest build to be considered at the input step, then could you just do this: milestone() milestone() input "proceed?" The first build will stop at the input step. The second build will pass both milestones, aborting the first build, and then stop at the input step. Edit : Okay no this doesn't work and I misunderstood how this step works. In case it helps other, this doesn't work because the first and second build still pass both milestones in order. When the second build passes the second milestone, the first build has already passed it. Therefore, the second build has not passed a milestone that the first one has not passed, and so the first build is not aborted.

          This works for me:

           

          stage ('1st Stage') {
            milestone ()
            echo "Stage1"
            sh 'sleep 1'
          }

          stage ('2nd Stage') {
            milestone ()
            echo "Stage 2"
            sh 'sleep 1'
          }

          stage ('3rd Stage') {
            for (int i = 0; i < (BUILD_NUMBER as int); i++) {milestone()}
            input ("Proceed")
            node ('javaAgent') {
              echo "something else to do"
              sh 'sleep 1'
            }

            milestone()

          }

          In Stage 3, prior to the input step, loop through all the past build numbers to set the milestone.  Newer builds will have a higher build number and should cancel out previous builds.

          Jonathan David added a comment - This works for me:   stage ('1st Stage') {   milestone ()   echo "Stage1"   sh 'sleep 1' } stage ('2nd Stage') {   milestone ()   echo "Stage 2"   sh 'sleep 1' } stage ('3rd Stage') {   for (int i = 0; i < (BUILD_NUMBER as int); i++) {milestone()}   input ("Proceed")   node ('javaAgent') {     echo "something else to do"     sh 'sleep 1'   }   milestone() } In Stage 3, prior to the input step, loop through all the past build numbers to set the milestone.  Newer builds will have a higher build number and should cancel out previous builds.

          Konstantin Tkachenko added a comment - - edited

          I've tried to use another suggestion with milestone (https://support.cloudbees.com/hc/en-us/articles/360034881371-How-can-I-abort-a-running-Pipeline-build-if-a-new-one-is-started-), which works great for cases without input, but it hasn't helped me:

          def getMilestoneNumber(){
            def milestoneNumber = env.BUILD_NUMBER as int
            if (milestoneNumber > 1) {
              milestoneNumber = milestoneNumber - 1
            }
            return milestoneNumber
          }
          pipeline{
          ...
              stage('Deploy') {
                when { branch 'master'}
                stages {
                  stage('Confirm'){
                    steps {
                      milestone getMilestoneNumber()
                      input(message: "Should we deploy?", id: "deploy_to_tst")
                      milestone(ordinal: getMilestoneNumber() + 1, label: "deployment confirmed")
                    }
                  }
                ...

          The problem with scripting-solution using Jenkins.instance. or currentBuild.rawBuild() are also not acceptable because of security impacts.

          Konstantin Tkachenko added a comment - - edited I've tried to use another suggestion with milestone ( https://support.cloudbees.com/hc/en-us/articles/360034881371-How-can-I-abort-a-running-Pipeline-build-if-a-new-one-is-started- ), which works great for cases without input, but it hasn't helped me: def getMilestoneNumber(){ def milestoneNumber = env.BUILD_NUMBER as int if (milestoneNumber > 1) { milestoneNumber = milestoneNumber - 1 } return milestoneNumber } pipeline{ ... stage( 'Deploy' ) { when { branch 'master' } stages { stage( 'Confirm' ){ steps { milestone getMilestoneNumber() input(message: "Should we deploy?" , id: "deploy_to_tst" ) milestone(ordinal: getMilestoneNumber() + 1, label: "deployment confirmed" ) } } ... The problem with scripting-solution using Jenkins.instance. or currentBuild.rawBuild() are also not acceptable because of security impacts.

            amuniz Antonio Muñiz
            jeroen_muis jeroen_muis
            Votes:
            6 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated: