Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63942

Allow milestone step to repeat ordinals or execute within a loop

      The milestone plugin always feels on the surface like the perfect solution for many CI/CD pipelines my team works with.  The idea of aborting old pipeline instances if a new one surpasses it feels right.  In practice, however, we are often waiting for something outside of the pipeline's control; something like a scheduled release time.  This often results in some kind of polling loop or `waitUntil` step to wait for that scheduled time.

      If two executions are running, and waiting at a polling stage, the milestone step breaks down.  The ordinal value for the milestone is not allowed to be repeated, so placing the milestone step within a waitUntil or other loop causes the following error:
      ERROR: Invalid ordinal 1, as the previous one was 1
      Repeating the ordinal FEELS right in this situation, but perhaps not.  Any alternative approach I've come up with is either very complex, or doesn't resolve the problem consistently.  I've attempted a number of things including the lock step, incrementing the ordinal with a variable, and simply moving it to after the loop, and some declarative tries.  Any approach I take appears to take me further from my goal of simply aborting all earlier executions that have made it to a specific stage.

      My suggestion would be to add a new argument (perhaps `lastOrdinalRepeatable: true`).  If the milestone is passed a second time it should abort the job if any newer executions have passed (just like it would have on the first run through).

      I'm open to other approaches.  I've come back to this problem over and over and it seems like a reasonable use case.

      Attached at the end is an example pipeline describing the behavior comment in the desired milestone line for a few variations.  

      pipeline{
        agent none
        stages{
          stage('build'){
            steps
            {
                sleep time: 3, unit: 'SECONDS'
            }
          }
          stage('wait'){
            steps{
              echo "${currentBuild.timeInMillis}"
              timeout(time: 1, unit: 'DAYS') {
                waitUntil{
                  script{
                    echo "${currentBuild.timeInMillis}+${currentBuild.duration}"
                    // The following milestone step causes an error:
                    // ERROR: Invalid ordinal 1, as the previous one was 1
                    //milestone(1)
      
                    // Poor-man's logic to block till deployment time
                    if ((currentBuild.timeInMillis+currentBuild.duration)<1602694800000)
                    {
                        return false
                    }
                    return true
                  }
                }
              }
              // The following milestone step executes, but is inconsistent
              // in its results. (not all old executions are aborted)
              // it also executes the entire waitUntil, because the milestone
              // hasn't been passed till after waitUntil completes
              milestone(1)
            }
          }
          stage('deploy'){
            steps{
                echo "simulate deploy"
            }
          }
        }
      } 

       

      Below is a screen shot of that pipeline:

      #95-96 - executed concurrently with the same "scheduled time", with the milestone commented in AFTER the waitUntil block

      #98-97 - executed concurrently with the same "scheduled time", with the milestone commented in AFTER the waitUntil block - inconsistent results to previous test

      #99-100 - executed concurrently with the same "scheduled time", with the milestone commented in WITHIN the waitUntil block, and fails with the expected ordinal error

       

       

       

          [JENKINS-63942] Allow milestone step to repeat ordinals or execute within a loop

          Adam Krapfl added a comment -

          abayer jglick am I totally off-base here?  Is there another approach I'm not thinking about?  Obviously not getting any votes on this one, so it must be my perspective.  Or is milestone just kinda dead or not turning out the way we'd like?

          I'd love some advice.

          Adam Krapfl added a comment - abayer jglick  am I totally off-base here?  Is there another approach I'm not thinking about?  Obviously not getting any votes on this one, so it must be my perspective.  Or is milestone just kinda dead or not turning out the way we'd like? I'd love some advice.

          Jesse Glick added a comment -

          You cannot meaningfully use milestone inside a loop. I am not really clear what problem you are attempting to solve here. Maybe better to ask on the users’ list, stackoverflow.com, etc.

          By the way I have not maintained this plugin or anything else in Pipeline for years. I occasionally contribute small patches.

          Jesse Glick added a comment - You cannot meaningfully use milestone inside a loop. I am not really clear what problem you are attempting to solve here. Maybe better to ask on the users’ list, stackoverflow.com, etc. By the way I have not maintained this plugin or anything else in Pipeline for years. I occasionally contribute small patches.

          Adam Krapfl added a comment -

          No problem.  I see you all over so I assume you're "the guy".  I'll see if I can dig up the maintainer of the plugin.  For the most part, I'm not TRYING to use a complex structure like a loop.  All I really want is an input.  If two executions are blocked by an input, the milestone is worthless.

          I'll put some thought into a simpler example.  Perhaps what I'm really asking for is milestone to be implemented as an option at the stage level, rather than a straight-up step.

          Adam Krapfl added a comment - No problem.  I see you all over so I assume you're "the guy".  I'll see if I can dig up the maintainer of the plugin.  For the most part, I'm not TRYING to use a complex structure like a loop.  All I really want is an input.  If two executions are blocked by an input, the milestone is worthless. I'll put some thought into a simpler example.  Perhaps what I'm really asking for is milestone to be implemented as an option at the stage level, rather than a straight-up step.

          Adam Krapfl added a comment -

          amuniz Do you have any thoughts?  Are you the maintainer?  Perhaps the problem is that this plugin hasn't kept up with the times with support for declarative.

          Adam Krapfl added a comment - amuniz  Do you have any thoughts?  Are you the maintainer?  Perhaps the problem is that this plugin hasn't kept up with the times with support for declarative.

          Adam Krapfl added a comment - - edited

          Figured out how to format the code block above better.  It's still a hack, I admit.  All that code is trying to demonstrate is the following use case:

          I want to queue up a number of builds and release the latest one for deployment at a specified time.  If deployment time is scheduled for 5pm, and there are 10 builds queued up before then, I want the latest of the 10 to release, and all earlier builds to abort.  As it stands it is indeterministic which build hits the milestone first.  It will not be the newest build, as expected.

          Should I re-post this as a simple defect?  It doesn't work specifically because of the waitUntil before it.  But I believe attempted similar behavior with locks and other mechanisms to queue up several pipelines.

          Adam Krapfl added a comment - - edited Figured out how to format the code block above better.  It's still a hack, I admit.  All that code is trying to demonstrate is the following use case: I want to queue up a number of builds and release the latest one for deployment at a specified time.  If deployment time is scheduled for 5pm, and there are 10 builds queued up before then, I want the latest of the 10 to release, and all earlier builds to abort.  As it stands it is indeterministic which build hits the milestone first.  It will not be the newest build, as expected. Should I re-post this as a simple defect?  It doesn't work specifically because of the waitUntil before it.  But I believe attempted similar behavior with locks and other mechanisms to queue up several pipelines.

          > Repeating the ordinal FEELS right in this situation

          No. That would break the entire logic of the step.

          What you are experiencing is expected given that `waitUntil` step evaluates its block every 15 seconds by default. So once you have all the builds on the waiting step they are checking the condition every 15 seconds, which means that any step could check first once the timing condition is true, which explains the "random" behaviour you are explaining.

          So `milestone` step is behaving as designed, the problem here is the timing conditions and how `waitUntil` works. I don't think there is a built-in way to cover your use case, maybe the easiest is to use `input` and write something external (using the the REST API, to retrieve in-progress builds and then approve the newest input step - have a look to the wfapi). This could be a shell script running in a job in Jenkins which is triggered just at the right time.

          Not ideal, but I don't see any clear way to do this without writing some ad-hoc step/code.

          Antonio Muñiz added a comment - > Repeating the ordinal FEELS right in this situation No. That would break the entire logic of the step. What you are experiencing is expected given that `waitUntil` step evaluates its block every 15 seconds by default. So once you have all the builds on the waiting step they are checking the condition every 15 seconds, which means that any step could check first once the timing condition is true, which explains the "random" behaviour you are explaining. So `milestone` step is behaving as designed, the problem here is the timing conditions and how `waitUntil` works. I don't think there is a built-in way to cover your use case, maybe the easiest is to use `input` and write something external (using the the REST API, to retrieve in-progress builds and then approve the newest input step - have a look to the wfapi ). This could be a shell script running in a job in Jenkins which is triggered just at the right time. Not ideal, but I don't see any clear way to do this without writing some ad-hoc step/code.

          Adam Krapfl added a comment -

          I understand.  Thanks for the response.  I completely agree that blocking several pipelines till a designated time is what's causing this, and it's the interaction between that and the milestone step causing it.  And I agree, I don't have an idea of how to workaround it.

           

          This still feels like a use case that would benefit the plugin if it could be cracked.  What I would like to do is close this issue and open a new one describing my suggested new behavior more succinctly.   Perhaps the community can help come up with a solution.  Would you be OK with that amuniz ?

          Adam Krapfl added a comment - I understand.  Thanks for the response.  I completely agree that blocking several pipelines till a designated time is what's causing this, and it's the interaction between that and the milestone step causing it.  And I agree, I don't have an idea of how to workaround it.   This still feels like a use case that would benefit the plugin if it could be cracked.  What I would like to do is close this issue and open a new one describing my suggested new behavior more succinctly.   Perhaps the community can help come up with a solution.  Would you be OK with that amuniz ?

          Sure, feel free to file an issue explaining the use case, I can't say I'll have time in the short term to attend it, but maybe someone else in the community can take care of it.

          Antonio Muñiz added a comment - Sure, feel free to file an issue explaining the use case, I can't say I'll have time in the short term to attend it, but maybe someone else in the community can take care of it.

            amuniz Antonio Muñiz
            akrapfl Adam Krapfl
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: