Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33846

Restart pipeline from specific stage, after failure

    • Icon: New Feature New Feature
    • Resolution: Unresolved
    • Icon: Major Major
    • pipeline
    • None

      This feature is unplanned and not actively worked on

      The use case for restarting stages of a Pipeline is an important one and as such we intend to include this feature in Declarative Pipeline (JENKINS-45455) as open source. I have put this feature on the Declarative Pipeline roadmap. I encourage you to review JENKINS-45455 and leave a comment to ensure that your use case is covered.

      For the purpose of clarity, we will not be providing this feature for scripted pipelines.

      After review of the Checkpoints feature, the CloudBees team have come to the decision that will not be open sourcing Checkpoints. I know we had signalled in this issue that we would be open sourcing this feature and I am sincerely sorry to disappoint those who have been waiting for us to do so.

      Thanks,
      James Dumay
      Product Manager, CloudBees Inc

      It would be fine, and customers ask me for that continuously to have the function to again start a failed pipeline,but not at the very beginning, rather at a stage along the pipeline chain, particularly a stage which failed before during a former run,and the cause of the failure was fixed.

      Maybe I've chosen the wrong "component" for this ticket. Please feel free to adjust accordingly.

      Tested on Jenkins ver. 2.0-beta-1, and its new Pi Pipeline item.

          [JENKINS-33846] Restart pipeline from specific stage, after failure

          Martin Ba added a comment - - edited

          jamesdumay - so ...

          scripted pipeline is extremely open ended. As I wrote on JENKINS-45455 "There is less complexity and edge cases needed to be handled by making this functionality a Declarative Pipeline feature only".

          ... I'm rather unsure how to interpret this: checkpoint is already implemented, so any reference to "less complexity" is either handwaving, or the checkpoint feature you currently want to us to pay for^[a]^ doesn't work properly.

           

          I'll note, as others, that the inability to have checkpoints in jenkins pipeline makes it pretty much useless to us. (workarounds like https://stackoverflow.com/a/40351693/321013 and other mentioned ones non-withstanding)

          ([a]: I'm fine with you trying to make a buck. It's just that I feel the checkpoint feature is not enterprisey at all - it seems vital for any non-trivial long-running pipeline, also (and maybe especially) for small teams.)

          Martin Ba added a comment - - edited jamesdumay - so ... scripted pipeline is extremely open ended. As I wrote on JENKINS-45455 "There is less complexity and edge cases needed to be handled by making this functionality a Declarative Pipeline feature only". ... I'm rather unsure how to interpret this: checkpoint is already implemented, so any reference to "less complexity" is either handwaving, or the checkpoint feature you currently want to us to pay for^ [a] ^ doesn't work properly.   I'll note, as others, that the inability to have checkpoints in jenkins pipeline makes it pretty much useless to us. (workarounds like https://stackoverflow.com/a/40351693/321013 and other mentioned ones non-withstanding) ( [a] : I'm fine with you trying to make a buck. It's just that I feel the checkpoint feature is not enterprisey at all - it seems vital for any non-trivial long-running pipeline, also (and maybe especially) for small teams.)

          Lorenz Mende added a comment -

          I have to agree Martin, the checkpoint feature seems like a vital function for pipeline workflows.

          We are running jobs which last hours to a day and are hardly to modularize further...

           

          Very disappointing decision.

          Lorenz Mende added a comment - I have to agree Martin, the checkpoint feature seems like a vital function for pipeline workflows. We are running jobs which last hours to a day and are hardly to modularize further...   Very disappointing decision.

          Tharun John added a comment -

          It is like one step forward and 2 steps backwards. I understand we all need to make money. But this is a genuine request from the community. We are a small team and don't have powerful machines to rerun tests from the beginning. So we still have not yet adopted the groovy code and still running manual pipelines views to resume jobs. Does anybody have a work around for this problem ?

           

          Tharun John added a comment - It is like one step forward and 2 steps backwards. I understand we all need to make money. But this is a genuine request from the community. We are a small team and don't have powerful machines to rerun tests from the beginning. So we still have not yet adopted the groovy code and still running manual pipelines views to resume jobs. Does anybody have a work around for this problem ?  

          Oliver Nocon added a comment - - edited

          We are using a workaround with a custom step within a pipeline library similar to this:

          //restartIt.goovy
          def call(Map parameters = [:], body) {
            def timeoutInSeconds = parameters.get('timeoutInSeconds', 900)
            def restart = true
            while (restart) {
              try {
                body()
                restart = false
              } catch (Throwable err) {
                echo "ERROR occured: ${err}"
                //optional email notification
                try {
                  timeout(time: timeoutInSeconds, unit: 'SECONDS') {
                    input message: 'Do you want to restart?', ok: 'Restart'
                  }
                } catch(e) {
                  restart = false
                  throw err
                }
              }
            }
          }

          This can the be used like:

          restartIt(timeout:1200) {
            node {
              //here the steps you want to possibly restart
            }
          }

           

          For sure, there are some caveats. For example, you should use a node to not block executors.

          Anyway, it helps us in cases where e.g. deployments sometime fail.

          Maybe it helps you as well ...

           

          Oliver Nocon added a comment - - edited We are using a workaround with a custom step within a pipeline library similar to this: //restartIt.goovy def call(Map parameters = [:] , body) {   def timeoutInSeconds = parameters.get('timeoutInSeconds', 900)   def restart = true   while (restart) {     try {       body()       restart = false     } catch (Throwable err) {       echo "ERROR occured: ${err}"       //optional email notification       try {         timeout(time: timeoutInSeconds, unit: 'SECONDS') {           input message: 'Do you want to restart?', ok: 'Restart'         }       } catch(e) {         restart = false         throw err       }     }   } } This can the be used like: restartIt(timeout:1200) {   node {     //here the steps you want to possibly restart   } }   For sure, there are some caveats. For example, you should use a node to not block executors. Anyway, it helps us in cases where e.g. deployments sometime fail. Maybe it helps you as well ...  

          very misguided decision.  our situation is that we enrolled in the paid Jenkins Team version so we are PAYING customers to Cloudbees AND still don't get access to this very basic feature.  It's really sad and unfair to both customers and the community. Feels like a ripoff

          We're looking into options from workarounds mentioned already to jumping ship from Cloudbees+Jenkins entirely:

          Note that every other CI/CD system offers this very basic functionality out of the box.  It's that basic:

          AWS' offering: https://docs.aws.amazon.com/codepipeline/latest/userguide/actions-retry.html 

          CircleCI: https://circleci.com/docs/2.0/workflows/#rerunning-a-workflow-from-a-failed-job

           

           

           

          Christopher Proto added a comment - very misguided decision.  our situation is that we enrolled in the paid Jenkins Team version so we are PAYING customers to Cloudbees AND still don't get access to this very basic feature.  It's really sad and unfair to both customers and the community. Feels like a ripoff We're looking into options from workarounds mentioned already to jumping ship from Cloudbees+Jenkins entirely: Note that every other CI/CD system offers this very basic functionality out of the box.  It's that basic: AWS' offering:  https://docs.aws.amazon.com/codepipeline/latest/userguide/actions-retry.html   CircleCI: https://circleci.com/docs/2.0/workflows/#rerunning-a-workflow-from-a-failed-job      

          +1 Completely agree with comments above - it's a shame that Jenkins lacks this obvious feature. 

          Constantin Bugneac added a comment - +1 Completely agree with comments above - it's a shame that Jenkins lacks this obvious feature. 

          One year later and this issue is already open and not done. In time, I switch to gitlab CI and this does the work even if functions are not at same level. 

          Jean-Marc Collin added a comment - One year later and this issue is already open and not done. In time, I switch to gitlab CI and this does the work even if functions are not at same level. 

          Nat Sr added a comment -

          Is it possible to not only for after failure? Sometime I need to get redo the same stage, for example building a new build and testing until all pass before going to next step

           

          Nat Sr added a comment - Is it possible to not only for after failure? Sometime I need to get redo the same stage, for example building a new build and testing until all pass before going to next step  

          James Dumay added a comment -

          The plan is to allow this to work for any previously executed stage

          James Dumay added a comment - The plan is to allow this to work for any previously executed stage

          Nat Sr added a comment - - edited

          jamesdumay that's nice. Thanks James. Hopefully, we can get it ASAP.

          BTW, I saw this ticket which already has a PR up (https://issues.jenkins-ci.org/browse/JENKINS-45455) which I am not sure if it is the same thing or not but it looks like it will restart a specific stage from "Previous build". Can we restart the same build? Not kick off another build and run the specific stage?

          Ex. 
          1) Executed Stage 1
          2) Executed Stage 2
          3) Re-executed Stage 2
          4) Executed Stage 3 ....... and so on

          Nat Sr added a comment - - edited jamesdumay that's nice. Thanks James. Hopefully, we can get it ASAP. BTW, I saw this ticket which already has a PR up ( https://issues.jenkins-ci.org/browse/JENKINS-45455 ) which I am not sure if it is the same thing or not but it looks like it will restart a specific stage from " Previous build ". Can we restart the same build? Not kick off another build and run the specific stage? Ex.  1) Executed Stage 1 2) Executed Stage 2 3) Re-executed Stage 2 4) Executed Stage 3 ....... and so on

            Unassigned Unassigned
            michaelhuettermann Michael Hüttermann
            Votes:
            156 Vote for this issue
            Watchers:
            194 Start watching this issue

              Created:
              Updated: