Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33846

Restart pipeline from specific stage, after failure

    XMLWordPrintable

Details

    • New Feature
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • pipeline
    • None

    Description

      This feature is unplanned and not actively worked on

      The use case for restarting stages of a Pipeline is an important one and as such we intend to include this feature in Declarative Pipeline (JENKINS-45455) as open source. I have put this feature on the Declarative Pipeline roadmap. I encourage you to review JENKINS-45455 and leave a comment to ensure that your use case is covered.

      For the purpose of clarity, we will not be providing this feature for scripted pipelines.

      After review of the Checkpoints feature, the CloudBees team have come to the decision that will not be open sourcing Checkpoints. I know we had signalled in this issue that we would be open sourcing this feature and I am sincerely sorry to disappoint those who have been waiting for us to do so.

      Thanks,
      James Dumay
      Product Manager, CloudBees Inc

      It would be fine, and customers ask me for that continuously to have the function to again start a failed pipeline,but not at the very beginning, rather at a stage along the pipeline chain, particularly a stage which failed before during a former run,and the cause of the failure was fixed.

      Maybe I've chosen the wrong "component" for this ticket. Please feel free to adjust accordingly.

      Tested on Jenkins ver. 2.0-beta-1, and its new Pi Pipeline item.

      Attachments

        Issue Links

          Activity

            This feature is available in a proprietary plugin by CloudBees, this is the documentation.

            amuniz Antonio Muñiz added a comment - This feature is available in a proprietary plugin by CloudBees, this is the documentation .

            Hi amuniz,
            thanks for the hint. I think introducing a "checkpoint" is not necessary, since every single Stage should operate as a checkpoint.
            I'd like to see this in the free Jenkins distribution.

            michaelhuettermann Michael Hüttermann added a comment - Hi amuniz , thanks for the hint. I think introducing a "checkpoint" is not necessary, since every single Stage should operate as a checkpoint. I'd like to see this in the free Jenkins distribution.
            jglick Jesse Glick added a comment -

            No current plans to release this feature as OSS.

            jglick Jesse Glick added a comment - No current plans to release this feature as OSS.

            Such a shame... Without this feature, Workflow will NEVER replace "Deployment Pipeline" plugin in a PRODuction environment.
            Pipeline can be wide and slow and restarting a Pipeline entirely at each failure is simply not imaginable in the real life (but only in demos).

            I really don't understand that Cloudbees could be an obstacle to the adoption of Workflow. It never have such a position (for what I'm knowing) in the past.

            Can you please consider re-opening this fundamental issue ?

            Jean-Marc

            jmcollin Jean-Marc Collin added a comment - Such a shame... Without this feature, Workflow will NEVER replace "Deployment Pipeline" plugin in a PRODuction environment. Pipeline can be wide and slow and restarting a Pipeline entirely at each failure is simply not imaginable in the real life (but only in demos). I really don't understand that Cloudbees could be an obstacle to the adoption of Workflow. It never have such a position (for what I'm knowing) in the past. Can you please consider re-opening this fundamental issue ? Jean-Marc
            nnordrum Noah Nordrum added a comment -

            michaelhuettermann jmcollin

            Seems like there's an opportunity to create an OSS plugin that does this. Tweet me if you guys want to work together to make one.

            Noah

            nnordrum Noah Nordrum added a comment - michaelhuettermann jmcollin Seems like there's an opportunity to create an OSS plugin that does this. Tweet me if you guys want to work together to make one. Noah

            nnordrum I would be with great pleasure but :

            1. i have no time for this,
            2. i never develop a Jenkins Plugin. May be the gap is heavy ?

            Have you an idea of the complexity of such a plugin ?

            jmcollin Jean-Marc Collin added a comment - nnordrum I would be with great pleasure but : i have no time for this, i never develop a Jenkins Plugin. May be the gap is heavy ? Have you an idea of the complexity of such a plugin ?
            anudeeplalam Anudeep Lalam added a comment - - edited

            amuniz : The Documentation link is invalid. Can you please share the correct link. Also, proprietary Plug-in is not available in Jenkins. Is that the correct plug-in name? If not can you please share the correct name.

            anudeeplalam Anudeep Lalam added a comment - - edited amuniz : The Documentation link is invalid. Can you please share the correct link. Also, proprietary Plug-in is not available in Jenkins. Is that the correct plug-in name? If not can you please share the correct name.

            Here are the appropriate links to the Cloudbees Checkpoint feature...

            https://www.cloudbees.com/products/cloudbees-jenkins-platform/enterprise-edition/features/checkpoints-plugin
            https://go.cloudbees.com/docs/cloudbees-documentation/cje-user-guide/chapter-workflow.html?q=checkpoints

            I too would really like to see this as an OSS plugin and agree that fundamentally every stage should operate in this way. It's an obstacle for me trying to develop a flow in pipeline.

            I'll admit that my perspective is slightly different in that I'm coming from a hardware design flow and still in the early phase of investigating and trying out Jenkins for automation. My flow is fairly complex and stages are probably much longer than their typical software counterparts, i.e. hours/days and can't really just start over at the beginning every time. I think this is an important consideration for the Jenkins team to further push into other arenas (like the Embedded use-case on the website). I could see that this should possibly fall under enterprise-level support, however, as a side-note, my experience so far that the pay for executor business model doesn't scale well to our type of compute usage. I realize everybody needs to make money, but at my peon level have a hard time asking management to additionally pay to make use of our servers.

            jp83 Jonathan Phelps added a comment - Here are the appropriate links to the Cloudbees Checkpoint feature... https://www.cloudbees.com/products/cloudbees-jenkins-platform/enterprise-edition/features/checkpoints-plugin https://go.cloudbees.com/docs/cloudbees-documentation/cje-user-guide/chapter-workflow.html?q=checkpoints I too would really like to see this as an OSS plugin and agree that fundamentally every stage should operate in this way. It's an obstacle for me trying to develop a flow in pipeline. I'll admit that my perspective is slightly different in that I'm coming from a hardware design flow and still in the early phase of investigating and trying out Jenkins for automation. My flow is fairly complex and stages are probably much longer than their typical software counterparts, i.e. hours/days and can't really just start over at the beginning every time. I think this is an important consideration for the Jenkins team to further push into other arenas (like the Embedded use-case on the website). I could see that this should possibly fall under enterprise-level support, however, as a side-note, my experience so far that the pay for executor business model doesn't scale well to our type of compute usage. I realize everybody needs to make money, but at my peon level have a hard time asking management to additionally pay to make use of our servers.

            I have to agree with the comments made before. I understand that cloudbees needs to make money, but this might stop many users of the free version from adopting the new pipeline functionality. Seems like two conflicting goals.

            fschroder Federico Schroder added a comment - I have to agree with the comments made before. I understand that cloudbees needs to make money, but this might stop many users of the free version from adopting the new pipeline functionality. Seems like two conflicting goals.
            kmorgan Katherine Morgan added a comment - - edited

            I really really really want this feature. And it is absolutely going to stop me from using the pipeline functionality.

            I just spent a full day implementing my build and deployment process with the pipeline functionality. I have builds that progress through these stages: BUILD > DEV > STAGE > INTEG > PREPROD > PROD. I'm using the 'waiting for input' feature to capture human approval to progress the deployment to the next environment. In testing it, I found that the pipelines would fill my 'build executor status' list whilst they were 'waiting for approval', and would block jenkins from doing any more work. Just not usable.

            If the 'restart a stage' feature existed, I could then deploy any pipeline to any environment, achieving build progression and rollback.

            UPDATE: I've since learnt that I need to move my input to outside of a node, this resolves my blocked executors.

            kmorgan Katherine Morgan added a comment - - edited I really really really want this feature. And it is absolutely going to stop me from using the pipeline functionality. I just spent a full day implementing my build and deployment process with the pipeline functionality. I have builds that progress through these stages: BUILD > DEV > STAGE > INTEG > PREPROD > PROD. I'm using the 'waiting for input' feature to capture human approval to progress the deployment to the next environment. In testing it, I found that the pipelines would fill my 'build executor status' list whilst they were 'waiting for approval', and would block jenkins from doing any more work. Just not usable. If the 'restart a stage' feature existed, I could then deploy any pipeline to any environment, achieving build progression and rollback. UPDATE: I've since learnt that I need to move my input to outside of a node, this resolves my blocked executors.

            Ideally this plugin should be organized as it is done in DeliveryPlugin: you can group steps into stage, display each steps and stage and add a restart button on each steps and stage.
            Please take a look at the delivery Pipeline link here. I wonder if merging the both pulgin to take the best of each is not the best solution.

            jmcollin Jean-Marc Collin added a comment - Ideally this plugin should be organized as it is done in DeliveryPlugin: you can group steps into stage, display each steps and stage and add a restart button on each steps and stage. Please take a look at the delivery Pipeline link here . I wonder if merging the both pulgin to take the best of each is not the best solution.
            romanp Roman Pickl added a comment -

            This is a show stopper

            romanp Roman Pickl added a comment - This is a show stopper
            caley Caley Goff added a comment -

            Hi, this is a major show stopper for our PRODUCTION workflow.

            Like Katherine I currently use the Job Chained Workflow to move batches from BUILD> DEV> CI> QA> TEST> PROD, I cannot afford to have to restart the pipeline from the beginning with a failure that is somewhere in the chain.

            I've added check points as instructed

            stage('deploy ci'){
                node('windows') {
                    //code block
                }
                checkpoint 'Completed deploy ci'
            }
            
            java.lang.NoSuchMethodError: No such DSL method 'checkpoint' found among steps 
            

            We're running Jenkins ver. 1.651.3.1 (CloudBees Jenkins Enterprise 16.06)

            caley Caley Goff added a comment - Hi, this is a major show stopper for our PRODUCTION workflow. Like Katherine I currently use the Job Chained Workflow to move batches from BUILD> DEV> CI> QA> TEST> PROD, I cannot afford to have to restart the pipeline from the beginning with a failure that is somewhere in the chain. I've added check points as instructed stage( 'deploy ci' ){ node( 'windows' ) { //code block } checkpoint 'Completed deploy ci' } java.lang.NoSuchMethodError: No such DSL method 'checkpoint' found among steps We're running Jenkins ver. 1.651.3.1 (CloudBees Jenkins Enterprise 16.06)
            jglick Jesse Glick added a comment -

            caley

            We're running Jenkins ver. 1.651.3.1 (CloudBees Jenkins Enterprise 16.06)

            Please file a support ticket with CloudBees rather than discussing the issue here.

            jglick Jesse Glick added a comment - caley We're running Jenkins ver. 1.651.3.1 (CloudBees Jenkins Enterprise 16.06) Please file a support ticket with CloudBees rather than discussing the issue here.
            mrichar2 Mark R added a comment -

            Why won't this be implemented?

            mrichar2 Mark R added a comment - Why won't this be implemented?
            dantran dan tran added a comment -

            same here, it is very convenient with the build pipeline plugin. Can this be reopened and have user vote?

            dantran dan tran added a comment - same here, it is very convenient with the build pipeline plugin. Can this be reopened and have user vote?
            dantran dan tran added a comment - - edited

            Ah, Jenkins Enterprise supports this feature

            dantran dan tran added a comment - - edited Ah, Jenkins Enterprise supports this feature
            schulzha Hans Schulz added a comment -

            I also don't see how this is resolved.
            CloudBees not releasing their plugin is one thing, but this feature is still desireable in OSS Jenkins.

            schulzha Hans Schulz added a comment - I also don't see how this is resolved. CloudBees not releasing their plugin is one thing, but this feature is still desireable in OSS Jenkins.
            zigac Ziga Ciglar added a comment -

            I think this feature is necessary for new pipeline plugin. Otherwise OSS Jenkins staff should recommend to fall back to old "Build Pipeline Plugin".

            zigac Ziga Ciglar added a comment - I think this feature is necessary for new pipeline plugin. Otherwise OSS Jenkins staff should recommend to fall back to old "Build Pipeline Plugin".
            sadiqkhoja Sadiq Khoja added a comment -

            we need this

            sadiqkhoja Sadiq Khoja added a comment - we need this
            jglick Jesse Glick added a comment -

            checkpoint is a proprietary feature and will not be implemented in OSS.

            For Declarative Pipelines, in principle you could skip some early stage’s and just start running on a later one, using FlowCopier.

            jglick Jesse Glick added a comment - checkpoint is a proprietary feature and will not be implemented in OSS. For Declarative Pipelines, in principle you could skip some early stage ’s and just start running on a later one, using FlowCopier .
            eguess74 eguess74 added a comment -

            For all who wants this feature - there is a workaround:
            Create a "configuration job" that is a parameterized job. Fill it with the parameters that are reflecting the names of stages in the pipeline. for example (STAGE1=run/skip, STAGE2=run/skip)
            Use post build action trigger parameterized build to pass all those stages as parameters to the pipeline job.
            Use conditional build step in pipeline to determine if the step needs to be executed or not. (https://jenkins.io/blog/2017/01/19/converting-conditional-to-pipeline/)
            If the pipeline fails at STAGE2 you can rerun it starting from config job and you can say STAGE1=skip, STAGE2=run

            there are additional benefits as you can always go back to every run and see which parameters were used and so on.

            I hope that helps.

            eguess74 eguess74 added a comment - For all who wants this feature - there is a workaround: Create a "configuration job" that is a parameterized job. Fill it with the parameters that are reflecting the names of stages in the pipeline. for example (STAGE1=run/skip, STAGE2=run/skip) Use post build action trigger parameterized build to pass all those stages as parameters to the pipeline job. Use conditional build step in pipeline to determine if the step needs to be executed or not. ( https://jenkins.io/blog/2017/01/19/converting-conditional-to-pipeline/ ) If the pipeline fails at STAGE2 you can rerun it starting from config job and you can say STAGE1=skip, STAGE2=run there are additional benefits as you can always go back to every run and see which parameters were used and so on. I hope that helps.
            grahamlyons Graham Lyons added a comment -

            We have also been bitten by this and won't be able to adopt the new pipelines.

            A side effect of the stages of the pipeline being tightly coupled in one job if you don't keep a full build history (which is inadvisable as Jenkins becomes incredibly slow) then any failures anywhere in the pipeline will result in older builds being discarded and eventually you'll lose the record of e.g. your last successful deployment to production.

            grahamlyons Graham Lyons added a comment - We have also been bitten by this and won't be able to adopt the new pipelines. A side effect of the stages of the pipeline being tightly coupled in one job if you don't keep a full build history (which is inadvisable as Jenkins becomes incredibly slow) then any failures anywhere in the pipeline will result in older builds being discarded and eventually you'll lose the record of e.g. your last successful deployment to production.

            This is a slap in face for writing all the pipeline code and finding out that its not possible to restart a stage in OSS jenkins. I don't know what to do at this point as the workarounds does not help my case. 

            rgodavarthi Ravi Godavarthi added a comment - This is a slap in face for writing all the pipeline code and finding out that its not possible to restart a stage in OSS jenkins. I don't know what to do at this point as the workarounds does not help my case. 

            Agreed, it is a clear blocker for adoption. Especially if you have used the Delivery Pipeline plugin in the past in conjunction with the Rebuild plugin, it worked really nicely.

            pmatignon Patrice Matignon added a comment - Agreed, it is a clear blocker for adoption. Especially if you have used the Delivery Pipeline plugin in the past in conjunction with the Rebuild plugin, it worked really nicely.

            Would love this too!

            mmitche Matthew Mitchell added a comment - Would love this too!

            Would love to be able to pay for this feature, as the full Enterprise offering containing checkpoint is out of reach for me.

            giorgiosironi Giorgio Sironi added a comment - Would love to be able to pay for this feature, as the full Enterprise offering containing checkpoint is out of reach for me.

            This missing feature is the only reason stopping us from using the pipeline plugin. We use the Tikal Multijob plugin which allows us to resume from a phase in the pipeline. We really need Checkpoint to be open sourced in order for us to migrate over to use it. 

            emoshaya_cognitoiq Ebrahim Moshaya added a comment - This missing feature is the only reason stopping us from using the pipeline plugin. We use the Tikal Multijob plugin which allows us to resume from a phase in the pipeline. We really need Checkpoint to be open sourced in order for us to migrate over to use it. 
            mmaglana Mark Maglana added a comment -

            The policy of not releasing the plugin to OSS Jenkins is strange and will contribute to the public's perception that Pipeline is not ready for primetime. Personally, this is frustrating after weeks of advocating Jenkins 2.x and the Pipeline plugin only to find out about this policy which means I'll have to re-do my work.

            Also, while an argument can be made that the Checkpoint functionality is a plugin, it's not a standalone one. It exists because Pipeline exists and because the latter is broken without it. This makes Pipeline a partly open-sourced, partly closed-source product which reminds me of similar projects that have gone down in history as projects we'd rather not remember.

            mmaglana Mark Maglana added a comment - The policy of not releasing the plugin to OSS Jenkins is strange and will contribute to the public's perception that Pipeline is not ready for primetime. Personally, this is frustrating after weeks of advocating Jenkins 2.x and the Pipeline plugin only to find out about this policy which means I'll have to re-do my work. Also, while an argument can be made that the Checkpoint functionality is a plugin, it's not a standalone one. It exists because Pipeline exists and because the latter is broken without it. This makes Pipeline a partly open-sourced, partly closed-source product which reminds me of similar projects that have gone down in history as projects we'd rather not remember.

            mmaglana: This is something you should take up with CloudBees; this is the open source project issue tracker, which is not affiliated with that company (nor am I).

            If there's something you saw on the Jenkins website or wiki which led you to believe that some sort of checkpointing functionality was built in to the Pipeline suite of plugins, beyond the pause/resume and durability features, please let us know so that we can fix that, until the time that someone decides to work on an open-source solution.

            orrc Christopher Orr added a comment - mmaglana : This is something you should take up with CloudBees; this is the open source project issue tracker, which is not affiliated with that company (nor am I). If there's something you saw on the Jenkins website or wiki which led you to believe that some sort of checkpointing functionality was built in to the Pipeline suite of plugins, beyond the pause/resume and durability features, please let us know so that we can fix that, until the time that someone decides to work on an open-source solution.
            mmaglana Mark Maglana added a comment -

            Thanks for the response orrc. I will do as you suggest. However, I think my comment is still relevant in this issue since the Pipeline plugin is open source and appears to be an official component of this project based on this ticket's metadata. Also, checkpoints are an integral feature of the Pipeline plugin if we are to use it as it is intended (continuous delivery). For example, should a pipeline run fail because of some external service failure (staging environment went down, network latency deteriorated, some n00b with sudo access ran `rm -rf /`, etc) and not because of the build itself, there's no reason to restart the whole thing because then that would be a totally new build!

            As for the documentation, firstly, I want to make it clear that the website and wiki have been awesome resources for me and my team as we learned to use the Pipeline plugin. This speaks volumes to the dedication of the community and maintainers. Despite this very important feature being missing (or at least being made unavailable), Jenkins is still, hands down, the best CI/CD tool out there for me. If I remember correctly, nothing led me to believe that the checkpointing fucntionality was built in. However, it would have been helpful if this product limitation was mentioned in the Pipeline docs.

            mmaglana Mark Maglana added a comment - Thanks for the response orrc . I will do as you suggest. However, I think my comment is still relevant in this issue since the Pipeline plugin is open source and appears to be an official component of this project based on this ticket's metadata. Also, checkpoints are an integral feature of the Pipeline plugin if we are to use it as it is intended (continuous delivery). For example, should a pipeline run fail because of some external service failure (staging environment went down, network latency deteriorated, some n00b with sudo access ran `rm -rf /`, etc) and not because of the build itself, there's no reason to restart the whole thing because then that would be a totally new build! As for the documentation, firstly, I want to make it clear that the website and wiki have been awesome resources for me and my team as we learned to use the Pipeline plugin. This speaks volumes to the dedication of the community and maintainers. Despite this very important feature being missing (or at least being made unavailable), Jenkins is still, hands down, the best CI/CD tool out there for me. If I remember correctly, nothing led me to believe that the checkpointing fucntionality was built in. However, it would have been helpful if this product limitation was mentioned in the Pipeline docs.
            grahamlyons Graham Lyons added a comment -

            This is solving the problem for me at the moment: http://concourse.ci/

            The pipelines in Jenkins lack a lot of what you would expect from a CI/CD pipeline, beyond just the restarting aspect.

            This is an interesting discussion on the subject:

            https://highops.com/insights/continuous-delivery-pipelines-gocd-vs-jenkins/

             

            grahamlyons Graham Lyons added a comment - This is solving the problem for me at the moment: http://concourse.ci/ The pipelines in Jenkins lack a lot of what you would expect from a CI/CD pipeline, beyond just the restarting aspect. This is an interesting discussion on the subject: https://highops.com/insights/continuous-delivery-pipelines-gocd-vs-jenkins/  
            hrmpw Patrick Wolf added a comment - - edited

            I'm glad to see the level of interest here. I have withheld comment here until it was official but I'm happy to announce that CloudBees will be be releasing the pipeline-checkpoint plugin to open source in the near future.

            This decision has been coming for some time but we wanted to examine this from all angles before making a final decision. I will post a blog and we will begin moving the code to jenkinsci on Github soon. 

            hrmpw Patrick Wolf added a comment - - edited I'm glad to see the level of interest here. I have withheld comment here until it was official but I'm happy to announce that CloudBees will be be releasing the pipeline-checkpoint plugin to open source in the near future. This decision has been coming for some time but we wanted to examine this from all angles before making a final decision. I will post a blog and we will begin moving the code to jenkinsci on Github soon. 

            I just checked my email and logged in just for saying "Arigatho" hrmpw . This is the best news I've heard in a while. YAY

            rgodavarthi Ravi Godavarthi added a comment - I just checked my email and logged in just for saying "Arigatho" hrmpw . This is the best news I've heard in a while. YAY
            schulzha Hans Schulz added a comment -

            That is great news, thank you!

            schulzha Hans Schulz added a comment - That is great news, thank you!

            Wonderfull, it is the news I've been waiting for more than one year. Many thanks to Cloudbees for this decision.

             

            Jean-Marc

            jmcollin Jean-Marc Collin added a comment - Wonderfull, it is the news I've been waiting for more than one year. Many thanks to Cloudbees for this decision.   Jean-Marc

            hrmpw: Thanks for the update, that's certainly interesting news!

            Everybody else: thanks for your enthusiasm, but please remember that this is an issue tracker, and isn't a suitable place for extended discussions.  Every time you add (or edit) a comment, it currently gets sent to at least 50 people.  If you're interested in updates to this issue, please click "Watch this issue" on the right, rather than commenting here.

            Please take a look at https://jenkins.io/participate/ for more information on the various ways you can participate in the project

            orrc Christopher Orr added a comment - hrmpw : Thanks for the update, that's certainly interesting news! Everybody else: thanks for your enthusiasm, but please remember that this is an issue tracker, and isn't a suitable place for extended discussions.  Every time you add (or edit) a comment, it currently gets sent to at least 50 people.  If you're interested in updates to this issue, please click "Watch this issue" on the right, rather than commenting here. Please take a look at https://jenkins.io/participate/  for more information on the various ways you can participate in the project
            jenkins_pipeline_blue Sumit Kumar added a comment -

            Do we have a timeline on making checkpoint plugin opensource?

            This is a major blocker for us to adobt Jenkins 2.0 Pipeline.

            We should be able to deploy previous versions to the higher environments i.e. In terms of pipeline, resuming from "Deploy to Dev" stage from an earlier build to proceed to "Deploy to QA" Stage using the respective workspace.

            jenkins_pipeline_blue Sumit Kumar added a comment - Do we have a timeline on making checkpoint plugin opensource? This is a major blocker for us to adobt Jenkins 2.0 Pipeline. We should be able to deploy previous versions to the higher environments i.e. In terms of pipeline, resuming from "Deploy to Dev" stage from an earlier build to proceed to "Deploy to QA" Stage using the respective workspace.
            lesinge Yoh Ann added a comment -

            Agree with jenkins_pipeline_blue, major blocker to adopt !

            lesinge Yoh Ann added a comment - Agree with jenkins_pipeline_blue , major blocker to adopt !

            I agree, this non-yet-existing-feature is a major blocker for CI/CD in our company. GoCD has this feature free out of the box.

            elzinko Thomas Couderc added a comment - I agree, this non-yet-existing-feature is a major blocker for CI/CD in our company. GoCD has this feature free out of the box.

            Bump on this issue.

            It is over 3 months since the statement that this will become avaliable in the OSS version, but then totally quiet.

            Either things will start happening, or the OSS community needs to prioritize this as a missing feature and re-implement it, which would be insanely stupid if it already works...

            Please Cloudbees, give us at least a timeframe for when this will become publicly avaliable.

             

            Regards

            Sofus

            sofusalbertsen Sofus Albertsen added a comment - Bump on this issue. It is over 3 months since the statement that this will become avaliable in the OSS version, but then totally quiet. Either things will start happening, or the OSS community needs to prioritize this as a missing feature and re-implement it, which would be insanely stupid if it already works... Please Cloudbees, give us at least a timeframe for when this will become publicly avaliable.   Regards Sofus
            jamesdumay James Dumay added a comment -

            Hi there,

            After review of the checkpoints feature, the CloudBees team have come to the decision that will not be open sourcing Checkpoints. I know we had signalled in this issue that we would be open sourcing this feature and I am sincerely sorry to disappoint those who have been waiting for us to do so.

            The use case for restarting stages of a Pipeline is an important one and as such we intend to include this feature in Declarative Pipeline (JENKINS-45455) as open source. I have put this feature on the Declarative Pipeline roadmap. I encourage you to review JENKINS-45455 and leave a comment to ensure that your use case is covered.

            For the purpose of clarity, we will not be providing this feature for scripted pipelines. If you have any questions, please don't hesitate to comment.

            Thanks,
            James Dumay
            Product Manager, CloudBees Inc

            jamesdumay James Dumay added a comment - Hi there, After review of the checkpoints feature, the CloudBees team have come to the decision that will not be open sourcing Checkpoints. I know we had signalled in this issue that we would be open sourcing this feature and I am sincerely sorry to disappoint those who have been waiting for us to do so. The use case for restarting stages of a Pipeline is an important one and as such we intend to include this feature in Declarative Pipeline ( JENKINS-45455 ) as open source. I have put this feature on the Declarative Pipeline roadmap . I encourage you to review JENKINS-45455 and leave a comment to ensure that your use case is covered. For the purpose of clarity, we will not be providing this feature for scripted pipelines. If you have any questions, please don't hesitate to comment. Thanks, James Dumay Product Manager, CloudBees Inc
            mkobit Mike Kobit added a comment -

            Sad to see this, as we were really looking forward to this feature.

            What is the motivation to only provide this to declarative pipelines and not for scripted pipelines?

            mkobit Mike Kobit added a comment - Sad to see this, as we were really looking forward to this feature. What is the motivation to only provide this to declarative pipelines and not for scripted pipelines?
            jamesdumay James Dumay added a comment -

            mkobit scripted pipeline is extremely open ended. As I wrote on JENKINS-45455 "There is less complexity and edge cases needed to be handled by making this functionality a Declarative Pipeline feature only".

            jamesdumay James Dumay added a comment - mkobit scripted pipeline is extremely open ended. As I wrote on JENKINS-45455 "There is less complexity and edge cases needed to be handled by making this functionality a Declarative Pipeline feature only".

            Declarative Pipeline is so immature and only applicable to applications deployed on few environments. The idea of having a seperate Git repo per application where branches correspond to environments is like forcing people to use Branch per Environment strategy which is completely not applicable where there are multiple environments using the same branch.

            e.g. Application1.git (branches) : dev1, dev2, qa, prod, prodb, reg

            If this is to promote Blueocean which is completely useless when you want to run a set of tasks everyday on 20+ environments translates to creating 20 branches and maintain 20 jenkinsFiles only because CloudBees holds the restart Checkpoint plugin. 

            So basically this is gonna be free for small applications and when it actually comes down to a Production scale complex orchestrations you gotta buy Cloudbees Enterprise version

            This is unfortunate since I've spent hundreds of hours on this and I can't afford to pay for enterprise license only to get this single feature. I guess investing time into making my own checkpoint plugin, even if it is as simple as saving the ``` binding.properties and sh 'env' ``` into a file and loading it at begining of every stage

            rgodavarthi Ravi Godavarthi added a comment - Declarative Pipeline is so immature and only applicable to applications deployed on few environments. The idea of having a seperate Git repo per application where branches correspond to environments is like forcing people to use Branch per Environment strategy which is completely not applicable where there are multiple environments using the same branch. e.g. Application1.git (branches) : dev1, dev2, qa, prod, prodb, reg If this is to promote Blueocean which is completely useless when you want to run a set of tasks everyday on 20+ environments translates to creating 20 branches and maintain 20 jenkinsFiles only because CloudBees holds the restart Checkpoint plugin.  So basically this is gonna be free for small applications and when it actually comes down to a Production scale complex orchestrations you gotta buy Cloudbees Enterprise version This is unfortunate since I've spent hundreds of hours on this and I can't afford to pay for enterprise license only to get this single feature. I guess investing time into making my own checkpoint plugin, even if it is as simple as saving the ``` binding.properties and sh 'env' ``` into a file and loading it at begining of every stage
            schulzha Hans Schulz added a comment -

            Agreeing with what rgodavarthi said.

            Declarative pipelines are very restrictive and unsuitable for our pipelines. We have huge Jenkins libraries (>10k lines of code) which basically build an ecosystem around our development and release environment, and we can't afford to refactor these to fit into declarative pipelines, just to get checkpoints working. This is a huge disappointment after the announcement of hrmpw.

            schulzha Hans Schulz added a comment - Agreeing with what rgodavarthi said. Declarative pipelines are very restrictive and unsuitable for our pipelines. We have huge Jenkins libraries (>10k lines of code) which basically build an ecosystem around our development and release environment, and we can't afford to refactor these to fit into declarative pipelines, just to get checkpoints working. This is a huge disappointment after the announcement of hrmpw .
            tobybro Toby Broyles added a comment -

            My understanding was that scripted pipelines were the way to go due to being unrestricted and more powerful. There was nothing in declarative pipelines that you couldn't do in scripted pipelines but now it sounds like that is not going to be the case? 

             

            I am very sad to to see this.

            tobybro Toby Broyles added a comment - My understanding was that scripted pipelines were the way to go due to being unrestricted and more powerful. There was nothing in declarative pipelines that you couldn't do in scripted pipelines but now it sounds like that is not going to be the case?    I am very sad to to see this.

            I echo the same sentiment. I get that declarative pipelines are meant to be super-easy to use and that's why work on declarative is prioritize. But, for power users it's a problem when something is in declarative and not in scripted, since we prefer the power/freedom that scripted pipelines provide.

            adelcast Alejandro del Castillo added a comment - I echo the same sentiment. I get that declarative pipelines are meant to be super-easy to use and that's why work on declarative is prioritize. But, for power users it's a problem when something is in declarative and not in scripted, since we prefer the power/freedom that scripted pipelines provide.

            Couldn't agree more with the comments since the "we've changed our minds and we're not gonna open source it" announcement a few days ago.

            The old bait and switch... gotta love it.

            ljohnston Lance Johnston added a comment - Couldn't agree more with the comments since the "we've changed our minds and we're not gonna open source it" announcement a few days ago. The old bait and switch... gotta love it.
            btb3 Martin Ba added a comment - - edited

            jamesdumay - so ...

            scripted pipeline is extremely open ended. As I wrote on JENKINS-45455 "There is less complexity and edge cases needed to be handled by making this functionality a Declarative Pipeline feature only".

            ... I'm rather unsure how to interpret this: checkpoint is already implemented, so any reference to "less complexity" is either handwaving, or the checkpoint feature you currently want to us to pay for^[a]^ doesn't work properly.

             

            I'll note, as others, that the inability to have checkpoints in jenkins pipeline makes it pretty much useless to us. (workarounds like https://stackoverflow.com/a/40351693/321013 and other mentioned ones non-withstanding)

            ([a]: I'm fine with you trying to make a buck. It's just that I feel the checkpoint feature is not enterprisey at all - it seems vital for any non-trivial long-running pipeline, also (and maybe especially) for small teams.)

            btb3 Martin Ba added a comment - - edited jamesdumay - so ... scripted pipeline is extremely open ended. As I wrote on JENKINS-45455 "There is less complexity and edge cases needed to be handled by making this functionality a Declarative Pipeline feature only". ... I'm rather unsure how to interpret this: checkpoint is already implemented, so any reference to "less complexity" is either handwaving, or the checkpoint feature you currently want to us to pay for^ [a] ^ doesn't work properly.   I'll note, as others, that the inability to have checkpoints in jenkins pipeline makes it pretty much useless to us. (workarounds like https://stackoverflow.com/a/40351693/321013 and other mentioned ones non-withstanding) ( [a] : I'm fine with you trying to make a buck. It's just that I feel the checkpoint feature is not enterprisey at all - it seems vital for any non-trivial long-running pipeline, also (and maybe especially) for small teams.)
            lmende Lorenz Mende added a comment -

            I have to agree Martin, the checkpoint feature seems like a vital function for pipeline workflows.

            We are running jobs which last hours to a day and are hardly to modularize further...

             

            Very disappointing decision.

            lmende Lorenz Mende added a comment - I have to agree Martin, the checkpoint feature seems like a vital function for pipeline workflows. We are running jobs which last hours to a day and are hardly to modularize further...   Very disappointing decision.
            tjj225 Tharun John added a comment -

            It is like one step forward and 2 steps backwards. I understand we all need to make money. But this is a genuine request from the community. We are a small team and don't have powerful machines to rerun tests from the beginning. So we still have not yet adopted the groovy code and still running manual pipelines views to resume jobs. Does anybody have a work around for this problem ?

             

            tjj225 Tharun John added a comment - It is like one step forward and 2 steps backwards. I understand we all need to make money. But this is a genuine request from the community. We are a small team and don't have powerful machines to rerun tests from the beginning. So we still have not yet adopted the groovy code and still running manual pipelines views to resume jobs. Does anybody have a work around for this problem ?  
            nocono Oliver Nocon added a comment - - edited

            We are using a workaround with a custom step within a pipeline library similar to this:

            //restartIt.goovy
            def call(Map parameters = [:], body) {
              def timeoutInSeconds = parameters.get('timeoutInSeconds', 900)
              def restart = true
              while (restart) {
                try {
                  body()
                  restart = false
                } catch (Throwable err) {
                  echo "ERROR occured: ${err}"
                  //optional email notification
                  try {
                    timeout(time: timeoutInSeconds, unit: 'SECONDS') {
                      input message: 'Do you want to restart?', ok: 'Restart'
                    }
                  } catch(e) {
                    restart = false
                    throw err
                  }
                }
              }
            }

            This can the be used like:

            restartIt(timeout:1200) {
              node {
                //here the steps you want to possibly restart
              }
            }

             

            For sure, there are some caveats. For example, you should use a node to not block executors.

            Anyway, it helps us in cases where e.g. deployments sometime fail.

            Maybe it helps you as well ...

             

            nocono Oliver Nocon added a comment - - edited We are using a workaround with a custom step within a pipeline library similar to this: //restartIt.goovy def call(Map parameters = [:] , body) {   def timeoutInSeconds = parameters.get('timeoutInSeconds', 900)   def restart = true   while (restart) {     try {       body()       restart = false     } catch (Throwable err) {       echo "ERROR occured: ${err}"       //optional email notification       try {         timeout(time: timeoutInSeconds, unit: 'SECONDS') {           input message: 'Do you want to restart?', ok: 'Restart'         }       } catch(e) {         restart = false         throw err       }     }   } } This can the be used like: restartIt(timeout:1200) {   node {     //here the steps you want to possibly restart   } }   For sure, there are some caveats. For example, you should use a node to not block executors. Anyway, it helps us in cases where e.g. deployments sometime fail. Maybe it helps you as well ...  

            very misguided decision.  our situation is that we enrolled in the paid Jenkins Team version so we are PAYING customers to Cloudbees AND still don't get access to this very basic feature.  It's really sad and unfair to both customers and the community. Feels like a ripoff

            We're looking into options from workarounds mentioned already to jumping ship from Cloudbees+Jenkins entirely:

            Note that every other CI/CD system offers this very basic functionality out of the box.  It's that basic:

            AWS' offering: https://docs.aws.amazon.com/codepipeline/latest/userguide/actions-retry.html 

            CircleCI: https://circleci.com/docs/2.0/workflows/#rerunning-a-workflow-from-a-failed-job

             

             

             

            cproto Christopher Proto added a comment - very misguided decision.  our situation is that we enrolled in the paid Jenkins Team version so we are PAYING customers to Cloudbees AND still don't get access to this very basic feature.  It's really sad and unfair to both customers and the community. Feels like a ripoff We're looking into options from workarounds mentioned already to jumping ship from Cloudbees+Jenkins entirely: Note that every other CI/CD system offers this very basic functionality out of the box.  It's that basic: AWS' offering:  https://docs.aws.amazon.com/codepipeline/latest/userguide/actions-retry.html   CircleCI: https://circleci.com/docs/2.0/workflows/#rerunning-a-workflow-from-a-failed-job      

            +1 Completely agree with comments above - it's a shame that Jenkins lacks this obvious feature. 

            cb_tes_global Constantin Bugneac added a comment - +1 Completely agree with comments above - it's a shame that Jenkins lacks this obvious feature. 

            One year later and this issue is already open and not done. In time, I switch to gitlab CI and this does the work even if functions are not at same level. 

            jmcollin Jean-Marc Collin added a comment - One year later and this issue is already open and not done. In time, I switch to gitlab CI and this does the work even if functions are not at same level. 
            comscience15 Nat Sr added a comment -

            Is it possible to not only for after failure? Sometime I need to get redo the same stage, for example building a new build and testing until all pass before going to next step

             

            comscience15 Nat Sr added a comment - Is it possible to not only for after failure? Sometime I need to get redo the same stage, for example building a new build and testing until all pass before going to next step  
            jamesdumay James Dumay added a comment -

            The plan is to allow this to work for any previously executed stage

            jamesdumay James Dumay added a comment - The plan is to allow this to work for any previously executed stage
            comscience15 Nat Sr added a comment - - edited

            jamesdumay that's nice. Thanks James. Hopefully, we can get it ASAP.

            BTW, I saw this ticket which already has a PR up (https://issues.jenkins-ci.org/browse/JENKINS-45455) which I am not sure if it is the same thing or not but it looks like it will restart a specific stage from "Previous build". Can we restart the same build? Not kick off another build and run the specific stage?

            Ex. 
            1) Executed Stage 1
            2) Executed Stage 2
            3) Re-executed Stage 2
            4) Executed Stage 3 ....... and so on

            comscience15 Nat Sr added a comment - - edited jamesdumay that's nice. Thanks James. Hopefully, we can get it ASAP. BTW, I saw this ticket which already has a PR up ( https://issues.jenkins-ci.org/browse/JENKINS-45455 ) which I am not sure if it is the same thing or not but it looks like it will restart a specific stage from " Previous build ". Can we restart the same build? Not kick off another build and run the specific stage? Ex.  1) Executed Stage 1 2) Executed Stage 2 3) Re-executed Stage 2 4) Executed Stage 3 ....... and so on

            People

              Unassigned Unassigned
              michaelhuettermann Michael Hüttermann
              Votes:
              155 Vote for this issue
              Watchers:
              193 Start watching this issue

              Dates

                Created:
                Updated: