This feature is available in a proprietary plugin by CloudBees, this is the documentation.

Hi amuniz,
thanks for the hint. I think introducing a "checkpoint" is not necessary, since every single Stage should operate as a checkpoint.
I'd like to see this in the free Jenkins distribution.

No current plans to release this feature as OSS.

Such a shame... Without this feature, Workflow will NEVER replace "Deployment Pipeline" plugin in a PRODuction environment.
Pipeline can be wide and slow and restarting a Pipeline entirely at each failure is simply not imaginable in the real life (but only in demos).

I really don't understand that Cloudbees could be an obstacle to the adoption of Workflow. It never have such a position (for what I'm knowing) in the past.

Can you please consider re-opening this fundamental issue ?

Jean-Marc

michaelhuettermann jmcollin

Seems like there's an opportunity to create an OSS plugin that does this. Tweet me if you guys want to work together to make one.

Noah

nnordrum I would be with great pleasure but :

1. i have no time for this,
2. i never develop a Jenkins Plugin. May be the gap is heavy ?

Have you an idea of the complexity of such a plugin ?

amuniz : The Documentation link is invalid. Can you please share the correct link. Also, proprietary Plug-in is not available in Jenkins. Is that the correct plug-in name? If not can you please share the correct name.

Here are the appropriate links to the Cloudbees Checkpoint feature...

https://www.cloudbees.com/products/cloudbees-jenkins-platform/enterprise-edition/features/checkpoints-plugin
https://go.cloudbees.com/docs/cloudbees-documentation/cje-user-guide/chapter-workflow.html?q=checkpoints

I too would really like to see this as an OSS plugin and agree that fundamentally every stage should operate in this way. It's an obstacle for me trying to develop a flow in pipeline.

I'll admit that my perspective is slightly different in that I'm coming from a hardware design flow and still in the early phase of investigating and trying out Jenkins for automation. My flow is fairly complex and stages are probably much longer than their typical software counterparts, i.e. hours/days and can't really just start over at the beginning every time. I think this is an important consideration for the Jenkins team to further push into other arenas (like the Embedded use-case on the website). I could see that this should possibly fall under enterprise-level support, however, as a side-note, my experience so far that the pay for executor business model doesn't scale well to our type of compute usage. I realize everybody needs to make money, but at my peon level have a hard time asking management to additionally pay to make use of our servers.

I have to agree with the comments made before. I understand that cloudbees needs to make money, but this might stop many users of the free version from adopting the new pipeline functionality. Seems like two conflicting goals.

I really really really want this feature. And it is absolutely going to stop me from using the pipeline functionality.

I just spent a full day implementing my build and deployment process with the pipeline functionality. I have builds that progress through these stages: BUILD > DEV > STAGE > INTEG > PREPROD > PROD. I'm using the 'waiting for input' feature to capture human approval to progress the deployment to the next environment. In testing it, I found that the pipelines would fill my 'build executor status' list whilst they were 'waiting for approval', and would block jenkins from doing any more work. Just not usable.

If the 'restart a stage' feature existed, I could then deploy any pipeline to any environment, achieving build progression and rollback.

UPDATE: I've since learnt that I need to move my input to outside of a node, this resolves my blocked executors.

Ideally this plugin should be organized as it is done in DeliveryPlugin: you can group steps into stage, display each steps and stage and add a restart button on each steps and stage.
Please take a look at the delivery Pipeline link here. I wonder if merging the both pulgin to take the best of each is not the best solution.

This is a show stopper

Hi, this is a major show stopper for our PRODUCTION workflow.

Like Katherine I currently use the Job Chained Workflow to move batches from BUILD> DEV> CI> QA> TEST> PROD, I cannot afford to have to restart the pipeline from the beginning with a failure that is somewhere in the chain.

I've added check points as instructed

stage('deploy ci'){
    node('windows') {
        //code block
    }
    checkpoint 'Completed deploy ci'
}

java.lang.NoSuchMethodError: No such DSL method 'checkpoint' found among steps 

We're running Jenkins ver. 1.651.3.1 (CloudBees Jenkins Enterprise 16.06)

caley

We're running Jenkins ver. 1.651.3.1 (CloudBees Jenkins Enterprise 16.06)

Please file a support ticket with CloudBees rather than discussing the issue here.

Why won't this be implemented?

same here, it is very convenient with the build pipeline plugin. Can this be reopened and have user vote?

Ah, Jenkins Enterprise supports this feature

I also don't see how this is resolved.
CloudBees not releasing their plugin is one thing, but this feature is still desireable in OSS Jenkins.

I think this feature is necessary for new pipeline plugin. Otherwise OSS Jenkins staff should recommend to fall back to old "Build Pipeline Plugin".

we need this

checkpoint is a proprietary feature and will not be implemented in OSS.

For Declarative Pipelines, in principle you could skip some early stage’s and just start running on a later one, using FlowCopier.

For all who wants this feature - there is a workaround:
Create a "configuration job" that is a parameterized job. Fill it with the parameters that are reflecting the names of stages in the pipeline. for example (STAGE1=run/skip, STAGE2=run/skip)
Use post build action trigger parameterized build to pass all those stages as parameters to the pipeline job.
Use conditional build step in pipeline to determine if the step needs to be executed or not. (https://jenkins.io/blog/2017/01/19/converting-conditional-to-pipeline/)
If the pipeline fails at STAGE2 you can rerun it starting from config job and you can say STAGE1=skip, STAGE2=run

there are additional benefits as you can always go back to every run and see which parameters were used and so on.

I hope that helps.

We have also been bitten by this and won't be able to adopt the new pipelines.

A side effect of the stages of the pipeline being tightly coupled in one job if you don't keep a full build history (which is inadvisable as Jenkins becomes incredibly slow) then any failures anywhere in the pipeline will result in older builds being discarded and eventually you'll lose the record of e.g. your last successful deployment to production.

This is a slap in face for writing all the pipeline code and finding out that its not possible to restart a stage in OSS jenkins. I don't know what to do at this point as the workarounds does not help my case. 

Agreed, it is a clear blocker for adoption. Especially if you have used the Delivery Pipeline plugin in the past in conjunction with the Rebuild plugin, it worked really nicely.

Would love this too!

Would love to be able to pay for this feature, as the full Enterprise offering containing checkpoint is out of reach for me.

This missing feature is the only reason stopping us from using the pipeline plugin. We use the Tikal Multijob plugin which allows us to resume from a phase in the pipeline. We really need Checkpoint to be open sourced in order for us to migrate over to use it. 

The policy of not releasing the plugin to OSS Jenkins is strange and will contribute to the public's perception that Pipeline is not ready for primetime. Personally, this is frustrating after weeks of advocating Jenkins 2.x and the Pipeline plugin only to find out about this policy which means I'll have to re-do my work.

Also, while an argument can be made that the Checkpoint functionality is a plugin, it's not a standalone one. It exists because Pipeline exists and because the latter is broken without it. This makes Pipeline a partly open-sourced, partly closed-source product which reminds me of similar projects that have gone down in history as projects we'd rather not remember.

mmaglana: This is something you should take up with CloudBees; this is the open source project issue tracker, which is not affiliated with that company (nor am I).

If there's something you saw on the Jenkins website or wiki which led you to believe that some sort of checkpointing functionality was built in to the Pipeline suite of plugins, beyond the pause/resume and durability features, please let us know so that we can fix that, until the time that someone decides to work on an open-source solution.

Thanks for the response orrc. I will do as you suggest. However, I think my comment is still relevant in this issue since the Pipeline plugin is open source and appears to be an official component of this project based on this ticket's metadata. Also, checkpoints are an integral feature of the Pipeline plugin if we are to use it as it is intended (continuous delivery). For example, should a pipeline run fail because of some external service failure (staging environment went down, network latency deteriorated, some n00b with sudo access ran `rm -rf /`, etc) and not because of the build itself, there's no reason to restart the whole thing because then that would be a totally new build!

As for the documentation, firstly, I want to make it clear that the website and wiki have been awesome resources for me and my team as we learned to use the Pipeline plugin. This speaks volumes to the dedication of the community and maintainers. Despite this very important feature being missing (or at least being made unavailable), Jenkins is still, hands down, the best CI/CD tool out there for me. If I remember correctly, nothing led me to believe that the checkpointing fucntionality was built in. However, it would have been helpful if this product limitation was mentioned in the Pipeline docs.

This is solving the problem for me at the moment: http://concourse.ci/

The pipelines in Jenkins lack a lot of what you would expect from a CI/CD pipeline, beyond just the restarting aspect.

This is an interesting discussion on the subject:

https://highops.com/insights/continuous-delivery-pipelines-gocd-vs-jenkins/

I'm glad to see the level of interest here. I have withheld comment here until it was official but I'm happy to announce that CloudBees will be be releasing the pipeline-checkpoint plugin to open source in the near future.

This decision has been coming for some time but we wanted to examine this from all angles before making a final decision. I will post a blog and we will begin moving the code to jenkinsci on Github soon. 

I just checked my email and logged in just for saying "Arigatho" hrmpw . This is the best news I've heard in a while. YAY

That is great news, thank you!

Wonderfull, it is the news I've been waiting for more than one year. Many thanks to Cloudbees for this decision.

Jean-Marc

hrmpw: Thanks for the update, that's certainly interesting news!

Everybody else: thanks for your enthusiasm, but please remember that this is an issue tracker, and isn't a suitable place for extended discussions.  Every time you add (or edit) a comment, it currently gets sent to at least 50 people.  If you're interested in updates to this issue, please click "Watch this issue" on the right, rather than commenting here.

Please take a look at https://jenkins.io/participate/ for more information on the various ways you can participate in the project

Do we have a timeline on making checkpoint plugin opensource?

This is a major blocker for us to adobt Jenkins 2.0 Pipeline.

We should be able to deploy previous versions to the higher environments i.e. In terms of pipeline, resuming from "Deploy to Dev" stage from an earlier build to proceed to "Deploy to QA" Stage using the respective workspace.

Agree with jenkins_pipeline_blue, major blocker to adopt !

I agree, this non-yet-existing-feature is a major blocker for CI/CD in our company. GoCD has this feature free out of the box.

Bump on this issue.

It is over 3 months since the statement that this will become avaliable in the OSS version, but then totally quiet.

Either things will start happening, or the OSS community needs to prioritize this as a missing feature and re-implement it, which would be insanely stupid if it already works...

Please Cloudbees, give us at least a timeframe for when this will become publicly avaliable.

Regards

Sofus

Hi there,

After review of the checkpoints feature, the CloudBees team have come to the decision that will not be open sourcing Checkpoints. I know we had signalled in this issue that we would be open sourcing this feature and I am sincerely sorry to disappoint those who have been waiting for us to do so.

The use case for restarting stages of a Pipeline is an important one and as such we intend to include this feature in Declarative Pipeline (JENKINS-45455) as open source. I have put this feature on the Declarative Pipeline roadmap. I encourage you to review JENKINS-45455 and leave a comment to ensure that your use case is covered.

For the purpose of clarity, we will not be providing this feature for scripted pipelines. If you have any questions, please don't hesitate to comment.

Thanks,
James Dumay
Product Manager, CloudBees Inc

Sad to see this, as we were really looking forward to this feature.

What is the motivation to only provide this to declarative pipelines and not for scripted pipelines?

mkobit scripted pipeline is extremely open ended. As I wrote on JENKINS-45455 "There is less complexity and edge cases needed to be handled by making this functionality a Declarative Pipeline feature only".

Declarative Pipeline is so immature and only applicable to applications deployed on few environments. The idea of having a seperate Git repo per application where branches correspond to environments is like forcing people to use Branch per Environment strategy which is completely not applicable where there are multiple environments using the same branch.

e.g. Application1.git (branches) : dev1, dev2, qa, prod, prodb, reg

If this is to promote Blueocean which is completely useless when you want to run a set of tasks everyday on 20+ environments translates to creating 20 branches and maintain 20 jenkinsFiles only because CloudBees holds the restart Checkpoint plugin. 

So basically this is gonna be free for small applications and when it actually comes down to a Production scale complex orchestrations you gotta buy Cloudbees Enterprise version

This is unfortunate since I've spent hundreds of hours on this and I can't afford to pay for enterprise license only to get this single feature. I guess investing time into making my own checkpoint plugin, even if it is as simple as saving the ``` binding.properties and sh 'env' ``` into a file and loading it at begining of every stage

Agreeing with what rgodavarthi said.

Declarative pipelines are very restrictive and unsuitable for our pipelines. We have huge Jenkins libraries (>10k lines of code) which basically build an ecosystem around our development and release environment, and we can't afford to refactor these to fit into declarative pipelines, just to get checkpoints working. This is a huge disappointment after the announcement of hrmpw.

My understanding was that scripted pipelines were the way to go due to being unrestricted and more powerful. There was nothing in declarative pipelines that you couldn't do in scripted pipelines but now it sounds like that is not going to be the case? 

I am very sad to to see this.

I echo the same sentiment. I get that declarative pipelines are meant to be super-easy to use and that's why work on declarative is prioritize. But, for power users it's a problem when something is in declarative and not in scripted, since we prefer the power/freedom that scripted pipelines provide.

Couldn't agree more with the comments since the "we've changed our minds and we're not gonna open source it" announcement a few days ago.

The old bait and switch... gotta love it.

jamesdumay - so ...

scripted pipeline is extremely open ended. As I wrote on JENKINS-45455 "There is less complexity and edge cases needed to be handled by making this functionality a Declarative Pipeline feature only".

... I'm rather unsure how to interpret this: checkpoint is already implemented, so any reference to "less complexity" is either handwaving, or the checkpoint feature you currently want to us to pay for^[a]^ doesn't work properly.

I'll note, as others, that the inability to have checkpoints in jenkins pipeline makes it pretty much useless to us. (workarounds like https://stackoverflow.com/a/40351693/321013 and other mentioned ones non-withstanding)

([a]: I'm fine with you trying to make a buck. It's just that I feel the checkpoint feature is not enterprisey at all - it seems vital for any non-trivial long-running pipeline, also (and maybe especially) for small teams.)

I have to agree Martin, the checkpoint feature seems like a vital function for pipeline workflows.

We are running jobs which last hours to a day and are hardly to modularize further...

Very disappointing decision.

It is like one step forward and 2 steps backwards. I understand we all need to make money. But this is a genuine request from the community. We are a small team and don't have powerful machines to rerun tests from the beginning. So we still have not yet adopted the groovy code and still running manual pipelines views to resume jobs. Does anybody have a work around for this problem ?

We are using a workaround with a custom step within a pipeline library similar to this:

//restartIt.goovy
def call(Map parameters = [:], body) {
  def timeoutInSeconds = parameters.get('timeoutInSeconds', 900)
  def restart = true
  while (restart) {
    try {
      body()
      restart = false
    } catch (Throwable err) {
      echo "ERROR occured: ${err}"
      //optional email notification
      try {
        timeout(time: timeoutInSeconds, unit: 'SECONDS') {
          input message: 'Do you want to restart?', ok: 'Restart'
        }
      } catch(e) {
        restart = false
        throw err
      }
    }
  }
}

This can the be used like:

restartIt(timeout:1200) {
  node {
    //here the steps you want to possibly restart
  }
}

For sure, there are some caveats. For example, you should use a node to not block executors.

Anyway, it helps us in cases where e.g. deployments sometime fail.

Maybe it helps you as well ...

very misguided decision.  our situation is that we enrolled in the paid Jenkins Team version so we are PAYING customers to Cloudbees AND still don't get access to this very basic feature.  It's really sad and unfair to both customers and the community. Feels like a ripoff

We're looking into options from workarounds mentioned already to jumping ship from Cloudbees+Jenkins entirely:

Note that every other CI/CD system offers this very basic functionality out of the box.  It's that basic:

AWS' offering: https://docs.aws.amazon.com/codepipeline/latest/userguide/actions-retry.html 

CircleCI: https://circleci.com/docs/2.0/workflows/#rerunning-a-workflow-from-a-failed-job

+1 Completely agree with comments above - it's a shame that Jenkins lacks this obvious feature. 

One year later and this issue is already open and not done. In time, I switch to gitlab CI and this does the work even if functions are not at same level. 

Is it possible to not only for after failure? Sometime I need to get redo the same stage, for example building a new build and testing until all pass before going to next step

The plan is to allow this to work for any previously executed stage

jamesdumay that's nice. Thanks James. Hopefully, we can get it ASAP.

BTW, I saw this ticket which already has a PR up (https://issues.jenkins-ci.org/browse/JENKINS-45455) which I am not sure if it is the same thing or not but it looks like it will restart a specific stage from "Previous build". Can we restart the same build? Not kick off another build and run the specific stage?

Ex. 
1) Executed Stage 1
2) Executed Stage 2
3) Re-executed Stage 2
4) Executed Stage 3 ....... and so on