ccaraivan perhaps better covered by JENKINS-36089 or whatever it morphs into.

I agree with ccaraivan this use case could be useful functionality to have.

I'll add that this is critical functionality for the company that I work for. We're looking at reverting to the older Deployment or Build Pipeline view plugins, and if this isn't far away I'd much rather wait for this to be implemented.

Adding my agreement with Costin and others... being able to restart from any stage is crucial.

I am trying to migrate projects to Jenkins and the pipeline really needs this feature.

Our stages are defined as:

Build -> Integration Tests -> Deploy to QA -> Deploy To Live

The 'Deploy to QA' stage has an input() step with a timeout to make it a manual step,

We need to re-run this and subsequent stages when approved by QAs. This is an essential feature that other CI servers provide.

Starting to think this through. Do we want to add an option for "preserve this Pipeline for resuming after the last successful top-level stage (because we're not going to mess around with nested parallel/sequential stages) including workspace etc"? Because we definitely don't want to do all that automatically for every top-level stage of every build - that's a big performance hit to stash everything after every stage, etc. Or do we want to go simpler and allow any Declarative Pipeline to be "resumed" from a given top-level stage, with the Pipeline author responsible for making sure that the workspace is stashed and unstashed, etc? The latter seems more reasonable to me, albeit not perfect.

I think preserving(like pinning) the pipeline is cool idea, I think the expected simple behavior is based on "number of last builds to keep" function or maybe "keep last number of pipelines visible in stage view"

We need this feature : Retry a stage! 

Hrm, I misremembered how stashes work. Unless we explicitly override the normal behavior, all stashes for a build get wiped out when the build finishes. I think what we'll have to do to support reusing stashes from the original build is probably add an option to preserve stashes for a Declarative Pipeline, defaulting to off.

abayer right, something like that. Currently checkpoint skirts the issue by just suppressing stash cleanup unconditionally (and there is no way to instruct Jenkins that a given build has been definitively closed for resumption and storage can be reclaimed, other than deleting the whole build).

For purposes of this feature, if there is no explicit marker that the Pipeline is ready for stage restarting—and perhaps there ought to be?—then you would need to be careful to ensure that most stashes are by default cleaned up most of the time. Perhaps you could clean up all but the last build (which gets tricky if concurrent builds are enabled), or clean up successful builds, etc.

What are the complications with retrying a stage for scripted pipeline?

I have implemented a shared library that allows users to declaratively explain what's inside a repository with some platform specific parameters, so my library then based on that input produces a scripted pipeline, so that we have unified pipelines within the company that covers supported platforms. I think to switch this library to be producing declarative-style DSL would be kind of costly, and I'm not really sure all the things I've been using would be available there.

Is there anything else to read except https://jenkins.io/doc/book/pipeline/syntax/? This is not clear from there what are the benefits and limitations for each approach, i.e. what would be not available for me should I go with Declarative style vs Scripted one. I hope Jenkins team are not trying to put users community into a position of choosing what feature is more important where feature A would be available Scripted only and feature B Declarative only and both are needed. This causes so many confusion and frustration.

jamesdumay - so re: reusing the old build etc. We definitely can't use the same build number - that's just not possible. So BUILD_ID will be different for the restarted build than for the original. But we can make the visualization treat it as one build - by recording that a build is a restarted build of the original build number, and marking top-level stages on the restarted build as skipped for restart until we get to the one we're restarting from, it should be possible for Blue Ocean to "replace" the visualization for the original build, using the original build's visualization for every top-level stage (and their children) that's skipped for restart, and the new build's visualization for everything after (and replacing the original build's links for tests, artifacts, etc with the restarted build's equivalent). How's that sound?

llibicpep - with Declarative, we can guarantee consistency in behavior. Everything but the actual steps will be executed for every single Declarative Pipeline in the same order and way. This makes it much easier to say "Re-run build #X, with the same parameters, SCM revision, checkouts, etc, but don't run any stages until the one we've specified", because we know, for example, how agents, environment variables, etc are handled outside of the actual stages themselves. That means we can treat each top-level stage in a Declarative Pipeline as an atomic entity to some extent, at least enough to know that we can pick back up where we left off without having to do weird Groovy hacks to ensure we're at exactly the right point in a loop or a conditional etc.

And as a general rule, everything in Scripted Pipeline is available in Declarative (though outside of script blocks in Declarative, you can't do things like set/declare variables, use if statements, for loops, etc, but Declarative itself is built on top of Scripted, so you're not going to find steps that work in Scripted but not Declarative, etc), but not necessarily vice versa. That's the whole reason for Declarative - to be able to implement functionality and syntactic sugar that we can't do easily for Scripted because, well, it's a full-blown programming language and covering every possible use case scenario is just not viable.

For those who are watching this ticket, we have JENKINS-48643 to cover the UX for this in Blue Ocean and a few mockups you can view

After reading through this issue, reading up on blue ocean, the checkpoint plugin, etc., I'm still not clear on something...

Will this feature allow for the creation of manual pipeline steps? For example, the pipeline runs up to the production deploy step, but does not invoked it automatically, instead waiting for someone to manually execute that step (without doing the silly "prompt" thing that just causes the pipeline to run forever)?

>Will this feature allow for the creation of manual pipeline steps?

I'm interested in this too, I guess it's possible if the pipeline can be aware of whether it's a fresh run or a restarted run. If there's some kind of condition that works with when blocks, then your pipeline can skip certain stages (probably at the end) and complete as normal.

I've also found the input step isn't a great solution for this pattern.

Something like that could be possible, yeah.

stevenfoster mind creating a new issue that lays your use case out with examples? a ideal Jenkinsfile would be cool too

jamesdumay stevenfoster - don't even bother, I just added a new when condition, isRestartedRun(), to the WIP PR for this that does exactly what it says on the tin. =)

abayer nice. Lets make sure that it gets doc'd too

Yeah, the whole thing is gonna need docs. But I've added it to the scope in the ticket description to make sure we don't forget.

abayer ah whoops, should have checked before submitting

As to the manual pipeline step I feel like it's some misunderstanding happening there. The input step done outside of `node {}` is not really running, it's just the UI renders it as so. Initially I was, like every one else human being, feeling uncomfortable about the change (comparing to the old pipelines). But changing the status quo is not always a bad thing. Given that nothing really running in terms of consuming your resources, and it does not blocks your queue or nodes, and also thinking about your pipeline as a bit of wider process than just a piece of software - it is something really 'in progress' for the whole lifecycle up to prod or till overran by the newest progress (in combination with `milestones` plugin). This is literally represents WIP state and nature of the given version of the artifact - given that DoD is correctly set in your team (availability on prod). It's being built but not in prod yet - let's say it's in QA and some testing happening there - so the team are working on this version of artifact, so the WIP. Then all the interim results (older pipelines that never reached prod because of being aborted by `milestone`) - marked as failed because they are literally failed in terms of that DoD - they never made it to prod. So the new render makes a perfect sense to me.

To extend Steven Foster's comment, would it be possible to create a new step that explicitly exits allowing for restart?  I'm horrible with names, but what if there was a:

hibernate() //intentionally stop the build, allowing it to be resumed later

I know it's just a variation on the normal error() syntax, but it makes more sense to read the code, and the UX wouldn't have to display as an error if it's a deliberate stop.

akrapfl thats what you should use input 'Continue?' for. We are going to allow a stage to define an input without having consume an agent as part of JENKINS-48379.

llibicpep you might be interested in reading JENKINS-48379. It will allow you to define input on a stage without consuming an agent.

Forgot to link the work in progress PR - https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/228

Are there any updates?

This is a big feature for me. Considering the code is almost done, when might it hit the plugin? 

deniskrizanovic, we are working through the user experience design now to be sure that we take various use cases into account. We have an offsite next week with the Blue Ocean and Pipeline teams, and one of our sessions is to define that UX. We'll then need to do that work. I cannot commit to a specific timeline for when it will be merged to the main Blue Ocean plugin, but we are working on it. I will be sure to update this ticket after our offsite, as well as the corresponding ticket for the UX.

So is this a Blue Ocean feature, or a Pipeline feature? I know I would be pleased to have restart capability even if it was outside Blue Ocean, even if it was just an API call.

Thanks for the update Jenn. Will asks a good question. I hope this is a pipeline feature - we are using declarative, but not blue ocean. While Blue Ocean is very pretty and looks very promising, we can't switch until it's all encompassing. But we do need the restart functionality. Or perhaps I'm just confused and the pipeline rendering itself is part of Blue Ocean without even having BO enabled...

seanwcom wsaxon - this will not be Blue Ocean-exclusive - there'll be a classic UI for it, you can interact with it via the REST API, etc. We're just holding up merging/releasing it because we want to make sure we've got the Blue Ocean user experience for stage restarting nailed down and on its way to completion beforehand, so that we know the Blue Ocean side is coming soon and we know that the Declarative side doesn't need to be changed post-release to work with the right approach in the Blue Ocean UX. Coordination is fun. =)

(also we've got to get JENKINS-46809 merged before we can merge this, and that little bugger requires, at a minimum, updates to the Blue Ocean Pipeline editor to handle changes to the intermediate representation of a Declarative Pipeline or it just blows up horribly)

Just to add - this really is a feature that we are looking forward to. Are there any potential estimates of how long 46809 might take to push through? 

vaivars, I cannot commit us to a specific date, except to say that it's our top priority right now.

seanwcom, will you please email me at jbriden@gmail.com so that we can discuss your feedback about the gaps that you see in the pipeline visualization UX that you see in Blue Ocean? I can also connect via the Jenkins IRC or gitter, if those are better for you.

Great feature!

When it says 'Declarative pipeline only' - does it mean strictly only declarative pipeline feature or are script blocks in declarative pipelines supported as well?

Should this be Resolved/Fixed?

Whoops, forgot to close this. Released in Declarative 1.3.

kutzi, this applies to the top-level stage, so it's irrelevant whether there are script blocks within a stage. If the format is syntactically correct declarative, then this will work for you.

We've added support in the declarative plugin. Blue Ocean support is now in development.

Hello, is there a plan to support restarting from sub-stages?

Let's say we have the following stages to build a multi-platform desktop application : 

Build-Win64 -> Test-Win64 -| 
Build-Win32 -> Test-Win32 -|-> Deploy
Build-MacOS -> Test-MacOS -|
Build-Linux -> Test-Linux -|

Where all the 'Build-XXX' are running in parallel within a global 'Build' stage. (The Build&Test stages could take quite a long time)

If let's say 'Build-Win32' fails for some unexpected environmental reasons, you do not want to rebuild the three other one. It would use unnecessary build server time. Same goes if 'Test-Win32' fails, you do not want to rebuild all the tests from the other platforms.

I'm sorry if there is already a way to do so, and if it is the case, could you provide me some guidance on how to achieve such a behavior?

Support for nested stages for parallel builds would be really cool. 

Most of the projects I work on do the heavy lifting in parallel stages and restart the failed part from parallel stage would be perfect.

Hi all,

In my project I use a Singleton-annotated class to store some variables across all stages. This object is not preserved when restarting a stage.

So, my question is: how can I persist some data during the execution so that I can get it later when restarting from a stage?

Thanks in advance.

When restarting a stage, in the console the follwoing output is shown

Restarted from build #4, stage foobar

Is there any way to access this information within the restarted build? Eg via ENV variables or globals in the Jenkinsfile?

franknarf8_ni, the current design restarts at the top-level stage. Please open a separate ticket to request the ability to restart at a parallel stage. 

afolgado - no, sorry - we don't preserve any Groovy state across restarts. If you need to preserve some data, you'd probably want to write it to a file, stash it, and then unstash it in the later stage. Make sure you have preserveStashes() (for the default of preserving the most recent build's stashes - pass a higher number as the argument if you need to preserve more) in the options block of your pipeline block or the stash will be cleared.

Regarding restarting from sub-stages, I created the following issue : https://issues.jenkins-ci.org/browse/JENKINS-52391

Please upvote if you are also looking forward to this feature.

Is there a way to always run a stage when restarting stages? Our pipeline requires a few things to be obtained dynamically in the beginning and when restarting stages we don't get that functionality.

I think something like a pre section (like post) would be really helpful here.

Example:

Stage A -> Stage B -> Stage C -> Stage D

I want to restart Stage C or Stage D but I want to ensure that Stage A always runs first.

I might be wrong, but I don't get how to switch off the restart stage option.

My pipelines currently don't work correctly after restarting a stage (I check out git only at first stage...)

There currently isn't a way to disable stage restart, but that's a good idea for us to add - could you create a new ticket for that? Thanks!

hoppej abayer please update this thread if you created a ticket to disable stage restart. We desperately need this.

This feature does not work at all with my pipelines (it does not run on the same node as the original run), so it's generating false positives and wreaking havoc with my customers. I'm considering implementing some custom CSS to hide the silly thing before it causes us any more grief!

ben5556 - No, we haven't found a way around this, so we're not taking advantage of stage restarting. I have use boolean parameters which can be unchecked to skip stages - which works as a sort of poor mans stage restarting.

To expand on my previous example, I have check boxes as Job parameters for Stage B, C, D. I always want Stage A to run so there's no check box for that. If I want to restart Stage D, I uncheck the boxes for Stages B, C. Each stage in the pipeline checks the true/false value before running the stage.

It's not ideal, and it's not really taking advantage of this feature, but it works pretty well.

I also did create https://issues.jenkins-ci.org/browse/JENKINS-53167, so you might want to watch that.

bdevcich: I have also voted for the ticket JENKINS-53167. For the workaround that you have in place, have you tied the job to a particular slave and use the same workspace each time the job runs? I am asking this because if a job ran the build and test phases successfully and then failed during packaging phase, how will you restart job and ensure that it picks up the already built binary for packaging phase.

I am also looking for running a particular phase each time (or may be having the option to run it whenever i want) irrespective of whichever phase i restart my job from. I have included my use case in your ticket JENKINS-53167

gc: Yes, in this particular case, we have it tied to a slave for hardware dependency reasons.

Bulk closing resolved issues.