svanoort ideally we'd only like to show the specific stage that is unstable when the pipeline is unstable (not all stages). Is that feasible?

Its going to be fixed in JENKINS-39296, closing as duplicate.

This is not a duplicate, or else that other one does not fix this.

michaelneale Sorry, yes you are right. What I fixed is all steps showing it's own status.

yes this one isn't blue ocean specific either it seems.

svanoort Bismuth API returns all stage's status as run's status - that is if the build is completed.

StatusAndtiming.computeChunkStatus(@Nonnull WorkflowRun run,
                                                   @CheckForNull FlowNode before, @Nonnull FlowNode firstNode,
                                                   @Nonnull FlowNode lastNode, @CheckForNull FlowNode after)

I am using this method to compute status of chunk(stage). This method however does not seem to be doing the right thing. What I expect to do is to compute status of a chunk by inferring status based on the steps run inside it.

See https://github.com/svanoort/pipeline-graph-analysis-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/pipelinegraphanalysis/StatusAndTiming.java#L158.

My blueocean specific (pre-bismuth) implementation used logic to compute status of stage or parallel using underlying steps and last node of parallel branch and blocked stage.

• stage/parallel is FAILURE if at least one of more node inside it failed or last node of chunk had error.
• stage/parallel is RUNNING if at least one step inside it running
• stage/parallel is FINISHED if no steps inside it are running
• Else if FINISHED and no error on any step or last node on the chunk had no error it was SUCCESS

There doesn't seem to be a way to get ABORTED status on step so we can't really compute ABORTED stage. For all error pipeline API gives AbortException so there is no way to distinguish user aborted vs script FAILURE.

For UNSTABLE result, is there way to get step level UNSTABLE state?

Anyways, looks like I can't really use StatusAndTiming status APIs to get status correctly. Is this something you plan to implement, otherwise I will have to take care in blueocean code?

@vivek This is the expected and documented behavior - https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Stage+View+Plugin#PipelineStageViewPlugin-1.6%28July18%2C2016%29

There's no way for the pipeline to identify which step set the build status to unstable, so we have to assume it was one of the otherwise successful steps.

> There doesn't seem to be a way to get ABORTED status on step so we can't really compute ABORTED stage. For all error pipeline API gives AbortException so there is no way to distinguish user aborted vs script FAILURE.

If your step is the final step of the build and the build was aborted, then your step is aborted. Otherwise you are correct: there's no way to identify a failed versus manually aborted step. I'm not sure why you're saying that the status and timing APIs are the problem here – how would you propose I work around the data model limitation?

> stage/parallel is FAILURE if at least one of more node inside it failed or last node of chunk had error.

No, a stage/parallel is a FAILURE if the final node in it had an error – otherwise the error was handled in a try/catch block. This was the same mistake that legacy Stage View made. Unfortunately jglick shot down my proposal to properly track this status in pipeline since this is not the best way to handle this.

I've told jglick time and time again that our handling of status coding for steps is grossly inadequate, but still he closes proposals to add an unstable status to a step.

The right way to handle this is definitely JENKINS-26522 + step-level support for its custom statuses, but unfortunately that has been blocked by Blue Ocean itself.

There's no way for the pipeline to identify which step set the build status to unstable

Well, if the script just ran currentBuild.result = 'UNSTABLE' or a step called Run.setResult that is true; we would need a proper implementation of JENKINS-26522. (I am only opposed to partial implementations which do not actually update at least one UI to use the capability.) More likely we would want JENKINS-27395, which is what would accomplish the actual goal specified in the initial issue description here. In fact I think this is just a duplicate. Again the issue is not implementing the step & data model, which is trivial, but deciding which UIs should interpret that data and how.

For the case of ABORTED, typically this is set not due to a call to Run.setResult inside the script, but rather the program as a whole terminating with FlowInterruptedException. (That API supports any Result, but ABORTED is the most common.) In such cases you can tell where the exception was thrown by inspecting ErrorAction.

kzantow I would point out that this idiom from the master branch makes no sense at all. You could just delete the try and finally blocks and the result would still be FAILURE—but more easily analyzed.

Thanks svanoort.

>I'm not sure why you're saying that the status and timing APIs are the problem here – how would you propose I work around the data model limitation?

I didn't say StatusAndtiming is a problem. What I meant was, these API doesn't give how blueocean wants to repot per state status. For example, if overall run is UNSTABLE, this API will give UNSTABLE for all steps. Where in blueocean we want to repot per set status by computing if that step had any error or it's still running or not.

>No, a stage/parallel is a FAILURE if the final node in it had an error – otherwise the error was handled in a try/catch block. This was the same mistake that legacy Stage View made. Unfortunately Jesse Glick shot down my proposal to properly track this status in pipeline since this is not the best way to handle this.

Yes, makes sense.

>The right way to handle this is definitely JENKINS-26522 + step-level support for its custom statuses, but unfortunately that has been blocked by Blue Ocean itself.

I guess I need more context here, just so that blueocean can unblock it

jglick

For the case of ABORTED, typically this is set not due to a call to Run.setResult inside the script, but rather the program as a whole terminating with FlowInterruptedException. (That API supports any Result, but ABORTED is the most common.) In such cases you can tell where the exception was thrown by inspecting ErrorAction.

Thanks, that is a proposal that might solve that aspect. I think my concern is that we're building up a set of fairly complex rules now for what is the "status" of a FlowNode. Rather than doing that, we should just add a status coding (probably JENKINS-26522) and have steps set it where needed – we can cut out so much complexity very fast, and give more data and control to the user.

> Where in blueocean we want to repot per set status by computing if that step had any error or it's still running or not.

So, we still do this with the StatusAndTiming APIs – if you want to hide the uncertain UNSTABLE assignment until there's a better mechanism for attributing it, one solution is to coerce UNSTABLE status to SUCCESS (completed) in your mapping (always legal IIRC).

> I guess I need more context here, just so that blueocean can unblock it

Primarily that Stage View would be the logical case to demo a new JENKINS-26522 API in the UI and we don't do enhancements to it generally now. However if we just did explicit status coding with this and not tags / other metadata, then that's purely a change to pipeline-graph-analysis plugin.

svanoort no https://issues.jenkins-ci.org/browse/JENKINS-26522 would not fix it - it requires the users to do some work (likely after they have a build with test failures) - unless you mean that test reports could also use the same facility? If so (or if currentBuild.result setter would do it) then yes it might. There would be no GUI change on the blue ocean side needed to accomodate this.

jglick no I don't think this is a duplicate of having multiple test results on multiple stages (https://issues.jenkins-ci.org/browse/JENKINS-27395) - this is just about unstable status (there may or may not be tests involved, of course most of the time there is so...).

If JENKINS-26522 was implemented, there would be no GUI change required on blue ocean at least for it to work.

there may or may not be tests involved

There is not going to be any change to the semantics of setting currentBuild.result. That sets the overall build result, period.

The (vague) proposals being floated would allow the junit step to (somehow) include a test set identifier which would “scope” the results; and/or to allow a user to explicitly mark status (and other things—badges, comments, …) on some block, probably a stage. IIUC neither would affect the behavior of existing scripts: you would need to opt in somehow.

> There is not going to be any change to the semantics of setting currentBuild.result.

This has no relationship, because it completely bypasses the flow graph to directly modify the run result (making it something of a dirty hack). JUnit would only overlap with this slightly – because it would set an explicit step status.

Users want a way to set more discrete and more complex statuses on steps. Blue Ocean wants a way to do this. Rather than trying to hold back the tide, let's talk concrete proposals for how to make this happen: I'm not sure where jglick is getting "vague" from but I've laid out a concrete implementation proposal and a couple of proposed syntaxes in JENKINS-26522.

michaelneale I propose we close this one as duplicate and shuffle discussion there. Objections?

michaelneale To clarify: JUnit would set the explicit status coding via JENKINS-26522 on the appropriate step(s) - UNSTABLE for example. Then the status-determining logic in pipeline graph analysis plugin would pick this status coding up and apply it to appropriate step(s).

Thanks svanoort jglick that explanation makes sense to me, I think I get it now.
So yes, in that case, closing this in favour of JENKINS-26522

We are going to reopen this so the Blue Ocean team can track JENKINS-26522 and verify that it works as expected when it is completed.

Sorry for the updatespam – I had this confused with another similarly named bug/RFE that this appears to clone. In any case, jamesdumay I would not expect this to be a trivial change since you need to rework the way statuses are tracked.

Basically, you can increase priority as much as you like, but it needs a significant data model change to do it the right way. If you choose to do it the wrong way, that will have to be purely Blue Ocean side and supported, and we can make no guarantees it will remain functional in the face of future pipeline changes. If you do it the right way, it will take a while though.

Thanks svanoort! I understand this one is a difficult one to implement and its going to take its time. Im just adjusting tickets that are important to Blue Oceans 12 month roadmap.

Is it really only depending on Blue Ocean? We have the exact same behaviour in the legacy view.

All stages are shown unstable while only one step failed a specific stage (rending unstable). In the other hand, the stage that failed the build is shown "failed". Is it a different issue? 

shaiton no it doesn't really - its part of the model of pipeline, so not something blue ocean can solve (as you have guessed)

Any updates on this issue? From a develop's perpective it's weird to see all build steps unstable when there is only one step failing. So it's more time consuming as they need more time to identify where the problem occurred.

mdelapenya nothing to try yet - but most of the work is reworking pipeline to have fine grained statuses (which is non trivial, touches a lot of code). The UI side is relatively trivial once that is done. 

Testing Notes

• From the UI side, once implemented, automated test(s) using a predefined set of Pipelines would be valuable. The test could include the use of "unstable," "successful," and "only-stage-seven-fails" pipelines, for example, and verify that the correct statuses are reported.
• From the Pipeline side, unit tests should be made part of the PR. Considering the folks working on this change, I've got every confidence that it'll be well tested in this way.

This is a serious issue we have with the blue ocean and the legacy view, to the point that developers are asking to go back and not use "pipeline".

It would be really helpful if you can solve this issue.

eparimi You can always manually set the build result to SUCCESS rather than UNSTABLE (see: https://support.cloudbees.com/hc/en-us/articles/218554077-How-to-set-current-build-result-in-Pipeline-). 

abayer Has been working on this on and off for many months, because it actually requires adding a complex set of new features rather than just a simple bugfix, and we don't have a consensus on how to best support this.  We basically have to radically change how statuses are determined.

svanoort As far as I know, It's not possible to recover from an UNSTABLE state using currentBuild.result:

node {
  stage('Stage 1') {
    echo 'foo'
    currentBuild.result = 'SUCCESS'
  }
  stage('Stage 2') {
    echo 'foo'
    currentBuild.result = 'UNSTABLE'
  }
  stage('Stage 3') {
    echo 'foo'
    currentBuild.result = 'SUCCESS'
  } 
}

Results in UNSTABLE status for the whole pipeline, and all the stages appearing yellow colored.

luispiedra I don't think we are going to get a workaround for this. The work here to fix statuses needs to happen so it all works properly.

Hello,

Any updates on this issue plz? It's weird to see all build steps unstable when there is only one step failing.

This is a serious issue we have with the blue ocean view

hbedoui we still plan on fixing this but no ETA yet. Requires a big rework of how statuses are reporting in Pipeline, so its not as simple as it might seem.

So, we've been slowly adopting the new Pipeline APIs for all of our production builds across dozens of development teams. This build status issue is causing a very large amount of frustration and wastes a lot of developer time trying to dig through logs to deduce failure statuses. As such I'm hoping this is being considered a high-priroity item to be fixed regardless of how difficult it may be to do.

That small rant aside, might I suggest that if the "correct" fix for this turns out to be something incredibly complex and difficult - which, again, suggests that it may be months or years before it is fixed - could you not find some reasonable substitution or workaround for the problem in the meantime? Like, for example, could you perhaps create a clone of the `stage()` function, say, `manualStatusStage()`, that maybe ignores all of the current build status logic completely but allows the build status to be explicitly set using a custom variable or method, something like `currentBuild.hackStagedResult = 'SUCCESS"`? In this way users that are experiencing this pain point would have some temporary workaround to ease the current pain while allowing you some time to fix the underlying problem correctly.

NOTE: I'd even be happy if these "hacks" or "workarounds" were simply undocumented features that you put in place as temporary workarounds to this specific JIRA issue, with the expectation that once the underlying problems are fixed the "undocumented" functionality will simply be removed - and hence, no need to be maintained going forward.

Thoughts?

leedega Believe me, we know exactly how irritating these limitations of the Pipeline architecture are because we "drink our own champagne" internally and use test result reporting extensively in Jenkins Pipelines. The problem is that this is directly tied to the limitations of the Pipeline data model, which internally is pass/fail – everything else is computed in a logical way on the basis of ErrorActions, Input Actions, whether or not the FlowNode in question is the tip of a incomplete build (In-progress).

Anything that modifies how we store status is unfortunately a permanent change because it would have to work by recording data in the model, either by Actions on FlowNodes, or by some internal state, or both. Thus any "hacks" we added would be a part of Pipeline forever – we could ignore the extra data but it would introduce compatibility issues.

The extra level of complexity is that any behavior needs to be consistent across (potentially nested) parallels, nested blocks, and has to have something like a try/catch construct to let users override results – which has to behave logically in the face of a full & potentially complex graph structure. Take a look at Pipeline Graph Analysis plugin and you'll see what I mean about the complexity (also the APIs for working with the graph in https://github.com/jenkinsci/workflow-api-plugin/tree/master/src/main/java/org/jenkinsci/plugins/workflow/graphanalysis).

The simple fix would be to simply not display an UNSTABLE status at the UI level in any situation for Pipelines, only at the overall build level. Would that be preferred? Because that's a change we can do without too much extra hassle.

Then your architecture is wrong. It cannot support something fundamentally needed. The sooner you change it, the better.
This project goes nowhere without this.

leedega Perhaps someone (you or someone else) can implement such a “hack” as a plugin – probably with a fat disclaimer indicating that many of the scenarios (as mentioned by svanoort) are not covered.

That would at least reduce the pain a bit, and it would “buy some time” to get the real solution in place.

svanoort: Your suggestion to 

[…] simply not display an UNSTABLE status at the UI level in any situation […]

lqbweb It is a high priority to address this and in my personal upcoming work, but right now the highest priority is ensuring Pipeline does not cause system stability problems.

As far as the architecture: I took the time to write up a detailed explanation of the situation in hopes it might lead to something productive (maybe even a PR), and it really is not constructive to simply say "your architecture is wrong." While you're entitled to your opinion, let me ask you this: how would you feel if you were putting many of your nights and weekends into improving something and someone said that to you? This is an open source project and we only have so many engineers no matter how hard we work – if you think you could improve upon the architecture you're more than welcome to submit PRs that accomplish this goal.

znerd

Perhaps someone (you or someone else) can implement such a “hack” as a plugin – probably with a fat disclaimer indicating that many of the scenarios (as mentioned by Sam Van Oort) are not covered

I was actually going to suggest that. Some of those experiments can provide new ideas for how to implement a comprehensive feature – and if they work really well they may even be requested to directly merge into the core of Pipeline. More often they provide inspiration that leads one of us to hack out the ultimate solution.

Personally, I absolutely hate that we can't easily address this improvement, but past lessons have also taught us that a maintainable and robust solution is much better for this kind of challenge than a quick-and-dirty solution that opens up problems down the road. Maybe someone will be able to come up with a solution that does not impose those risks though.

 This build status issue is causing a very large amount of frustration and wastes a lot of developer time trying to dig through logs to deduce failure statuses

leedega we have had the same issue. Today we work extensively with Slack notifications that contain links to html reports (this is more important than just showing the correctly coloured box anyway, because even with that the developer would still have to check the log file for the actual root cause). 

In general I think it is great to have a per-pipeline-run build status - this probably shouldn't even be refactored: There are quite some use cases that need that: Notifications to source control if a feature branch can be merged, notifications about "back to normal", just showing a color in the build list in the left panel. That said I think it would be great to have additionally a per stage status, but not only for success/unstable/failed but with a much richer data model - it could e.g. be a list of info, warn and error messages that can be reduced to info/warn/error counts and even more reduced to success/unstable/failed per stage. If that was implemented as stage.info(...), stage.warn(...), stage.error(...) it could under the hood automatically set the per-pipeline-run build status. Plugins could without hassle one by one switch to the new way of setting the build status (while the old way stays intact).

All that being said, if for a certain use case it is really important to only "change the color of one box", there is currently one way to achieve it:

stage('Stage 1') { echo "green" }
try {
  stage('Stage 2') { throw new IllegalStateException("a failed stage") }
} catch(err) { echo "made the stage only red" }
stage('Stage 3') { echo "green" }
try {
  stage('Stage 4') {  throw new IllegalStateException("a failed stage") }
} catch(err) { echo "made the stage only red" }
stage('Stage 5') { echo "green" }

Will get you

svanoort As people's grief lies mostly in the UI presentation as it seems (and not in the underlying data model), maybe it could be a very simple change in the pipeline core plugin to show yellow for a predefined exception?

ghenzler Is it possible to do the same in declarative pipeline?

I'm failing to understand how this issue is any less than Critical.  It completely invalidates the result of any and every pipeline job run that has even the smallest subset of stage failures.   The problem even propagates outside of Jenkins and Blue Ocean as it leads to other plugins such as the Pipeline Githubnotify Step Plugin incorrectly setting FAILURE status on GitHub PR commit statuses set from stages in the pipeline.

This issue leads to developers having to waste time and drive up frustration having to hunt through the stages of their pipeline job runs to figure out which stage actually did fail.

Additionally, it's really disheartening to see that this issue is only as far as Planned on the Blue Ocean Roadmap. Surely the 116 (or more) votes and 143 (or more) watchers on this issue demonstrates how important it is for a lot of people.  Other items on the roadmap that are In Progress even, such as JENKINS-44699 don't have anywhere near as many votes and/or watchers.

Is there any ETA, no matter how much of guesstimate it is, available as to how long this issue will take to be fixed?

This critical issue is maddening. It's almost as if we have to separate our pipeline into small components (jobs) like we did with freestyle jobs, for the build status reporting to make sense.

Indeed yes, it is critical. This has somehow slipped. It was looked at some time ago, but unfortunately is a HUGE amount of work as there is no way in the pipeline engine to track this information (especially not if people are using scripted pipelines, which are still popular, and were included at the time). It is not at all a small job

cc vivek jbriden this somehow has slipped off the radar. It was on the old roadmap but don't know the status of it now. Given how old it is, it may be more feasible to do this in a declarative only context now and actually be possible. 

ghenzler - Thanks for the suggestion on using IllegalStateException to turn single stage red. Does this work for the Blue Ocean UI as well?

michaelneale - Are you suggesting that the scope of this issue be narrowed to declarative syntax? I would imagine that many of the voters and watchers of this issue are using scripted pipelines. I know we are, as there are many advanced scenarios that simply are not feasible using declarative syntax.

While I can appreciate the desire for a single solution to address both Pipeline styles, if a solution for one of the styles is relatively easy while the other quite complex, let's not let the perfect be the enemy of the good here.  Please.

Fully agree with brianjmurrell. But then should we split this ticket in two? Then…

• …one of them (declarative) could be picked up and resolved,
• …while the other one (scripted) could potentially be implemented on a different schedule.

An additional benefit could be that when we've got the solution in place for the declarative pipeline, the challenge for implementing the fix for scripted could be reduced, because we already have a guiding implementation in place (the one for declarative).

I do not see any reason to think that this issue is specific to Declarative vs. Scripted. Nor do I think it is necessarily a “huge amount of work”. There is already an API to indicate that a step failed: ErrorAction. This is added implicitly when StepContext.onFailure is called. The overall build status is set to failed (or some other statuses, acc. to FlowInterruptedException) if the FlowEndNode is marked this way, typically because the error was not caught and handled.

To set a less severe result, there is a StepContext.setResult API. Unfortunately the implementation in CpsStepContext simply sets an overall build status, which is not useful, and anyway JUnitResultsStepExecution does not call it. There could perhaps be a new action type retroactively implemented by ErrorAction, or somehow related to it, which would be added to a FlowNode when setResult is called and which just defines a getResult method.

This would allow JUnitResultsStepExecution to set a per-step UNSTABLE status instead of a build result, at least optionally. (I think this would need to be opt-in for compatibility reasons, unless you declare that setting a step result also makes the build result at least as bad, which may be unwanted.) Then FlowNode.getIconColor would honor the new action, causing Pipeline Steps in the classic UI to show yellow balls where appropriate, and Blue Ocean would need to be patched to do the same.

Is any progress being made on this issue?

I don't want to sound impatient but this is a pretty seriously debilitating issue!  Way beyond even just the graphical display of an entire pipeline of stages and steps looking like it's unstable, this issue completely defeats any and all {{post { }}} block processing after the first stage failure.  Every stage block from the point of a failed stage on gets processed as unstable, preventing one from doing any kind of finer (than entire pipeline) granulated processing and/or reporting of pipelines.

I now personally believe this issue to be higher than Critical and believe it to be an outright Blocker as it makes much of the usefulness of Pipelines void.  It makes me want to break my pipeline down into separate free-style jobs just so that I can get meaningful reporting and processing of jobs based on whether they are successful, unstable, failed, etc.  A bug serious enough to make one want to abandon Pipelines is by definition a Blocker to Pipelines.

So, can we get a status update on this?  Is anyone actively working on it or has it been thrown back on the pile?

Same here. Situation with post actions is very annoying for us. It's broking all logic of pipeline.

brianjmurrell zaharovdi Short term, what we can do is roll back the support for UNSTABLE status. That means you never have to worry about this "bug" because we don't try to signal that a stage somewhere set that status – we simply go back to ignoring it entirely. What do you think?

In the longer-term, we're building a new Pipeline engine that will replace CPS and provide solutions to a lot of painful issues (sandboxing, lack of an integrated debugger, stability and resource-use problems) and we're planning to address the way we store Step statuses there so that this ceases to be a problem. However I can't give you a firm timeline for that yet.

For those who will ask "Why isn't this super easy? Why can't it be done tomorrow?!" – the answer is: "There's nothing actually storing when we set unstable status. And storing that is deceptively complex because statuses can interact and carries a huge risk of regressions." In fact, all the statuses besides success and failure are computed based on a pretty complex and delicate process. Much, much, much more is happening under the surface than users will actually see.

abayer Would it be viable to add a custom "UNSTABLE OR HIGHER" post condition for the Declarative post conditions? I think that would resolve the part of this that seems to be serious (breaking post behaviors that expect success only).

zakharovdi brianjmurrell I've opened JENKINS-53889 which I think would resolve your biggest issue here by making it easy to catch and handle UNSTABLE statuses in declarative.