Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39203

All stages show up as UNSTABLE when only one stage should

    • Blue Ocean 1.2-beta2, Blue Ocean 1.2-beta3, Blue Ocean 1.2-beta4, Blue Ocean 1.2, Blue Ocean 1.3, Blue Ocean 1.4 - beta 1
    • Blue Ocean 1.17.0, Pipeline: API 2.34, Pipeline: Basic Steps 2.18, Pipeline: Graph Analysis 1.10, Pipeline: Groovy 2.70, Pipeline: Declarative 1.3.9, JUnit: 1.28, Warnings Next Generation 5.2.0

      Problem
      When there is a build which has a stage that marks the build as unstable, all the stages, parallels and steps are marked incorrectly as unstable than just the unstable stage, parallel and step that caused the Pipeline to be unstable.

      To reproduce

      1. Build the multibranch pipeline "kzantow/failure-project" from github
      2. Look at the "michaelneale" branch
      3. Note that all stages are unstable (check the api json, all stages are UNSTABLE but should not be, only the final stage should be).

          [JENKINS-39203] All stages show up as UNSTABLE when only one stage should

          Taylor Patton added a comment -

          This critical issue is maddening. It's almost as if we have to separate our pipeline into small components (jobs) like we did with freestyle jobs, for the build status reporting to make sense.

          Taylor Patton added a comment - This critical issue is maddening. It's almost as if we have to separate our pipeline into small components (jobs) like we did with freestyle jobs, for the build status reporting to make sense.

          Michael Neale added a comment -

          Indeed yes, it is critical. This has somehow slipped. It was looked at some time ago, but unfortunately is a HUGE amount of work as there is no way in the pipeline engine to track this information (especially not if people are using scripted pipelines, which are still popular, and were included at the time). It is not at all a small job

           

          cc vivek jbriden this somehow has slipped off the radar. It was on the old roadmap but don't know the status of it now. Given how old it is, it may be more feasible to do this in a declarative only context now and actually be possible. 

          Michael Neale added a comment - Indeed yes, it is critical. This has somehow slipped. It was looked at some time ago, but unfortunately is a HUGE amount of work as there is no way in the pipeline engine to track this information (especially not if people are using scripted pipelines, which are still popular, and were included at the time). It is not at all a small job   cc vivek jbriden this somehow has slipped off the radar. It was on the old roadmap but don't know the status of it now. Given how old it is, it may be more feasible to do this in a declarative only context now and actually be possible. 

          ghenzler - Thanks for the suggestion on using IllegalStateException to turn single stage red. Does this work for the Blue Ocean UI as well?

          michaelneale - Are you suggesting that the scope of this issue be narrowed to declarative syntax? I would imagine that many of the voters and watchers of this issue are using scripted pipelines. I know we are, as there are many advanced scenarios that simply are not feasible using declarative syntax.

          Brian Villanueva added a comment - ghenzler - Thanks for the suggestion on using IllegalStateException to turn single stage red. Does this work for the Blue Ocean UI as well? michaelneale - Are you suggesting that the scope of this issue be narrowed to declarative syntax? I would imagine that many of the voters and watchers of this issue are using scripted pipelines. I know we are, as there are many advanced scenarios that simply are not feasible using declarative syntax.

          While I can appreciate the desire for a single solution to address both Pipeline styles, if a solution for one of the styles is relatively easy while the other quite complex, let's not let the perfect be the enemy of the good here.  Please.

          Brian J Murrell added a comment - While I can appreciate the desire for a single solution to address both Pipeline styles, if a solution for one of the styles is relatively easy while the other quite complex, let's not let the perfect be the enemy of the good here.  Please.

          Ernst de Haan added a comment -

          Fully agree with brianjmurrell. But then should we split this ticket in two? Then…

          • …one of them (declarative) could be picked up and resolved,
          • …while the other one (scripted) could potentially be implemented on a different schedule.

          An additional benefit could be that when we've got the solution in place for the declarative pipeline, the challenge for implementing the fix for scripted could be reduced, because we already have a guiding implementation in place (the one for declarative).

          Ernst de Haan added a comment - Fully agree with brianjmurrell . But then should we split this ticket in two? Then… …one of them (declarative) could be picked up and resolved, …while the other one (scripted) could potentially be implemented on a different schedule. An additional benefit could be that when we've got the solution in place for the declarative pipeline, the challenge for implementing the fix for scripted could be reduced, because we already have a guiding implementation in place (the one for declarative).

          Jesse Glick added a comment -

          I do not see any reason to think that this issue is specific to Declarative vs. Scripted. Nor do I think it is necessarily a “huge amount of work”. There is already an API to indicate that a step failed: ErrorAction. This is added implicitly when StepContext.onFailure is called. The overall build status is set to failed (or some other statuses, acc. to FlowInterruptedException) if the FlowEndNode is marked this way, typically because the error was not caught and handled.

          To set a less severe result, there is a StepContext.setResult API. Unfortunately the implementation in CpsStepContext simply sets an overall build status, which is not useful, and anyway JUnitResultsStepExecution does not call it. There could perhaps be a new action type retroactively implemented by ErrorAction, or somehow related to it, which would be added to a FlowNode when setResult is called and which just defines a getResult method.

          This would allow JUnitResultsStepExecution to set a per-step UNSTABLE status instead of a build result, at least optionally. (I think this would need to be opt-in for compatibility reasons, unless you declare that setting a step result also makes the build result at least as bad, which may be unwanted.) Then FlowNode.getIconColor would honor the new action, causing Pipeline Steps in the classic UI to show yellow balls where appropriate, and Blue Ocean would need to be patched to do the same.

          Jesse Glick added a comment - I do not see any reason to think that this issue is specific to Declarative vs. Scripted. Nor do I think it is necessarily a “huge amount of work”. There is already an API to indicate that a step failed : ErrorAction . This is added implicitly when StepContext.onFailure is called. The overall build status is set to failed (or some other statuses, acc. to FlowInterruptedException ) if the FlowEndNode is marked this way, typically because the error was not caught and handled. To set a less severe result, there is a StepContext.setResult API. Unfortunately the implementation in CpsStepContext simply sets an overall build status, which is not useful, and anyway JUnitResultsStepExecution does not call it. There could perhaps be a new action type retroactively implemented by ErrorAction , or somehow related to it, which would be added to a FlowNode when setResult is called and which just defines a getResult method. This would allow JUnitResultsStepExecution to set a per-step UNSTABLE status instead of a build result, at least optionally. (I think this would need to be opt-in for compatibility reasons, unless you declare that setting a step result also makes the build result at least as bad, which may be unwanted.) Then FlowNode.getIconColor would honor the new action, causing Pipeline Steps in the classic UI to show yellow balls where appropriate, and Blue Ocean would need to be patched to do the same.

          Brian J Murrell added a comment - - edited

          Is any progress being made on this issue?

          I don't want to sound impatient but this is a pretty seriously debilitating issue!  Way beyond even just the graphical display of an entire pipeline of stages and steps looking like it's unstable, this issue completely defeats any and all {{post { }}} block processing after the first stage failure.  Every stage block from the point of a failed stage on gets processed as unstable, preventing one from doing any kind of finer (than entire pipeline) granulated processing and/or reporting of pipelines.

          I now personally believe this issue to be higher than Critical and believe it to be an outright Blocker as it makes much of the usefulness of Pipelines void.  It makes me want to break my pipeline down into separate free-style jobs just so that I can get meaningful reporting and processing of jobs based on whether they are successful, unstable, failed, etc.  A bug serious enough to make one want to abandon Pipelines is by definition a Blocker to Pipelines.

          So, can we get a status update on this?  Is anyone actively working on it or has it been thrown back on the pile?

          Brian J Murrell added a comment - - edited Is any progress being made on this issue? I don't want to sound impatient but this is a pretty seriously debilitating issue!  Way beyond even just the graphical display of an entire pipeline of stages and steps looking like it's unstable, this issue completely defeats any and all {{post { }}} block processing after the first stage failure.  Every stage block from the point of a failed stage on gets processed as unstable, preventing one from doing any kind of finer (than entire pipeline) granulated processing and/or reporting of pipelines. I now personally believe this issue to be higher than Critical and believe it to be an outright Blocker as it makes much of the usefulness of Pipelines void.  It makes me want to break my pipeline down into separate free-style jobs just so that I can get meaningful reporting and processing of jobs based on whether they are successful, unstable, failed, etc.  A bug serious enough to make one want to abandon Pipelines is by definition a Blocker to Pipelines. So, can we get a status update on this?  Is anyone actively working on it or has it been thrown back on the pile?

          Same here. Situation with post actions is very annoying for us. It's broking all logic of pipeline.

          Denis Zakharov added a comment - Same here. Situation with post actions is very annoying for us. It's broking all logic of pipeline.

          Sam Van Oort added a comment -

          brianjmurrell zaharovdi Short term, what we can do is roll back the support for UNSTABLE status. That means you never have to worry about this "bug" because we don't try to signal that a stage somewhere set that status – we simply go back to ignoring it entirely. What do you think?

          In the longer-term, we're building a new Pipeline engine that will replace CPS and provide solutions to a lot of painful issues (sandboxing, lack of an integrated debugger, stability and resource-use problems) and we're planning to address the way we store Step statuses there so that this ceases to be a problem. However I can't give you a firm timeline for that yet.

          For those who will ask "Why isn't this super easy? Why can't it be done tomorrow?!" – the answer is: "There's nothing actually storing when we set unstable status. And storing that is deceptively complex because statuses can interact and carries a huge risk of regressions." In fact, all the statuses besides success and failure are computed based on a pretty complex and delicate process. Much, much, much more is happening under the surface than users will actually see.

          abayer Would it be viable to add a custom "UNSTABLE OR HIGHER" post condition for the Declarative post conditions? I think that would resolve the part of this that seems to be serious (breaking post behaviors that expect success only).

          Sam Van Oort added a comment - brianjmurrell zaharovdi Short term, what we can do is roll back the support for UNSTABLE status. That means you never have to worry about this "bug" because we don't try to signal that a stage somewhere set that status – we simply go back to ignoring it entirely. What do you think? In the longer-term, we're building a new Pipeline engine that will replace CPS and provide solutions to a lot of painful issues (sandboxing, lack of an integrated debugger, stability and resource-use problems) and we're planning to address the way we store Step statuses there so that this ceases to be a problem. However I can't give you a firm timeline for that yet. For those who will ask "Why isn't this super easy? Why can't it be done tomorrow?!" – the answer is: "There's nothing actually storing when we set unstable status. And storing that is deceptively complex because statuses can interact and carries a huge risk of regressions." In fact, all the statuses besides success and failure are computed based on a pretty complex and delicate process. Much, much, much more is happening under the surface than users will actually see. abayer Would it be viable to add a custom "UNSTABLE OR HIGHER" post condition for the Declarative post conditions? I think that would resolve the part of this that seems to be serious (breaking post behaviors that expect success only).

          Sam Van Oort added a comment -

          zakharovdi brianjmurrell I've opened JENKINS-53889 which I think would resolve your biggest issue here by making it easy to catch and handle UNSTABLE statuses in declarative.

          Sam Van Oort added a comment - zakharovdi brianjmurrell I've opened JENKINS-53889 which I think would resolve your biggest issue here by making it easy to catch and handle UNSTABLE statuses in declarative.

          Short term, what we can do is roll back the support for UNSTABLE status. That means you never have to worry about this "bug" because we don't try to signal that a stage somewhere set that status – we simply go back to ignoring it entirely. What do you think?

          svanoort What is the value of currentBuild.currentResult at various stages with this proposal? Say I have 3 parallel stages A, B, and C and A is SUCCESSful and B is UNSTABLE and C is SUCCESSful. Will I be able to determine in stage C that it is SUCCESSful even though stage B was UNSTABLE? Because that is what currently doesn't happen. Currently, as soon as B hits UNSTABLE, every stage after it is UNSTABLE even if it's not really.

          Brian J Murrell added a comment - Short term, what we can do is roll back the support for UNSTABLE status. That means you never have to worry about this "bug" because we don't try to signal that a stage somewhere set that status – we simply go back to ignoring it entirely. What do you think? svanoort What is the value of currentBuild.currentResult at various stages with this proposal? Say I have 3 parallel stages A, B, and C and A is SUCCESSful and B is UNSTABLE and C is SUCCESSful. Will I be able to determine in stage C that it is SUCCESSful even though stage B was UNSTABLE? Because that is what currently doesn't happen. Currently, as soon as B hits UNSTABLE, every stage after it is UNSTABLE even if it's not really.

          Sam Van Oort added a comment -

          brianjmurrell currentBuild.currentResult would be entirely unchanged. It has no representation within the Pipeline data model (it directly modifies the build). What this would do is ensure that no stages show up as "unstable" for you. Which gives you what you are asking for.

          If you want the ability to keep the UNSTABLE status and figure out which stage actually set it... well, see above, that's way more work than one would reasonably expect (and much riskier too). Plus it requires reworking steps that can generate that status (generally steps that process unit test results, so JUnit and others).

          Sam Van Oort added a comment - brianjmurrell currentBuild.currentResult would be entirely unchanged. It has no representation within the Pipeline data model (it directly modifies the build). What this would do is ensure that no stages show up as "unstable" for you. Which gives you what you are asking for. If you want the ability to keep the UNSTABLE status and figure out which stage actually set it... well, see above, that's way more work than one would reasonably expect (and much riskier too). Plus it requires reworking steps that can generate that status (generally steps that process unit test results, so JUnit and others).

          svanoort So, no then, your proposal doesn't really help at all, unless there is an angle that I am missing.

          So to repeat the use-case:

          Say I have 3 parallel stages A, B, and C and A is SUCCESSful and B is UNSTABLE and C is SUCCESSful. Will I be able to determine in stage C that it is SUCCESSful even though stage B was UNSTABLE?

          And to further clarify further, the stages are parallel but A completes before B and B completes before C.

          I have to be able to determine in C that C was in fact successful. And I really only need to be able to do this because of this whole issue. Because the pass/fail stages cannot be determined any other way by the user, I have to post comments to the PR about which ones actually were UNSTABLE but I cannot even reliably do that if B's state is going to pollute C's state!

          Brian J Murrell added a comment - svanoort So, no then, your proposal doesn't really help at all, unless there is an angle that I am missing. So to repeat the use-case: Say I have 3 parallel stages A, B, and C and A is SUCCESSful and B is UNSTABLE and C is SUCCESSful. Will I be able to determine in stage C that it is SUCCESSful even though stage B was UNSTABLE? And to further clarify further, the stages are parallel but A completes before B and B completes before C. I have to be able to determine in C that C was in fact successful. And I really only need to be able to do this because of this whole issue. Because the pass/fail stages cannot be determined any other way by the user, I have to post comments to the PR about which ones actually were UNSTABLE but I cannot even reliably do that if B's state is going to pollute C's state!

          Sam Van Oort added a comment -

          brianjmurrell Unfortunately, that and the proposed mitigation in https://issues.jenkins-ci.org/browse/JENKINS-53889 are all we've got as viable short-term solutions. It's not a matter of "polluting state" - the data Pipeline needs to figure out which of the stages set the UNSTABLE status simply isn't there, and injecting that into the current system will probably break it completely.

          Several other engineers and I are working on a comprehensive solution to the things that make this problem hard to solve. We've already had several top Jenkins engineers spend literally months going in circles working on find a solid implementation that solves this specific issue without causing major regressions – the pull requests are out there if you're curious. Unless someone comes up with an unexpectedly clever solution, what's left is waiting for that larger effort to wrap up and then this specific problem will be solved along the way.

          And that sucks, and we know it's frustrating waiting, but we're doing the best we can to get that to the point of an alpha release.

          Sam Van Oort added a comment - brianjmurrell Unfortunately, that and the proposed mitigation in https://issues.jenkins-ci.org/browse/JENKINS-53889 are all we've got as viable short-term solutions. It's not a matter of "polluting state" - the data Pipeline needs to figure out which of the stages set the UNSTABLE status simply isn't there, and injecting that into the current system will probably break it completely. Several other engineers and I are working on a comprehensive solution to the things that make this problem hard to solve. We've already had several top Jenkins engineers spend literally months going in circles working on find a solid implementation that solves this specific issue without causing major regressions – the pull requests are out there if you're curious. Unless someone comes up with an unexpectedly clever solution, what's left is waiting for that larger effort to wrap up and then this specific problem will be solved along the way. And that sucks, and we know it's frustrating waiting, but we're doing the best we can to get that to the point of an alpha release.

          Georg Henzler added a comment -

          Following jglick's advice in his comment, I have digged a bit deeper into this issue and found a way to display what stages actually failed that works today without code change (still all stages show up as UNSTABLE, but the ones that have failed are marked as failed):

          stage('Build') { }     
          for(int i=1; i<=7; i++) {
              try{ 
                  stage('Test Suite '+i) { 
                      if(Math.floor(Math.random() * 3) % 3 == 0) {
                          throw new RuntimeException("tests failed, you would use a custom exception type here")
                      }
                  }
              } catch(/* your custom exception type here */ err) { currentBuild.result = 'UNSTABLE' }    
          }
          

          This creates a stage view as follows:

          This works because the pipeline-stage-view-plugin [1] takes the exception markers into account per stage. I'm sure blue ocean could also easily show this information.

          [1] https://github.com/jenkinsci/pipeline-stage-view-plugin/blob/47101e971da4d16a08a4a427c3d72f92cec8ff53/ui/src/main/js/view/templates/pipeline-staged.hbs#L100

          Georg Henzler added a comment - Following jglick 's advice in his comment , I have digged a bit deeper into this issue and found a way to display what stages actually failed that works today without code change (still all stages show up as UNSTABLE, but the ones that have failed are marked as failed): stage( 'Build' ) { } for ( int i=1; i<=7; i++) { try { stage( 'Test Suite ' +i) { if ( Math .floor( Math .random() * 3) % 3 == 0) { throw new RuntimeException( "tests failed, you would use a custom exception type here" ) } } } catch ( /* your custom exception type here */ err) { currentBuild.result = 'UNSTABLE' } } This creates a stage view as follows: This works because the pipeline-stage-view-plugin [1] takes the exception markers into account per stage. I'm sure blue ocean could also easily show this information. [1] https://github.com/jenkinsci/pipeline-stage-view-plugin/blob/47101e971da4d16a08a4a427c3d72f92cec8ff53/ui/src/main/js/view/templates/pipeline-staged.hbs#L100

          Stuart Rowe added a comment -

          ghenzler, I currently use FlowInterruptedException to achieve the same thing. The added benefit is this exception has a Result field.

          I've submitted pipeline-graph-analysis-plugin#PR22 to use the FlowInterruptedException's Result to determine the Stage status rather than using FAILURE (pipeline-graph-analysis-plugin 1.7) or ABORTED (pipeline-graph-analysis-plugin 1.9).

          Stuart Rowe added a comment - ghenzler , I currently use FlowInterruptedException  to achieve the same thing. The added benefit is this exception has a Result  field. I've submitted pipeline-graph-analysis-plugin#PR22 to use the FlowInterruptedException 's Result to determine the Stage status rather than using FAILURE (pipeline-graph-analysis-plugin 1.7) or ABORTED (pipeline-graph-analysis-plugin 1.9).

          jlpinardon added a comment - - edited

          Dear all,

          I want to add here a user experience.
          The team for which I setup the CI are really in a trouble when they see a totally yellow pipeline. They don't want to make the effort to check each stage one by one to find which is OK and which is not.
          I can understand that, and .. .well... users or customers are always right, isn't it ?
          I am afraid that the proposed stage view, with a simple red note will not help. Even if a green note is added for stage with status SUCCESS.

          In such a situation, have in mind that users don't want to adopt the pipeline. It is a real show-stopper. And they are currently asking me to come back to something which breaks the pipeline concept.

          So, I don't know what is the level of complexity, but to have green (or blue) color for stages ended with success, red for NOK ones and yellow for Unstable stages would be nothing else than a must have. That will really be user friendly and avoid headache for CI guys that must battle against upset users .

          jlpinardon added a comment - - edited Dear all, I want to add here a user experience. The team for which I setup the CI are really in a trouble when they see a totally yellow pipeline. They don't want to make the effort to check each stage one by one to find which is OK and which is not. I can understand that, and .. .well... users or customers are always right, isn't it ? I am afraid that the proposed stage view, with a simple red note will not help. Even if a green note is added for stage with status SUCCESS. In such a situation, have in mind that users don't want to adopt the pipeline. It is a real show-stopper. And they are currently asking me to come back to something which breaks the pipeline concept. So, I don't know what is the level of complexity, but to have green (or blue) color for stages ended with success, red for NOK ones and yellow for Unstable stages would be nothing else than a must have. That will really be user friendly and avoid headache for CI guys that must battle against upset users .

          This ticket now has 149 votes and 169 watchers, up from 116 and 143 only 4 months ago.  Needless to say that it's affecting a lot of people.

          It's also very disheartening to see such a critical issue has not made it beyond the same Planned stage on the Blue Ocean Roadmap that it was back at the beginning of September.  It's not even In Progress yet, 4+ months later.

          Does the Jenkins team really think this issue is that unimportant?

          I understand that it's complicated, but is that reason enough to not even get started on it?

          Brian J Murrell added a comment - This ticket now has 149 votes and 169 watchers, up from 116 and 143 only 4 months ago.  Needless to say that it's affecting a lot of people. It's also very disheartening to see such a critical issue has not made it beyond the same Planned stage on the Blue Ocean Roadmap  that it was back at the beginning of September.  It's not even In Progress yet, 4+ months later. Does the Jenkins team really think this issue is that unimportant? I understand that it's complicated, but is that reason enough to not even get started on it?

          boris ivan added a comment -

          Was so excited to move our production builds to using the pipeline and blue ocean visualization. With all the stages changing from green to unstable, it's a blocker. Even posting each stage status to slack becomes misleading, in that healthy stages post unstable, and there are instances of unstable stages posting as success, that's a deal breaker right off the bat.

          Can I suggest a short term workaround?

          If there was an option in the pipeline: "stageStatus=independent" (or something like that), which worked this way, it would allow us to have control and use blue ocean until it's resolved:

          • simply allow each stage icon to use the color/state of it's own independent activity

          That's it. I'm not sure if that's a flowNode or what, but whatever each icon is in any stage, they all seem to be capable (today) of executing and knowing if that execution was successful or unstable or failure or aborted. If that can simply be retained and displayed for that icon, this works for me. As far as what the overall build status ends up as in this case – if you allow us to set it at the end of the pipeline, we could code that to our own needs, based on the individual statuses per stage.

           

          boris ivan added a comment - Was so excited to move our production builds to using the pipeline and blue ocean visualization. With all the stages changing from green to unstable, it's a blocker. Even posting each stage status to slack becomes misleading, in that healthy stages post unstable, and there are instances of unstable stages posting as success, that's a deal breaker right off the bat. Can I suggest a short term workaround? If there was an option in the pipeline: "stageStatus=independent" (or something like that), which worked this way, it would allow us to have control and use blue ocean until it's resolved: simply allow each stage icon to use the color/state of it's own independent activity That's it. I'm not sure if that's a flowNode or what, but whatever each icon is in any stage, they all seem to be capable (today) of executing and knowing if that execution was successful or unstable or failure or aborted. If that can simply be retained and displayed for that icon, this works for me. As far as what the overall build status ends up as in this case – if you allow us to set it at the end of the pipeline, we could code that to our own needs, based on the individual statuses per stage.  

          Brian J Murrell added a comment - - edited

          This ticket has added 26 more votes and 20 more watchers in just a hair over a month for a total of 175 and 189, respectively.

          Could somebody please provide an update on the status of this issue to those 189 watchers please?

          Is this really (after all of this time and attention) still just in the Planned state that the  Blue Ocean Roadmap indicates?

          Brian J Murrell added a comment - - edited This ticket has added 26 more votes and 20 more watchers in just a hair over a month for a total of 175 and 189, respectively. Could somebody please provide an update on the status of this issue to those 189 watchers please? Is this really  (after all of this time and attention) still just in the  Planned state that the   Blue Ocean Roadmap indicates?

          ‪yair yavneel‬‏ added a comment - - edited

          this is necessary to enhance the pipeline status view, to be able to mark each stage status separately, to identify which of them if passed, unstable or failed. 
          It is should not be Jenkins's decision to abort the pipeline or not. 
          what if there are independent stages?
          what if I want to continue and just be aware about test/stage that was failed?
          I think the status and the abort action should be performed explicitly. 
          for example - stage function that received 3 arguments:
          1 - stage status (mandatory)
          2 - Boolean if need to abort (optional - default to continue)
          3 - message to be print in the stage view window (optional)

          it is will be very helpful.   

          ‪yair yavneel‬‏ added a comment - - edited this is necessary to enhance the pipeline status view, to be able to mark each stage status separately, to identify which of them if passed, unstable or failed.  It is should not be Jenkins's decision to abort the pipeline or not.  what if there are independent stages? what if I want to continue and just be aware about test/stage that was failed? I think the status and the abort action should be performed explicitly.  for example - stage function that received 3 arguments: 1 - stage status (mandatory) 2 - Boolean if need to abort (optional - default to continue) 3 - message to be print in the stage view window (optional) it is will be very helpful.   

          Alexandr Shchepkin added a comment - - edited

          Does the Jenkins team really think this issue is not important?

          Alexandr Shchepkin added a comment - - edited Does the Jenkins team really think this issue is not important?

          boris ivan added a comment - - edited

          I understand that the logic associated with the overall job status is established throughout Jenkins and that's difficult to adjust. (at least that's what I've heard the problem is, re: 'fixing' this). But in the meantime, can't we just leave that logic alone, and create a new property for the stage object, called 'stage status', that is limited in scope to the execution of that specific stage, and give the user an option to use legacy behavior, or display per-stage-status? The 'overall' job status logic could remain the same.

          To ignore the entire user base screaming about this about to drive me away, and is so different from the great experience and responsiveness I've had with the Jenkins team over the years.

          boris ivan added a comment - - edited I understand that the logic associated with the overall job status is established throughout Jenkins and that's difficult to adjust. (at least that's what I've heard the problem is, re: 'fixing' this). But in the meantime, can't we just leave that logic alone, and create a new property for the stage object, called 'stage status', that is limited in scope to the execution of that specific stage, and give the user an option to use legacy behavior, or display per-stage-status? The 'overall' job status logic could remain the same. To ignore the entire user base screaming about this about to drive me away, and is so different from the great experience and responsiveness I've had with the Jenkins team over the years.

          boris ivan added a comment -

          added an attachment 'allYellow.jpg' to illustrate the impact. The various grouping of 12 stages on the right of the pipeline are all test related. Before the pipeline got that far – everything was green, as all those previous stages were successful. And for the 12 test related stages on the right... 2 of them had some failures, the other 10 had no failures. But examining a visualization to get a good understand of what went well and what had issues.. all value is lost, nothing has meaning.

          boris ivan added a comment - added an attachment 'allYellow.jpg' to illustrate the impact. The various grouping of 12 stages on the right of the pipeline are all test related. Before the pipeline got that far – everything was green, as all those previous stages were successful. And for the 12 test related stages on the right... 2 of them had some failures, the other 10 had no failures. But examining a visualization to get a good understand of what went well and what had issues.. all value is lost, nothing has meaning.

          Brian J Murrell added a comment - - edited

          Completely agree with borisivan.  To the extent even that I have written Blue Ocean off.  It causes more questions and confusion that it adds value.

          Instead, I tell my users that GitHub's commit statuses are the point of truth and to that end provide a status for every pipeline stage complete with a URL to (non-Blue Ocean) stage results in the Details link of each GitHub status posted.

          Brian J Murrell added a comment - - edited Completely agree with  borisivan .  To the extent even that I have written Blue Ocean off.  It causes more questions and confusion that it adds value. Instead, I tell my users that GitHub's commit statuses are the point of truth and to that end provide a status for every pipeline stage complete with a URL to (non-Blue Ocean) stage results in the Details link of each GitHub status posted.

          vaishal shah added a comment -

          Hi Team,

          Did anyone found any solution for Declarative Pipeline?

          Please share details if any solution will be there.

           

          Thanks,

          Vaishal

          vaishal shah added a comment - Hi Team, Did anyone found any solution for Declarative Pipeline? Please share details if any solution will be there.   Thanks, Vaishal

          Michael Neale added a comment -

          this one requires a rewrite of a chunk of pipeline engine - abayer is it fair to say that this is part of new generation ?

           

          I don't see this changing for declarative (given all things use "currentBuild." which can't work with this by design) - so I am not sure if this will be done in this form. 

          Michael Neale added a comment - this one requires a rewrite of a chunk of pipeline engine - abayer is it fair to say that this is part of new generation ?   I don't see this changing for declarative (given all things use "currentBuild." which can't work with this by design) - so I am not sure if this will be done in this form. 

          hardi249 added a comment -

          From an outside perspective, this is ridiculous. It cannot be that hard to determine and save the result of a single node and color it accordingly.

          hardi249 added a comment - From an outside perspective, this is ridiculous. It cannot be that hard to determine and save the result of a single node and color it accordingly.

          Michael Neale added a comment -

          Yes it would seem odd from outside - but from inside it kind of makes sense. Plugins mark the whole run as unstable, not a stage. This is a pipeline thing misclassified as blue ocean (other visualisations do the same). 

          If a build failed then stopped - could just show the "last" stage as unstable only - but if it continues on (which some do by default) then that may be misleading as it may have been an earlier stage that showed it up. 

          Michael Neale added a comment - Yes it would seem odd from outside - but from inside it kind of makes sense. Plugins mark the whole run as unstable, not a stage. This is a pipeline thing misclassified as blue ocean (other visualisations do the same).  If a build failed then stopped - could just show the "last" stage as unstable only - but if it continues on (which some do by default) then that may be misleading as it may have been an earlier stage that showed it up. 

          I have a somewhat related question:

          Is there an API to access the results of the various stages in a pipeline? I have been playing tricks like setting a variable with the name of the current stage when a stage starts and checking its contents in the post for the pipeline to report which stage failed and would like to have a cleaner solution to work with what is already in place...

           

          Nancy Robertson added a comment - I have a somewhat related question: Is there an API to access the results of the various stages in a pipeline? I have been playing tricks like setting a variable with the name of the current stage when a stage starts and checking its contents in the post for the pipeline to report which stage failed and would like to have a cleaner solution to work with what is already in place...  

          Michael Neale added a comment -

          nancyrobertson2005 I think you hit the nail on the head - I am not sure such a facillity exists. If it did - things like this would be possible. 

          Michael Neale added a comment - nancyrobertson2005 I think you hit the nail on the head - I am not sure such a facillity exists. If it did - things like this would be possible. 

          boris ivan added a comment -

          I can understand the existing logic and methodology makes this difficult to change.
          But I don't understand why something simple like this couldn't be done as a workaround:

          • add a new property (doesn't exist today) to each stage object: stageIndependentStatus
          • when each stage completes, immediately save the value of the independent execution of that stage in stageIndependentStatus

          This way, you can let all the existing logic stay as-is regarding stage state, state propagation to other stages, etc – all the things that are difficult to change. The "overall build" would still be marked as it is today. Nothing needs to change.

          But with that separate property being available per stage, visualizers like blue ocean could implement a configuration: "showStageLegacyStatus" or "showStageIndependentStatus", defined in the options for a pipeline. If the user chooses the latter, visualizers like blue ocean could make their own choice to paint strictly by the stageIndependentStatus.

          boris ivan added a comment - I can understand the existing logic and methodology makes this difficult to change. But I don't understand why something simple like this couldn't be done as a workaround: add a new property (doesn't exist today) to each stage object: stageIndependentStatus when each stage completes, immediately save the value of the independent execution of that stage in stageIndependentStatus This way, you can let all the existing logic stay as-is regarding stage state, state propagation to other stages, etc – all the things that are difficult to change. The "overall build" would still be marked as it is today. Nothing needs to change. But with that separate property being available per stage, visualizers like blue ocean could implement a configuration: "showStageLegacyStatus" or "showStageIndependentStatus", defined in the options for a pipeline. If the user chooses the latter, visualizers like blue ocean could make their own choice to paint strictly by the stageIndependentStatus.

          I'v created a shared library, which contains an API to get the failed stages:

          https://github.com/roel0/jenkins-util-scripts

          You can then easily print out that data, or email it , or post it on slack or ...

           

          Example usage:

          try {
              super_stage("stage1") {
              }
              super_stage("stage2") {
                  error "oei"
              }
          } catch(e) {
          }
          
          println "Failed stages: ${get_failed_stages().join(',')}"
          

           

          roel postelmans added a comment - I'v created a shared library, which contains an API to get the failed stages: https://github.com/roel0/jenkins-util-scripts You can then easily print out that data, or email it , or post it on slack or ...   Example usage: try { super_stage( "stage1" ) { } super_stage( "stage2" ) { error "oei" } } catch (e) { } println "Failed stages: ${get_failed_stages().join( ',' )}"  

          Devin Nusbaum added a comment - - edited

          Some changes have recently been released to address this issue for Blue Ocean. All of the changes require Jenkins 2.138.4 or newer. The main fixes are in Pipeline: API Plugin version 2.34 and Pipeline: Graph Analysis Plugin version 1.10, but in most cases you will need to change your Pipelines to be able to take advantage of the fixes:

          • If you are currently setting the build result to unstable using currentBuild.result = 'UNSTABLE', you need to update to Pipeline: Basic Steps Plugin version 2.16 and start using the unstable step instead. See the changelog here for details. You may also be interested in the new warnError step.
            • In case you are curious why we can't make currentBuild.result = 'UNSTABLE' work as-is, the problem is that currentBuild is implemented as a GlobalVariable rather than a regular Pipeline Step, so there is no way to associate the setting of the build result with the current Pipeline stage, especially if a parallel step is being executed.
          • If the build result is being set to unstable by the junit step from the JUnit Plugin, then update the JUnit Plugin to version 1.28 and the visualization should be fixed.
          • If the build result is being set to unstable by the publishIssues or recordIssues steps in the Warnings NG Plugin, then keep an eye on this in-progress PR which will update those steps to use a new API that will fix the visualization.
          • If the build result is being set to unstable by a Pipeline step in some other plugin, then file a new issue with the component set to that plugin, and ask the maintainer to start using the new WarningAction API from Pipeline: Basic Steps Plugin version 2.16 wherever they change the build result.

          In case of problems, the new behavior can be disabled by setting the system property org.jenkinsci.plugins.workflow.pipelinegraphanalysis.StatusAndTiming.DISABLE_WARNING_ACTION_LOOKUP to "true", or by setting the static variable of the same name to true via the Jenkins script console.

          Note that the Blue Ocean steps view will not yet display the status of individual steps using the new API correctly, keep an eye on this in-progress PR which will fix that issue. Once that PR is merged and released, I plan to mark this issue as closed.

          The visualization provided by Pipeline: Stage View Plugin has not been integrated with the new API added in Pipeline: API 2.34, and so that visualization is not currently affected by these changes.

          Devin Nusbaum added a comment - - edited Some changes have recently been released to address this issue for Blue Ocean. All of the changes require Jenkins 2.138.4 or newer. The main fixes are in Pipeline: API Plugin version 2.34 and Pipeline: Graph Analysis Plugin version 1.10, but in most cases you will need to change your Pipelines to be able to take advantage of the fixes: If you are currently setting the build result to unstable using currentBuild.result = 'UNSTABLE' , you need to update to Pipeline: Basic Steps Plugin version 2.16 and start using the unstable step instead. See the changelog here for details. You may also be interested in the new warnError step. In case you are curious why we can't make currentBuild.result = 'UNSTABLE' work as-is, the problem is that currentBuild is implemented as a GlobalVariable rather than a regular Pipeline Step, so there is no way to associate the setting of the build result with the current Pipeline stage, especially if a parallel step is being executed. If the build result is being set to unstable by the junit step from the JUnit Plugin, then update the JUnit Plugin to version 1.28 and the visualization should be fixed. If the build result is being set to unstable by the publishIssues or recordIssues steps in the Warnings NG Plugin, then keep an eye on this in-progress PR which will update those steps to use a new API that will fix the visualization. If the build result is being set to unstable by a Pipeline step in some other plugin, then file a new issue with the component set to that plugin, and ask the maintainer to start using the new WarningAction API from Pipeline: Basic Steps Plugin version 2.16 wherever they change the build result. In case of problems, the new behavior can be disabled by setting the system property  org.jenkinsci.plugins.workflow.pipelinegraphanalysis.StatusAndTiming.DISABLE_WARNING_ACTION_LOOKUP  to "true", or by setting the static variable of the same name to true  via the Jenkins script console. Note that the Blue Ocean steps view will not yet display the status of individual steps using the new API correctly, keep an eye on this in-progress PR which will fix that issue. Once that PR is merged and released, I plan to mark this issue as closed. The visualization provided by Pipeline: Stage View Plugin has not been integrated with the new API added in Pipeline: API 2.34, and so that visualization is not currently affected by these changes.

          Hugo L added a comment - - edited

          Thank you for making progress on this issue.

          I confirm it works, using it as follow:

           

          steps { 
              script {
                   def r = sh(returnStatus: true, script: "my_script");
                   if (r != 0) {
                       unstable(message: "Failed: ${STAGE_NAME} => return UNSTABLE status")
                   }
              }
          } 
          
          

           edit: ok it runs smoothly

          Hugo L added a comment - - edited Thank you for making progress on this issue. I confirm it works, using it as follow:   steps {      script {         def r = sh(returnStatus: true , script: "my_script" );          if (r != 0) {              unstable(message: "Failed: ${STAGE_NAME} => return UNSTABLE status" )          }     } }   edit: ok it runs smoothly

          I tried the new functionality dnusbaum mentioned above, but using anything else than the default values in catchError e.g. catchError(buildResult: 'SUCCESS') results in:

           
          org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
          WorkflowScript: 11: Expecting "class hudson.model.Result" for parameter "buildResult" but got "SUCCESS" of type class java.lang.String instead @ line 11, column 41.
          catchError(buildResult: 'SUCCESS')

          Jenkins version: 2.164.3

          Pipeline Basic Steps: 2.16

          (blue ocean is not installed)

          Torsten Walter added a comment - I tried the new functionality dnusbaum mentioned above, but using anything else than the default values in catchError e.g. catchError(buildResult: 'SUCCESS') results in:   org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: WorkflowScript: 11: Expecting "class hudson.model.Result" for parameter "buildResult" but got "SUCCESS" of type class java.lang.String instead @ line 11, column 41. catchError(buildResult: 'SUCCESS') Jenkins version:  2.164.3 Pipeline Basic Steps: 2.16 (blue ocean is not installed)

          vaishal shah added a comment - - edited

          It's Working!!!!

           Thanks Devin

           

          vaishal shah added a comment - - edited It's Working!!!!  Thanks Devin  

          Yves Schumann added a comment -

          Can one please post a screenshot how this looks now on traditional view and on BlueOcean?

          Yves Schumann added a comment - Can one please post a screenshot how this looks now on traditional view and on BlueOcean?

          Reinhold Füreder added a comment - - edited

          Hm, while it seems to work fine for junit steps like in the screenshot it does not visualize the unstable stage when using code like hugo_l (in a replayed scripted pipeline):
          How comes!?

          Reinhold Füreder added a comment - - edited Hm, while it seems to work fine for junit steps like in the screenshot it does not visualize the unstable stage when using code like hugo_l (in a replayed scripted pipeline): How comes!?

          twalter: using the fully qualified names like hudson.model.Result.FAILURE works for me.

          Daan van den Akker added a comment - twalter : using the fully qualified names like hudson.model.Result.FAILURE works for me.

          Devin Nusbaum added a comment - - edited

          twalter I just tested your code and it works ok for me in some basic tests. Please open a new issue for your problem, assign it to me, add any additional information or context that might be useful and include a list of all plugins you have installed and their versions. Thanks!

          reinholdfuereder Can you please open a new issue, including a minimal Pipeline that reproduces the issue and assign the issue to me? Thanks!

          Also I was not explicit in my original post that these fixes do not currently affect Stage View at all, they only affect Blue Ocean, so I will update my post to make that clear.

          Devin Nusbaum added a comment - - edited twalter I just tested your code and it works ok for me in some basic tests. Please open a new issue for your problem, assign it to me, add any additional information or context that might be useful and include a list of all plugins you have installed and their versions. Thanks! reinholdfuereder Can you please open a new issue, including a minimal Pipeline that reproduces the issue and assign the issue to me? Thanks! Also I was not explicit in my original post that these fixes do not currently affect Stage View at all, they only affect Blue Ocean, so I will update my post to make that clear.

          Reinhold Füreder added a comment - - edited

          Reinhold Füreder added a comment - - edited dnusbaum Please see JENKINS-57579

          The visualization provided by Pipeline: Stage View Plugin has not been integrated with the new API added in Pipeline: API 2.34, and so that visualization is not currently affected by these changes.

          I looked but could not find one; is the intention to have a separate ticket for the Stage View plugin integration?

          Chris Burroughs added a comment - The visualization provided by Pipeline: Stage View Plugin has not been integrated with the new API added in Pipeline: API 2.34, and so that visualization is not currently affected by these changes. I looked but could not find one; is the intention to have a separate ticket for the Stage View plugin integration?

          Devin Nusbaum added a comment -

          I just published a blog post with an overview of this issue and how to take advantage of the recent changes in your Pipeline to improve the way it is visualized in Blue Ocean: https://jenkins.io/blog/2019/07/05/jenkins-pipeline-stage-result-visualization-improvements/

          cburroughs Yes, feel free to open a new issue for Stage View integration and link it to this one.

          Devin Nusbaum added a comment - I just published a blog post with an overview of this issue and how to take advantage of the recent changes in your Pipeline to improve the way it is visualized in Blue Ocean:  https://jenkins.io/blog/2019/07/05/jenkins-pipeline-stage-result-visualization-improvements/ .  cburroughs Yes, feel free to open a new issue for Stage View integration and link it to this one.

          Devin Nusbaum added a comment -

          Given the core changes have been released, I am closing this issue. As noted in my blog post, if you are still having problems with a Pipeline step from specific please file a new issue with the component set to that plugin (after checking for duplicates), clearly describing which step has the problem and under what circumstances it occurs, and link to the developers section of the post as a reference for how the maintainer might be able to address the problem.
           

          Devin Nusbaum added a comment - Given the core changes have been released, I am closing this issue. As noted in my blog post, if you are still having problems with a Pipeline step from specific please  file a new issue  with the component set to that plugin (after checking for duplicates), clearly describing which step has the problem and under what circumstances it occurs, and link to the developers section of the post  as a reference for how the maintainer might be able to address the problem.  

          Just created the issue for Pipeline: Stage View Plugin -> https://issues.jenkins-ci.org/browse/JENKINS-58783

          Jonathan Araña Cruz added a comment - Just created the issue for Pipeline: Stage View Plugin -> https://issues.jenkins-ci.org/browse/JENKINS-58783

          Vincent Yan added a comment - - edited

          Thanks for the hard work. I've just recently started to use Jenkins but on a older version. Then came along this issue while I was googling about the stage result issue. Glad there's already a solution in the recent version!  Appreciate it!

          Vincent Yan added a comment - - edited Thanks for the hard work. I've just recently started to use Jenkins but on a older version. Then came along this issue while I was googling about the stage result issue. Glad there's already a solution in the recent version!   Appreciate it!

          Was this more than just a visualisation problem or was this actually making stage B run it's post->unstable steps if stage A was UNSTABLE and set the overall build status to UNSTABLE?  Because that would be very bad/wrong.

          Brian J Murrell added a comment - Was this more than just a visualisation problem or was this actually making stage B run it's post -> unstable steps if stage A was UNSTABLE and set the overall build status to UNSTABLE ?  Because that would be very bad/wrong.

          Pete W added a comment -

          Devin's blog post explains this issue and the fix well brianjmurrell

          The fix doesn't make B Stage run, or not run - it still leaves it down to the implementor to choose if they want B to run or not

          Pete W added a comment - Devin's blog post explains this issue and the fix well brianjmurrell The fix doesn't make B Stage run, or not run - it still leaves it down to the implementor to choose if they want B to run or not

          I have read Devin's blog post.  It does not answer my question AFAIU.  But you might think it could because you misunderstood my question.

          My question was not about whole stages running or not running.  Both stages A and B are running.  B was SUCCESSFUL and A was UNSTABLE.  What I am seeing however is that B's post->unstable is being run even though B was successful.

          I am trying to determine if that mis-behaviour is a component of this bug or if this is something different/new.

          Brian J Murrell added a comment - I have read Devin's blog post.  It does not answer my question AFAIU.  But you might think it could because you misunderstood my question. My question was not about whole stages running or not running.  Both stages A  and B  are running.  B  was SUCCESSFUL  and A  was UNSTABLE .  What I am seeing however is that B 's post -> unstable is being run even though B was successful. I am trying to determine if that mis-behaviour is a component of this bug or if this is something different/new.

          Devin Nusbaum added a comment -

          brianjmurrell Declarative post conditions look at the overall build result, so if a step like unstable or junit marks the build as unstable in stage A and the Pipeline continues executing, unstable post conditions in later stages such as B will execute (unless the build result becomes worse than unstable, e.g. failure). As far as I know Declarative has always worked this way. Before the fix for this issue, both stage A and stage B would have been shown as being unstable in Blue Ocean, but after the fix for this issue, only stage A is shown as unstable (the overall build result was shown as unstable both before and after the fix).

          Devin Nusbaum added a comment - brianjmurrell Declarative post conditions look at the overall build result, so if a step like unstable or junit marks the build as unstable in stage A and the Pipeline continues executing, unstable post conditions in later stages such as B will execute (unless the build result becomes worse than unstable, e.g. failure). As far as I know Declarative has always worked this way. Before the fix for this issue, both stage A and stage B would have been shown as being unstable in Blue Ocean, but after the fix for this issue, only stage A is shown as unstable (the overall build result was shown as unstable both before and after the fix).

          dnusbaum Whoa!  Are you serious (and sure of what you are saying)?  It seems very un-intuitive, at best, that the status of any individual given stage should be able to alter the status of any other stage.  I would absolutely agree that the outer pipeline {} block's post should be executed per the overall build result, but most certainly not any individual stage.

          I suppose there must be some, can't imagine a use-case where that is useful.  On the contrary, if one has a parallel block of test stages, why would I want the unstable post block of test stage A to run when test stage A passed, but test stage B failed?

          Ultimately, I suppose this should move to a new ticket.  I will open one.  But I probably need to mock up a minimal reproducer before I do.

          Brian J Murrell added a comment - dnusbaum Whoa!  Are you serious (and sure of what you are saying)?  It seems very un-intuitive,  at best , that the status of any individual given stage should be able to alter the status of any other stage.  I would absolutely agree that the outer pipeline { } block's post should be executed per the overall build result, but most certainly not any individual stage. I suppose there must be some, can't imagine a use-case where that is useful.  On the contrary, if one has a parallel block of test stages, why would I want the unstable post block of test stage A to run when test stage A passed, but test stage B failed? Ultimately, I suppose this should move to a new ticket.  I will open one.  But I probably need to mock up a minimal reproducer before I do.

          Brian J Murrell added a comment - - edited

          dnusbaum Can you please respond to the previous comment?  Is it really the expected behaviour that all stages in a parallel block execute their post > failure blocks when even just 1 other stage in the parallel group fails?  The result being that stages that actually were successful don't actually process their own post > successful block but instead process their own post -> failure block?

          Quite frankly, this behaviour is hideous.  In my case, a stage is not actually even failing.  In a block of 3 parallel stages if just one of them invokes a catchError(stageResult: 'UNSTABLE', buildResult: 'SUCCESS'), the remaining stages take their post > failure exit path, even though they are SUCCESSFUL, and even though the stage that invoked the catchError() takes it's own post > successful path!

          I cannot imagine how this is not a bug.  This makes catchError(stageResult: 'UNSTABLE', buildResult: 'SUCCESS') quite useless in fact if it can bomb the exit of other stages.

          Brian J Murrell added a comment - - edited dnusbaum  Can you please respond to the previous comment?  Is it really the expected behaviour that all stages in a parallel block execute their post > failure blocks when even just 1 other stage in the parallel group fails?  The result being that stages that actually were successful don't actually process their own post > successful block but instead process their own post -> failure block? Quite frankly, this behaviour is hideous.  In my case, a stage is not actually even failing.  In a block of 3 parallel stages if just one of them invokes a catchError(stageResult: 'UNSTABLE', buildResult: 'SUCCESS') , the remaining stages take their post > failure exit path, even though they are SUCCESSFUL, and even though the stage that invoked the catchError() takes it's own post > successful path! I cannot imagine how this is not a bug.  This makes catchError(stageResult: 'UNSTABLE', buildResult: 'SUCCESS') quite useless in fact if it can bomb the exit of other stages.

          Kalle Niemitalo added a comment - - edited

          JENKINS-57801 has been filed about declarative Pipeline stage post conditions being affected by the result of the whole pipeline.

          Kalle Niemitalo added a comment - - edited JENKINS-57801 has been filed about declarative Pipeline stage post conditions being affected by the result of the whole pipeline.

            dnusbaum Devin Nusbaum
            michaelneale Michael Neale
            Votes:
            233 Vote for this issue
            Watchers:
            237 Start watching this issue

              Created:
              Updated:
              Resolved: