Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-57801

Execution of post stage block is based on pipeline status, not stage result which docs says it should

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Blocker Blocker
    • pipeline
    • None

      The declarative pipeline documentation on "post" says the following: "These condition blocks allow the execution of steps inside each condition depending on the completion status of the Pipeline or stage" 

      Currently, the post execution of a stage seems to refer to the whole pipeline status at that point and not the completion status of the stage as documented.

      Given the following pipeline:

      pipeline {
        agent any
        stages {
          stage('First') {
            steps { echo 'Hello' }
            post { success { echo 'First stage is success' } }
          }
          stage('Second') {
            steps { sh """ echo '<testsuite errors="1" failures="0" name="test" skipped="0" tests="1" time="1"><testcase classname="" file="file.txt" name="name" time="0.000"><error message="error"></error></testcase></testsuite>' > test-results.xml """ }
            post {
              always { junit 'test-results.xml' }
              unstable { echo 'Second stage is unstable' }
            }
          }
          stage('Third') {
            steps { echo 'Hello' }
            post {
              success { echo 'Third stage is success' }
              unstable { echo 'Third stage is unstable! really???' }
            }
          }
        }
        post { unstable { echo 'Job is unstable' } }
      }
      
      

      I get the following output:

      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (First)
      [Pipeline] echo
      Hello
      Post stage
      [Pipeline] echo
      First stage is success
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] stage
      [Pipeline] { (Second)
      [Pipeline] sh
      + echo <testsuite errors="1" failures="0" name="test" skipped="0" tests="1" time="1"><testcase classname="" file="file.txt" name="name" time="0.000"><error message="error"></error></testcase></testsuite>
      Post stage
      [Pipeline] junit
      Recording test results
      [Pipeline] echo
      Second stage is unstable
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] stage
      [Pipeline] { (Third)
      [Pipeline] echo
      Hello
      Post stage
      [Pipeline] echo
      Third stage is unstable! really???
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] stage
      [Pipeline] { (Declarative: Post Actions)
      [Pipeline] echo
      Job is unstable
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      

      Clearly the third stage post unstable block is executed even though the stage itself is a success.

      This is also contradicted by the blue ocean stage view:

          [JENKINS-57801] Execution of post stage block is based on pipeline status, not stage result which docs says it should

          Marcus Philip added a comment -

          In JENKINS-37792 the stage post block functionality was introduced. Judging from the lack of tests of this in https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/30 I'm guessing it never worked.

          Marcus Philip added a comment - In JENKINS-37792 the stage post block functionality was introduced. Judging from the lack of tests of this in https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/30  I'm guessing it never worked.

          I have adjusted the priority of this ticket to Blocker as that is exactly what this issue is. This issue completely voids pipelines from running in any sensible manner. It forces CI devs to create pipelines that have false success as trying to indicate a proper failure in a single stage tanks the whole pipeline.

          This needs the highest attention!

          Brian J Murrell added a comment - I have adjusted the priority of this ticket to Blocker as that is exactly what this issue is. This issue completely voids pipelines from running in any sensible manner. It forces CI devs to create pipelines that have false success as trying to indicate a proper failure in a single stage tanks the whole pipeline. This needs the highest attention!

          Allan BURDAJEWICZ added a comment - - edited

          That BuildCondition is what seem to be used whether you are in a stage or in the root pipeline.
          In that combined result the build result might always take precedence if it is worse.

          Allan BURDAJEWICZ added a comment - - edited That BuildCondition is what seem to be used whether you are in a stage or in the root pipeline. In that combined result the build result might always take precedence if it is worse.

          Per Böhlin added a comment -

          I have not read the code, so I could very much be wrong in my assumptions, but based on the result presented below and more extensive tests that caused me to find the bug and find myself here in the first place, I wanted to present my observations - specifically for parallel stages.

          With this pipeline:

          pipeline {
              agent none
              options {
                      skipDefaultCheckout(true)
              }
              stages {
                  stage("Parallel stages") {
                      failFast false
                      parallel {
                          stage("1") {
                              agent {
                                  label 'linux'
                              }
                              steps {
                                  sh """ echo '<testsuite errors="1" failures="0" name="test" skipped="0" tests="1" time="1"><testcase classname="" file="file.txt" name="name" time="0.000"><error message="error"></error></testcase></testsuite>' > test-results.xml """
                              }
                              post {
                                  always {
                                      echo "1: always"
                                      junit 'test-results.xml'
                                  }
                                  unstable {
                                      echo "1: unstable (from junit test result)"
                                  }
                                  success {
                                      echo "1: success"
                                  }
                              }
                          }
                          stage("2") {
                              agent {
                                  label 'linux'
                              }
                              steps {
                                  echo "Sleeping a bit to ensure stage 1 has finished even though running in parallel"
                                  sh 'sleep 3; exit 0'
                              }
                              post {
                                  always {
                                      echo "2: always"
                                  }
                                  unstable {
                                      echo "2: unstable"
                                  }
                                  success {
                                      echo "2: success"
                                  }
                                  failure {
                                      echo "2: failure"
                                  }
                              }
                          }              
                      }
                  }
              }
          }
          

          The output is (re-order interlacing for clarity):

          [1] + echo '<testsuite errors="1" failures="0" name="test" skipped="0" tests="1" time="1"><testcase classname="" file="file.txt" name="name" time="0.000"><error message="error"></error></testcase></testsuite>'
          [1] 1: always
          [1] Recording test results
          [1] 1: unstable (from junit test result)
          [2] Sleeping a bit to ensure stage 1 has finished even though running in parallel
          [2] + sleep 3
          [2] + exit 0
          [2] 2: always
          [2] 2: unstable
          Finished: UNSTABLE
          

          If I cause stage 1 to fail with exit 1 after generating the test report I get:

          [1] + echo '<testsuite errors="1" failures="0" name="test" skipped="0" tests="1" time="1"><testcase classname="" file="file.txt" name="name" time="0.000"><error message="error"></error></testcase></testsuite>'
          [1] + exit 1
          [1] 1: always
          [1] Recording test results
          Failed in branch 1
          [2] Sleeping a bit to ensure stage 1 has finished even though running in parallel
          [2] + sleep 3
          [2] + exit 0
          [2] 2: always
          [2] 2: success
          ERROR: script returned exit code 1
          Finished: FAILURE
          

          This makes me make the following "conclusions":

          The parallel stage collects the result from each individual stage and reports the aggregate (worst). Each stage is independent (being parallel) as they should be, with regard to the outcome of the steps. However, the post-actions does not modify the result of the stage it is associated with only, but also the "global" result. At least in the specific case of the junit test collection that change the "global" result to unstable in addition to making stage 1 unstable. After stage 2 steps runs successfully and enters post, one would assume that stage 2's status would be success. However, since unstable is triggered, that makes me believe that post actions not only modify the "global" result but also are triggered by the combination of the "global" result and the stage's steps' result.

          Then we have the additional case of the post-action not needing to change the stage's status. In my second example with failure in stage 1's steps, the junit post-action doesn't need to change the stage's result and by that also, does not affect the "global" result. Because, when we enter stage 2's post-actions, it runs as for success.

          In summary:

          Post-actions, in addition to modifying the status of its associated stage, it also affects a "global" state. The post-action trigger (always/success/unstable/...) is affected by the aggregate of its associated stage's steps' result as well as the "global" result from other post-actions. An alternative interpretation would be for the post-block to be the same instance for all stages, and hence gets a semi-global transient state across the stages.

          Per Böhlin added a comment - I have not read the code, so I could very much be wrong in my assumptions, but based on the result presented below and more extensive tests that caused me to find the bug and find myself here in the first place, I wanted to present my observations - specifically for parallel stages. With this pipeline: pipeline { agent none options { skipDefaultCheckout( true ) } stages { stage( "Parallel stages" ) { failFast false parallel { stage( "1" ) { agent { label 'linux' } steps { sh """ echo '<testsuite errors=" 1 " failures=" 0 " name=" test " skipped=" 0 " tests=" 1 " time=" 1 "><testcase classname=" " file=" file.txt " name=" name " time=" 0.000 "><error message=" error "></error></testcase></testsuite>' > test-results.xml " "" } post { always { echo "1: always" junit 'test-results.xml' } unstable { echo "1: unstable (from junit test result)" } success { echo "1: success" } } } stage( "2" ) { agent { label 'linux' } steps { echo "Sleeping a bit to ensure stage 1 has finished even though running in parallel" sh 'sleep 3; exit 0' } post { always { echo "2: always" } unstable { echo "2: unstable" } success { echo "2: success" } failure { echo "2: failure" } } } } } } } The output is (re-order interlacing for clarity): [1] + echo '<testsuite errors="1" failures="0" name="test" skipped="0" tests="1" time="1"><testcase classname="" file="file.txt" name="name" time="0.000"><error message="error"></error></testcase></testsuite>' [1] 1: always [1] Recording test results [1] 1: unstable (from junit test result) [2] Sleeping a bit to ensure stage 1 has finished even though running in parallel [2] + sleep 3 [2] + exit 0 [2] 2: always [2] 2: unstable Finished: UNSTABLE If I cause stage 1 to fail with exit 1 after generating the test report I get: [1] + echo '<testsuite errors="1" failures="0" name="test" skipped="0" tests="1" time="1"><testcase classname="" file="file.txt" name="name" time="0.000"><error message="error"></error></testcase></testsuite>' [1] + exit 1 [1] 1: always [1] Recording test results Failed in branch 1 [2] Sleeping a bit to ensure stage 1 has finished even though running in parallel [2] + sleep 3 [2] + exit 0 [2] 2: always [2] 2: success ERROR: script returned exit code 1 Finished: FAILURE This makes me make the following "conclusions": The parallel stage collects the result from each individual stage and reports the aggregate (worst). Each stage is independent (being parallel) as they should be, with regard to the outcome of the steps. However, the post-actions does not modify the result of the stage it is associated with only, but also the "global" result. At least in the specific case of the junit test collection that change the "global" result to unstable in addition to making stage 1 unstable. After stage 2 steps runs successfully and enters post, one would assume that stage 2's status would be success. However, since unstable is triggered, that makes me believe that post actions not only modify the "global" result but also are triggered by the combination of the "global" result and the stage's steps' result. Then we have the additional case of the post-action not needing to change the stage's status. In my second example with failure in stage 1's steps, the junit post-action doesn't need to change the stage's result and by that also, does not affect the "global" result. Because, when we enter stage 2's post-actions, it runs as for success. In summary: Post-actions, in addition to modifying the status of its associated stage, it also affects a "global" state. The post-action trigger (always/success/unstable/...) is affected by the aggregate of its associated stage's steps' result as well as the "global" result from other post-actions. An alternative interpretation would be for the post-block to be the same instance for all stages, and hence gets a semi-global transient state across the stages.

          James Nord added a comment -

          The documentation is incorrect. 

          Currently success and failure can look at the individual stage status.  Aborted, changed, unstable can only look at the pipeline status.

          James Nord added a comment - The documentation is incorrect.  Currently success and failure can look at the individual stage status.  Aborted, changed, unstable can only look at the pipeline status.

          teilo I'm not quite sure how to interpret your previous comment in the context of my not wanting one failed (or unstable) stage in a parallel group to make other successful stages in the same parallel take their not their post-.{failed,unsuccessful} actions rather than taking their post->success actions.

          IOW, every stage in a parallel group should take the post action path for their own stage.  Not the worse of the parallel group, or the worst of the job.  I just cannot fathom how either of those latter cases make any sense at all.  The post action stages for an individual stage are going to deal with the status of that stage and having them deal with it incorrectly because of some higher level aggregate status, again, makes absolutely no sense.

          If one wants post actions to happen on the aggregate of a parallel group, or the whole pipeline, one can add post actions at those levels.

          Brian J Murrell added a comment - teilo I'm not quite sure how to interpret your previous comment in the context of my not wanting one failed (or unstable) stage in a parallel group to make other successful stages in the same parallel take their not their post-.{failed,unsuccessful} actions rather than taking their post->success actions. IOW, every stage in a parallel group should take the post action path for their own stage.  Not the worse of the parallel group, or the worst of the job.  I just cannot fathom how either of those latter cases make any sense at all.  The post action stages for an individual stage are going to deal with the status of that stage and having them deal with it incorrectly because of some higher level aggregate status, again, makes absolutely no sense. If one wants post actions to happen on the aggregate of a parallel group, or the whole pipeline, one can add post actions at those levels.

          Max Trunov added a comment - - edited

          >>These condition blocks allow the execution of steps inside each condition depending on the completion status of the Pipeline or stage.

          But NOT

          This issue main cause that we still use the scripted pipeline. We don't have "stage" state.

          We have GLOBAL "build" state only and every "local" post(for stage) use it.

           

          Max Trunov added a comment - - edited >>These condition blocks allow the execution of steps inside each condition depending on the completion status of the Pipeline or stage. But NOT This issue main cause that we still use the scripted pipeline. We don't have "stage" state. We have GLOBAL "build" state only and every "local" post(for stage) use it.  

          Johnny added a comment -

          Johnny added a comment - See https://issues.jenkins.io/browse/JENKINS-72481 https://uno-online.io

          johnnycraig What is the relevance of your comment to this issue?

          Brian J Murrell added a comment - johnnycraig What is the relevance of your comment to this issue?

          David added a comment -

          Just bring this back to life. This still seems to be an issue in the last Jenkins version.

          I think per_bohlin alluded to this already, but it does seem to be that if you run a step that sets the build status explicitly  then any future stage that completes is then forced into a failure related post block.

          As they said and I've also seen, it seems to be a bit easier to see this with parallel blocks, since you can set failFast to false and with some sleeps essentially see how a stage that fails with an 'raw' error, doesn't 'infect' the rest of the stages, but when something happens that sets the build result, such as `unstable` or `warnError`, or as others have noted the `junit` step, the other stages fall into the failure related post stages.

          pipeline {
              agent any 
              stages {
                  stage('parallel') {
                      failFast false
                      parallel {
                          stage ('stage 1') {
                              steps {
                                  echo 'pre sleep'
                                  sleep 5
                                  echo 'post sleep'
                              }
                              post {
                                  success {
                                      echo "${env.STAGE_NAME} Success Post Stage"
                                  }
                                  unsuccessful {
                                      echo "${env.STAGE_NAME} Unsuccessful Post Stage"
                                  }
                              }
                          }
                          
                          stage ('stage 2') {
                              steps {
                                  script {
                                      error 'BANG!'
                                  }
                              }
                              post {
                                  success {
                                      echo "${env.STAGE_NAME} Success Post Stage"
                                  }
                                  unsuccessful {
                                      echo "${env.STAGE_NAME} Unsuccessful Post Stage"
                                  }
                              }
                          }
                          stage ('stage 3') {
                              steps {
                                  echo 'pre sleep'
                                  sleep 5
                                  echo 'post sleep'
                              }
                              post {
                                  success {
                                      echo "${env.STAGE_NAME} Success Post Stage"
                                  }
                                  unsuccessful {
                                      echo "${env.STAGE_NAME} Unsuccessful Post Stage"
                                  }
                              }
                          }
                      }
                  }
              }    
              post {
              success {
                  echo "Whole Job Success Post Stage"
              }
              unsuccessful {
                  echo "Whole Job Unsuccessful Post Stage"
              }
          }
          }

          The above then outputs (summarized for simplicity)

          Stage 1: pre sleep
          Stage 1: Sleeping for 5 sec
          Stage 3: pre sleep
          Stage 3: Sleeping for 5 sec

          Stage 2: stage 2 Unsuccessful Post Stage
          Stage2 :Failed in branch stage 2

          Stage 3: stage 3 Success Post Stage
          Stage 1: stage 1 Success Post Stage

          Whole Job Unsuccessful Post Stage

          ERROR: BANG!
          Finished: FAILURE
           

          However if make a small change to that pipeline and have it use `unstable` we can see a difference in behavior

           

          pipeline {
              agent any 
              stages {
                  stage('parallel') {
                      failFast false
                      parallel {
                          stage ('stage 1') {
                              steps {
                                  echo 'pre sleep'
                                  sleep 5
                                  echo 'post sleep'
                              }
                              post {
                                  success {
                                      echo "${env.STAGE_NAME} Success Post Stage"
                                  }
                                  failure {
                                      echo "${env.STAGE_NAME} Unsuccessful Post Stage"
                                  }
                              }
                          }
                          
                          stage ('stage 2') {
                              steps {
                                  unstable 'BANG!'
                              }
                              post {
                                  success {
                                      echo "${env.STAGE_NAME} Success Post Stage"
                                  }
                                  unsuccessful {
                                      echo "${env.STAGE_NAME} Unsuccessful Post Stage"
                                  }
                              }
                          }
                          stage ('stage 3') {
                              steps {
                                  echo 'pre sleep'
                                  sleep 5
                                  echo 'post sleep'
                              }
                              post {
                                  success {
                                      echo "${env.STAGE_NAME} Success Post Stage"
                                  }
                                  failure {
                                      echo "${env.STAGE_NAME} Unsuccessful Post Stage"
                                  }
                              }
                          }
                      }
                  }
              }    
              post {
                  success {
                       echo "Whole Job Success Post Stage"
                  }
                  unsuccessful {
                      echo "Whole Job Unsuccessful Post Stage"
                  }
              }
          } 

          that outputs (again summarized)

          Stage 1: pre sleep
          Stage 1: Sleeping for 5 sec
          Stage 3: pre sleep
          Stage 3: Sleeping for 5 sec

          Stage 2: WARNING: a smaller bang!
          Stage2 : stage 2 Unsuccessful Post Stage

          Stage 3: stage 3 Unsuccessful Post Stage
          Stage 1: stage 1 Unsuccessful Post Stage

          Whole Job Unsuccessful Post Stage
          Finished: UNSTABLE
           

          David added a comment - Just bring this back to life. This still seems to be an issue in the last Jenkins version. I think per_bohlin alluded to this already, but it does seem to be that if you run a step that sets the build status explicitly  then any future stage that completes is then forced into a failure related post block. As they said and I've also seen, it seems to be a bit easier to see this with parallel blocks, since you can set failFast to false and with some sleeps essentially see how a stage that fails with an 'raw' error, doesn't 'infect' the rest of the stages, but when something happens that sets the build result, such as `unstable` or `warnError`, or as others have noted the `junit` step, the other stages fall into the failure related post stages. pipeline {     agent any      stages {         stage( 'parallel' ) {             failFast false             parallel {                 stage ( 'stage 1' ) {                     steps {                         echo 'pre sleep'                         sleep 5                         echo 'post sleep'                     }                     post {                         success {                             echo "${env.STAGE_NAME} Success Post Stage"                         }                         unsuccessful {                             echo "${env.STAGE_NAME} Unsuccessful Post Stage"                         }                     }                 }                                  stage ( 'stage 2' ) {                     steps {                         script {                             error 'BANG!'                         }                     }                     post {                         success {                             echo "${env.STAGE_NAME} Success Post Stage"                         }                         unsuccessful {                             echo "${env.STAGE_NAME} Unsuccessful Post Stage"                         }                     }                 }                 stage ( 'stage 3' ) {                     steps {                         echo 'pre sleep'                         sleep 5                         echo 'post sleep'                     }                     post {                         success {                             echo "${env.STAGE_NAME} Success Post Stage"                         } unsuccessful {                             echo "${env.STAGE_NAME} Unsuccessful Post Stage"                         }                     }                 }             }         }     }    post {     success {         echo "Whole Job Success Post Stage"     }     unsuccessful {         echo "Whole Job Unsuccessful Post Stage"     } } } The above then outputs (summarized for simplicity) Stage 1: pre sleep Stage 1: Sleeping for 5 sec Stage 3: pre sleep Stage 3: Sleeping for 5 sec Stage 2: stage 2 Unsuccessful Post Stage Stage2 :Failed in branch stage 2 Stage 3: stage 3 Success Post Stage Stage 1: stage 1 Success Post Stage Whole Job Unsuccessful Post Stage ERROR: BANG! Finished: FAILURE   However if make a small change to that pipeline and have it use `unstable` we can see a difference in behavior   pipeline {     agent any      stages {         stage( 'parallel' ) {             failFast false             parallel {                 stage ( 'stage 1' ) {                     steps {                         echo 'pre sleep'                         sleep 5                         echo 'post sleep'                     }                     post {                         success {                             echo "${env.STAGE_NAME} Success Post Stage"                         }                         failure {                             echo "${env.STAGE_NAME} Unsuccessful Post Stage"                         }                     }                 }                                  stage ( 'stage 2' ) {                     steps {                         unstable 'BANG!'                     }                     post {                         success {                             echo "${env.STAGE_NAME} Success Post Stage"                         }                         unsuccessful {                             echo "${env.STAGE_NAME} Unsuccessful Post Stage"                         }                     }                 }                 stage ( 'stage 3' ) {                     steps {                         echo 'pre sleep'                         sleep 5                         echo 'post sleep'                     }                     post {                         success {                             echo "${env.STAGE_NAME} Success Post Stage"                         }                         failure {                             echo "${env.STAGE_NAME} Unsuccessful Post Stage"                         }                     }                 }             }         }     }    post {     success {         echo "Whole Job Success Post Stage"     }     unsuccessful {         echo "Whole Job Unsuccessful Post Stage"     } } } that outputs (again summarized) Stage 1: pre sleep Stage 1: Sleeping for 5 sec Stage 3: pre sleep Stage 3: Sleeping for 5 sec Stage 2: WARNING: a smaller bang! Stage2 : stage 2 Unsuccessful Post Stage Stage 3: stage 3 Unsuccessful Post Stage Stage 1: stage 1 Unsuccessful Post Stage Whole Job Unsuccessful Post Stage Finished: UNSTABLE  

            Unassigned Unassigned
            dcendents Daniel Beland
            Votes:
            10 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated: