Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49073

build job propagate true propagates FAIL when state is UNSTABLE

    • pipeline-build-step 2.10, workflow-cps 2.77

      Im using a build command in a parallel step and, when the job finishes with UNSTABLE it stops execution because main job (which executes build) becomes in a FAILURE status.

          [JENKINS-49073] build job propagate true propagates FAIL when state is UNSTABLE

          Jesse Glick added a comment -

          As designed, except that the main build should be set to UNSTABLE as well.

          Jesse Glick added a comment - As designed, except that the main build should be set to UNSTABLE as well.

          Basil Crow added a comment -

          As designed,

          I have an upstream job that runs three downstream jobs in parallel to perform various types of testing (unit tests, integration tests, functional tests, etc). The behavior of the parallel step is currently problematic when combined with the default behavior of the build step (propagate: true). Suppose the following scenario occurs:

          1. The unit test job finishes first with a Result of SUCCESS.
          2. The integration test job finishes second with a Result of UNSTABLE.
          3. The functional test job finishes last with a Result of FAILURE.

          With the current behavior as of workflow-cps 2.74 and (the as-yet-unreleased) pipeline-build-step 2.10, the upstream job's Result will be UNSTABLE, because the integration test job (which finishes before the functional test job) has a Result of UNSTABLE, even though the functional test job has a Result of FAILURE. While I understand that this is behaving as designed/documented, the current behavior isn't very conducive to my use case. In my use case, I want to report to the end user the "worst" of the downstream results in the result of the upstream job; i.e., given a set of downstream results of SUCCESS, UNSTABLE, and FAILURE, I want the upstream job to report FAILURE since it is the most serious of the three. Reporting UNSTABLE gives a false impression to the user because it hides the fact that a more serious failure occurred in one of the downstream jobs, simply because that failure occurred after a less serious failure in a different downstream job (which is irrelevant to the end user).

          So far, I've been able to work around this by setting propagate: false and adding some hacky logic like this to my pipeline:

          parallel buildSteps
          
          def allNotBuilt = true
          def seenUnstable = false
          def seenFailure = false
          buildResults.each { jobName, jobResult ->
            if (jobResult.result != 'NOT_BUILT') {
              allNotBuilt = false
              if (jobResult.result == 'UNSTABLE') {
                seenUnstable = true
              }
              if (jobResult.result == 'FAILURE' || jobResult.result == 'ABORTED') {
                seenFailure = true
              }
            }
          }
          if (!allNotBuilt) {
            if (seenUnstable && !seenFailure) {
              currentBuild.result = 'UNSTABLE'
            } else if (seenFailure) {
              currentBuild.result = 'FAILURE'
            }
          } else {
            currentBuild.result = 'NOT_BUILT'
          }
          

          This type of logic is gross. Ideally, I think Pipeline would natively support this use case, perhaps through some new type of functionality in the parallel step that examines the result in each branch and propagates the "worst" one to the overall build result of the upstream job. What do you think? (Note that I'm interested in helping to implement this, if we want to go in this direction.)

          Basil Crow added a comment - As designed, I have an upstream job that runs three downstream jobs in parallel to perform various types of testing (unit tests, integration tests, functional tests, etc). The behavior of the parallel step is currently problematic when combined with the default behavior of the build step ( propagate: true ). Suppose the following scenario occurs: The unit test job finishes first with a Result of SUCCESS . The integration test job finishes second with a Result of UNSTABLE . The functional test job finishes last with a Result of FAILURE . With the current behavior as of workflow-cps 2.74 and (the as-yet-unreleased) pipeline-build-step 2.10, the upstream job's Result will be UNSTABLE , because the integration test job (which finishes before the functional test job) has a Result of UNSTABLE , even though the functional test job has a Result of FAILURE . While I understand that this is behaving as designed/documented, the current behavior isn't very conducive to my use case. In my use case, I want to report to the end user the "worst" of the downstream results in the result of the upstream job; i.e., given a set of downstream results of SUCCESS , UNSTABLE , and FAILURE , I want the upstream job to report FAILURE since it is the most serious of the three. Reporting UNSTABLE gives a false impression to the user because it hides the fact that a more serious failure occurred in one of the downstream jobs, simply because that failure occurred after a less serious failure in a different downstream job (which is irrelevant to the end user). So far, I've been able to work around this by setting propagate: false and adding some hacky logic like this to my pipeline: parallel buildSteps def allNotBuilt = true def seenUnstable = false def seenFailure = false buildResults.each { jobName, jobResult -> if (jobResult.result != 'NOT_BUILT' ) { allNotBuilt = false if (jobResult.result == 'UNSTABLE' ) { seenUnstable = true } if (jobResult.result == 'FAILURE' || jobResult.result == 'ABORTED' ) { seenFailure = true } } } if (!allNotBuilt) { if (seenUnstable && !seenFailure) { currentBuild.result = 'UNSTABLE' } else if (seenFailure) { currentBuild.result = 'FAILURE' } } else { currentBuild.result = 'NOT_BUILT' } This type of logic is gross. Ideally, I think Pipeline would natively support this use case, perhaps through some new type of functionality in the parallel step that examines the result in each branch and propagates the "worst" one to the overall build result of the upstream job. What do you think? (Note that I'm interested in helping to implement this, if we want to go in this direction.)

          Jesse Glick added a comment -

          For the record, workflow-cps #325 is handling the latter suggestion.

          Jesse Glick added a comment - For the record, workflow-cps #325 is handling the latter suggestion.

          Devin Nusbaum added a comment - - edited

          A fix for this issue was just released in Pipeline: Build Step Plugin version 2.10. Note that if you use this step inside of the parallel step, you may wish to update Pipeline: Groovy Plugin to version 2.77 at the same time so that the result of parallel steps is the worst result of all branches rather than the result of the first completed branch.

          Devin Nusbaum added a comment - - edited A fix for this issue was just released in Pipeline: Build Step Plugin version 2.10. Note that if you use this step inside of the parallel step, you may wish to update Pipeline: Groovy Plugin to version 2.77 at the same time so that the result of parallel steps is the worst result of all branches rather than the result of the first completed branch.

          Philipp Jung added a comment -

          I can reproduce this again on both our Jenkins instances: with Build-Step 2.12 and Pipeline Groovy 2.80.

          Here is a minimal example: The second part of the "unstable -> success"  stage is never executed.

          // Parent Job
          node(){
              def stage_map = [:]
              stage_map['success'] = trigger('SUCCESS')
              stage_map['unstable -> success']= {trigger('UNSTABLE').call(); trigger('SUCCESS').call()} // The second SUCCESS step is never triggered
              
              parallel(stage_map)
          }
          
          def trigger(status){
              return {
                  def result = build job:
                      'test_child',
                      parameters: [
                              string(name: 'status', value: status)
                          ]
              }
          }
          
          // Child Job
          node(){
                properties([
                  parameters([string(name: 'status', defaultValue: 'SUCCESS')])
              ]) 
              currentBuild.result = params.status
          }
          

          Philipp Jung added a comment - I can reproduce this again on both our Jenkins instances: with Build-Step 2.12 and Pipeline Groovy 2.80. Here is a minimal example: The second part of the "unstable -> success"  stage is never executed. // Parent Job node(){ def stage_map = [:] stage_map[ 'success' ] = trigger( 'SUCCESS' ) stage_map[ 'unstable -> success' ]= {trigger( 'UNSTABLE' ).call(); trigger( 'SUCCESS' ).call()} // The second SUCCESS step is never triggered parallel(stage_map) } def trigger(status){ return { def result = build job: 'test_child' , parameters: [ string(name: 'status' , value: status) ] } } // Child Job node(){ properties([ parameters([string(name: 'status' , defaultValue: 'SUCCESS' )]) ]) currentBuild.result = params.status }

          Jesse Glick added a comment -

          The second part of the "unstable -> success" stage is never executed.

          As designed. Your parent job should terminate at that point with an UNSTABLE status. This change does not make the build step complete normally; it just makes the exception thrown indicate to the Pipeline machinery that (if uncaught) the build should have the indicated status. If you do not want that, then use propagate: false as before.

          Jesse Glick added a comment - The second part of the "unstable -> success" stage is never executed. As designed. Your parent job should terminate at that point with an UNSTABLE status. This change does not make the build step complete normally; it just makes the exception thrown indicate to the Pipeline machinery that (if uncaught) the build should have the indicated status. If you do not want that, then use propagate: false as before.

          Philipp Jung added a comment -

          Thanks for clearing that up.

          Philipp Jung added a comment - Thanks for clearing that up.

          Jay Spang added a comment - - edited

          This fix introduced a new bug in my pipeline.

          The `build` step used to throw hudson.AbortException, but was changed to org.jenkinsci.plugins.workflow.steps.FlowInterruptedException.

          The problem: AbortException had a "message" which contained the job name/number, along with its status. We used this in our error handling to include context to the error. FlowInterruptedException contains no message, so our error handling fails. Unless I'm missing something, it contains almost no information on what actually failed.

          Should this bug be re-opened, or a new bug opened to pass FlowInterruptedException some contextual information?

          EDIT: Here's a code snippet to help explain:

          try{
              def results = build job: 'fooJob'
              def job_number = results.getNumber()
          } catch (e) {
              // AbortMessage included a message along the lines of...
              // "fooJob #13 completed with status FAILURE (propagate: false to ignore)"
              // We used this message to add context to the error
              def msg = e.getMessage() // Sadly, this is null for FlowInterruptedException
          }
          

          Jay Spang added a comment - - edited This fix introduced a new bug in my pipeline. The `build` step used to throw hudson.AbortException , but was changed to org.jenkinsci.plugins.workflow.steps.FlowInterruptedException . The problem: AbortException had a "message" which contained the job name/number, along with its status. We used this in our error handling to include context to the error. FlowInterruptedException contains no message, so our error handling fails. Unless I'm missing something, it contains almost no information on what actually failed. Should this bug be re-opened, or a new bug opened to pass FlowInterruptedException some contextual information? EDIT: Here's a code snippet to help explain: try { def results = build job: 'fooJob' def job_number = results.getNumber() } catch (e) { // AbortMessage included a message along the lines of... // "fooJob #13 completed with status FAILURE (propagate: false to ignore)" // We used this message to add context to the error def msg = e.getMessage() // Sadly, this is null for FlowInterruptedException }

          Jesse Glick added a comment -

          Should this bug be re-opened

          As a general rule: no, not unless the originally described bug was not in fact fixed. If a change causes some undesirable behavior, file a separate bug with a Link to the original.

          In this case there is no bug, your script was simply relying on undocumented behavior. The FlowInterruptedException does include all the same information, just in a DownstreamFailureCause rather than the Throwable.message.

          Anyway using a catch block and the exception here was the wrong idiom even before this change. Set propagate: false and inspect the resulting build object’s status and other fields.

          Jesse Glick added a comment - Should this bug be re-opened As a general rule: no, not unless the originally described bug was not in fact fixed. If a change causes some undesirable behavior, file a separate bug with a Link to the original. In this case there is no bug, your script was simply relying on undocumented behavior. The FlowInterruptedException does include all the same information, just in a DownstreamFailureCause rather than the Throwable.message . Anyway using a catch block and the exception here was the wrong idiom even before this change. Set propagate: false and inspect the resulting build object’s status and other fields.

            jglick Jesse Glick
            jlp01 Jesús Lunar Pérez
            Votes:
            5 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: