Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49073

build job propagate true propagates FAIL when state is UNSTABLE

    XMLWordPrintable

    Details

    • Similar Issues:
    • Released As:
      pipeline-build-step 2.10, workflow-cps 2.77

      Description

      Im using a build command in a parallel step and, when the job finishes with UNSTABLE it stops execution because main job (which executes build) becomes in a FAILURE status.

        Attachments

          Issue Links

            Activity

            jlp01 Jesús Lunar Pérez created issue -
            abayer Andrew Bayer made changes -
            Field Original Value New Value
            Component/s pipeline-build-step-plugin [ 21707 ]
            Component/s pipeline-utility-steps-plugin [ 21135 ]
            Assignee rsandell [ rsandell ]
            jglick Jesse Glick made changes -
            Assignee Jesse Glick [ jglick ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "#24 (Web Link)" [ 23044 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            Hide
            jglick Jesse Glick added a comment -

            As designed, except that the main build should be set to UNSTABLE as well.

            Show
            jglick Jesse Glick added a comment - As designed, except that the main build should be set to UNSTABLE as well.
            jglick Jesse Glick made changes -
            Released As 2.10
            Resolution Fixed [ 1 ]
            Status In Review [ 10005 ] Fixed but Unreleased [ 10203 ]
            Hide
            basil Basil Crow added a comment -

            As designed,

            I have an upstream job that runs three downstream jobs in parallel to perform various types of testing (unit tests, integration tests, functional tests, etc). The behavior of the parallel step is currently problematic when combined with the default behavior of the build step (propagate: true). Suppose the following scenario occurs:

            1. The unit test job finishes first with a Result of SUCCESS.
            2. The integration test job finishes second with a Result of UNSTABLE.
            3. The functional test job finishes last with a Result of FAILURE.

            With the current behavior as of workflow-cps 2.74 and (the as-yet-unreleased) pipeline-build-step 2.10, the upstream job's Result will be UNSTABLE, because the integration test job (which finishes before the functional test job) has a Result of UNSTABLE, even though the functional test job has a Result of FAILURE. While I understand that this is behaving as designed/documented, the current behavior isn't very conducive to my use case. In my use case, I want to report to the end user the "worst" of the downstream results in the result of the upstream job; i.e., given a set of downstream results of SUCCESS, UNSTABLE, and FAILURE, I want the upstream job to report FAILURE since it is the most serious of the three. Reporting UNSTABLE gives a false impression to the user because it hides the fact that a more serious failure occurred in one of the downstream jobs, simply because that failure occurred after a less serious failure in a different downstream job (which is irrelevant to the end user).

            So far, I've been able to work around this by setting propagate: false and adding some hacky logic like this to my pipeline:

            parallel buildSteps
            
            def allNotBuilt = true
            def seenUnstable = false
            def seenFailure = false
            buildResults.each { jobName, jobResult ->
              if (jobResult.result != 'NOT_BUILT') {
                allNotBuilt = false
                if (jobResult.result == 'UNSTABLE') {
                  seenUnstable = true
                }
                if (jobResult.result == 'FAILURE' || jobResult.result == 'ABORTED') {
                  seenFailure = true
                }
              }
            }
            if (!allNotBuilt) {
              if (seenUnstable && !seenFailure) {
                currentBuild.result = 'UNSTABLE'
              } else if (seenFailure) {
                currentBuild.result = 'FAILURE'
              }
            } else {
              currentBuild.result = 'NOT_BUILT'
            }
            

            This type of logic is gross. Ideally, I think Pipeline would natively support this use case, perhaps through some new type of functionality in the parallel step that examines the result in each branch and propagates the "worst" one to the overall build result of the upstream job. What do you think? (Note that I'm interested in helping to implement this, if we want to go in this direction.)

            Show
            basil Basil Crow added a comment - As designed, I have an upstream job that runs three downstream jobs in parallel to perform various types of testing (unit tests, integration tests, functional tests, etc). The behavior of the parallel step is currently problematic when combined with the default behavior of the build step ( propagate: true ). Suppose the following scenario occurs: The unit test job finishes first with a Result of SUCCESS . The integration test job finishes second with a Result of UNSTABLE . The functional test job finishes last with a Result of FAILURE . With the current behavior as of workflow-cps 2.74 and (the as-yet-unreleased) pipeline-build-step 2.10, the upstream job's Result will be UNSTABLE , because the integration test job (which finishes before the functional test job) has a Result of UNSTABLE , even though the functional test job has a Result of FAILURE . While I understand that this is behaving as designed/documented, the current behavior isn't very conducive to my use case. In my use case, I want to report to the end user the "worst" of the downstream results in the result of the upstream job; i.e., given a set of downstream results of SUCCESS , UNSTABLE , and FAILURE , I want the upstream job to report FAILURE since it is the most serious of the three. Reporting UNSTABLE gives a false impression to the user because it hides the fact that a more serious failure occurred in one of the downstream jobs, simply because that failure occurred after a less serious failure in a different downstream job (which is irrelevant to the end user). So far, I've been able to work around this by setting propagate: false and adding some hacky logic like this to my pipeline: parallel buildSteps def allNotBuilt = true def seenUnstable = false def seenFailure = false buildResults.each { jobName, jobResult -> if (jobResult.result != 'NOT_BUILT' ) { allNotBuilt = false if (jobResult.result == 'UNSTABLE' ) { seenUnstable = true } if (jobResult.result == 'FAILURE' || jobResult.result == 'ABORTED' ) { seenFailure = true } } } if (!allNotBuilt) { if (seenUnstable && !seenFailure) { currentBuild.result = 'UNSTABLE' } else if (seenFailure) { currentBuild.result = 'FAILURE' } } else { currentBuild.result = 'NOT_BUILT' } This type of logic is gross. Ideally, I think Pipeline would natively support this use case, perhaps through some new type of functionality in the parallel step that examines the result in each branch and propagates the "worst" one to the overall build result of the upstream job. What do you think? (Note that I'm interested in helping to implement this, if we want to go in this direction.)
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-cps #325 (Web Link)" [ 23702 ]
            Hide
            jglick Jesse Glick added a comment -

            For the record, workflow-cps #325 is handling the latter suggestion.

            Show
            jglick Jesse Glick added a comment - For the record, workflow-cps #325 is handling the latter suggestion.
            Hide
            dnusbaum Devin Nusbaum added a comment - - edited

            A fix for this issue was just released in Pipeline: Build Step Plugin version 2.10. Note that if you use this step inside of the parallel step, you may wish to update Pipeline: Groovy Plugin to version 2.77 at the same time so that the result of parallel steps is the worst result of all branches rather than the result of the first completed branch.

            Show
            dnusbaum Devin Nusbaum added a comment - - edited A fix for this issue was just released in Pipeline: Build Step Plugin version 2.10. Note that if you use this step inside of the parallel step, you may wish to update Pipeline: Groovy Plugin to version 2.77 at the same time so that the result of parallel steps is the worst result of all branches rather than the result of the first completed branch.
            dnusbaum Devin Nusbaum made changes -
            Released As 2.10 pipeline-build-step 2.10
            Status Fixed but Unreleased [ 10203 ] Resolved [ 5 ]
            dnusbaum Devin Nusbaum made changes -
            Released As pipeline-build-step 2.10 pipeline-build-step 2.10, workflow-cps 2.77
            dnusbaum Devin Nusbaum made changes -
            Link This issue causes JENKINS-60354 [ JENKINS-60354 ]
            dnusbaum Devin Nusbaum made changes -
            Link This issue causes JENKINS-60354 [ JENKINS-60354 ]
            dnusbaum Devin Nusbaum made changes -
            Link This issue causes JENKINS-60354 [ JENKINS-60354 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-27092 [ JENKINS-27092 ]
            Hide
            pjung Philipp Jung added a comment -

            I can reproduce this again on both our Jenkins instances: with Build-Step 2.12 and Pipeline Groovy 2.80.

            Here is a minimal example: The second part of the "unstable -> success"  stage is never executed.

            // Parent Job
            node(){
                def stage_map = [:]
                stage_map['success'] = trigger('SUCCESS')
                stage_map['unstable -> success']= {trigger('UNSTABLE').call(); trigger('SUCCESS').call()} // The second SUCCESS step is never triggered
                
                parallel(stage_map)
            }
            
            def trigger(status){
                return {
                    def result = build job:
                        'test_child',
                        parameters: [
                                string(name: 'status', value: status)
                            ]
                }
            }
            
            // Child Job
            node(){
                  properties([
                    parameters([string(name: 'status', defaultValue: 'SUCCESS')])
                ]) 
                currentBuild.result = params.status
            }
            
            Show
            pjung Philipp Jung added a comment - I can reproduce this again on both our Jenkins instances: with Build-Step 2.12 and Pipeline Groovy 2.80. Here is a minimal example: The second part of the "unstable -> success"  stage is never executed. // Parent Job node(){ def stage_map = [:] stage_map[ 'success' ] = trigger( 'SUCCESS' ) stage_map[ 'unstable -> success' ]= {trigger( 'UNSTABLE' ).call(); trigger( 'SUCCESS' ).call()} // The second SUCCESS step is never triggered parallel(stage_map) } def trigger(status){ return { def result = build job: 'test_child' , parameters: [ string(name: 'status' , value: status) ] } } // Child Job node(){ properties([ parameters([string(name: 'status' , defaultValue: 'SUCCESS' )]) ]) currentBuild.result = params.status }
            Hide
            jglick Jesse Glick added a comment -

            The second part of the "unstable -> success" stage is never executed.

            As designed. Your parent job should terminate at that point with an UNSTABLE status. This change does not make the build step complete normally; it just makes the exception thrown indicate to the Pipeline machinery that (if uncaught) the build should have the indicated status. If you do not want that, then use propagate: false as before.

            Show
            jglick Jesse Glick added a comment - The second part of the "unstable -> success" stage is never executed. As designed. Your parent job should terminate at that point with an UNSTABLE status. This change does not make the build step complete normally; it just makes the exception thrown indicate to the Pipeline machinery that (if uncaught) the build should have the indicated status. If you do not want that, then use propagate: false as before.
            Hide
            pjung Philipp Jung added a comment -

            Thanks for clearing that up.

            Show
            pjung Philipp Jung added a comment - Thanks for clearing that up.
            Hide
            jayspang Jay Spang added a comment - - edited

            This fix introduced a new bug in my pipeline.

            The `build` step used to throw hudson.AbortException, but was changed to org.jenkinsci.plugins.workflow.steps.FlowInterruptedException.

            The problem: AbortException had a "message" which contained the job name/number, along with its status. We used this in our error handling to include context to the error. FlowInterruptedException contains no message, so our error handling fails. Unless I'm missing something, it contains almost no information on what actually failed.

            Should this bug be re-opened, or a new bug opened to pass FlowInterruptedException some contextual information?

            EDIT: Here's a code snippet to help explain:

            try{
                def results = build job: 'fooJob'
                def job_number = results.getNumber()
            } catch (e) {
                // AbortMessage included a message along the lines of...
                // "fooJob #13 completed with status FAILURE (propagate: false to ignore)"
                // We used this message to add context to the error
                def msg = e.getMessage() // Sadly, this is null for FlowInterruptedException
            }
            
            Show
            jayspang Jay Spang added a comment - - edited This fix introduced a new bug in my pipeline. The `build` step used to throw hudson.AbortException , but was changed to org.jenkinsci.plugins.workflow.steps.FlowInterruptedException . The problem: AbortException had a "message" which contained the job name/number, along with its status. We used this in our error handling to include context to the error. FlowInterruptedException contains no message, so our error handling fails. Unless I'm missing something, it contains almost no information on what actually failed. Should this bug be re-opened, or a new bug opened to pass FlowInterruptedException some contextual information? EDIT: Here's a code snippet to help explain: try { def results = build job: 'fooJob' def job_number = results.getNumber() } catch (e) { // AbortMessage included a message along the lines of... // "fooJob #13 completed with status FAILURE (propagate: false to ignore)" // We used this message to add context to the error def msg = e.getMessage() // Sadly, this is null for FlowInterruptedException }
            Hide
            jglick Jesse Glick added a comment -

            Should this bug be re-opened

            As a general rule: no, not unless the originally described bug was not in fact fixed. If a change causes some undesirable behavior, file a separate bug with a Link to the original.

            In this case there is no bug, your script was simply relying on undocumented behavior. The FlowInterruptedException does include all the same information, just in a DownstreamFailureCause rather than the Throwable.message.

            Anyway using a catch block and the exception here was the wrong idiom even before this change. Set propagate: false and inspect the resulting build object’s status and other fields.

            Show
            jglick Jesse Glick added a comment - Should this bug be re-opened As a general rule: no, not unless the originally described bug was not in fact fixed. If a change causes some undesirable behavior, file a separate bug with a Link to the original. In this case there is no bug, your script was simply relying on undocumented behavior. The FlowInterruptedException does include all the same information, just in a DownstreamFailureCause rather than the Throwable.message . Anyway using a catch block and the exception here was the wrong idiom even before this change. Set propagate: false and inspect the resulting build object’s status and other fields.

              People

              Assignee:
              jglick Jesse Glick
              Reporter:
              jlp01 Jesús Lunar Pérez
              Votes:
              5 Vote for this issue
              Watchers:
              10 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: