Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61405

Stage post conditions doomed by build result becoming unstable

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      In a pipeline with multiple sequential stages that typically have either an unstable or a success result, stage post conditions can't be used to take any meaningful action, because one unstable stage dooms the subsequent stages to fire the unsuccessful post condition instead of the success post condition.

      Pipeline:

       

      pipeline {
          agent any
          stages {
              stage('First') {
                  steps {
                      echo 'Before step with error'
                      catchError(message: 'Step had error.', stageResult: 'UNSTABLE') {
                          sh 'nonexistentcommand'
                      }
                      echo 'After step with error'
                  }
                  post {
                      success {
                          echo 'First stage post success'
                      }
                      unsuccessful {
                          echo 'First stage post unsuccessful'
                      }
                  }
              }
              stage('Second') {
                  steps {
                      echo 'Stage after unstable stage'
                  }
                  post {
                      success {
                          echo 'Second stage post success'
                      }
                      unsuccessful {
                          echo 'Second stage post unsuccessful'
                      }
                  }
              }
          }
      }

      Console Output:

       

      Running in Durability level: MAX_SURVIVABILITY
      [Pipeline] Start of Pipeline
      [Pipeline] node
      Running on Jenkins in ...
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (First)
      [Pipeline] echo
      Before step with error
      [Pipeline] catchError
      [Pipeline] {
      [Pipeline] sh
      + nonexistentcommand
      .../script.sh: line 1: nonexistentcommand: command not found
      [Pipeline] }
      ERROR: Step had error.
      ERROR: script returned exit code 127
      [Pipeline] // catchError
      [Pipeline] echo
      After step with error
      Post stage
      [Pipeline] echo
      First stage post unsuccessful
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] stage
      [Pipeline] { (Second)
      [Pipeline] echo
      Stage after unstable stage
      Post stage
      [Pipeline] echo
      Second stage post unsuccessful
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      Finished: FAILURE
      

      Resolved issue https://issues.jenkins-ci.org/browse/JENKINS-52114 references making the post conditions "more stage specific", but from this test, the conditions don't seem stage-specific at all.  It's really unclear what the current behavior for the stage post conditions is supposed to be. 

      Unresolved issue https://issues.jenkins-ci.org/browse/JENKINS-59469 is a little bit similar, in that it asks for a variable with the stage result that can be used in the post section.  That would address our overall use case here too (which is sending per-stage notifications).  But just having post conditions that referred to the stage would be helpful.

      This behavior I think is also described in the commentary on JENKINS-39203https://issues.jenkins-ci.org/browse/JENKINS-39203?focusedCommentId=350599&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-350599

      If there are any workarounds for the current behavior, I'd love to give them a try.

        Attachments

          Activity

          Hide
          jkd Jim D added a comment -

          Please note, right now I'm testing a workaround that's based on the code in pipeline-model-extensions/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/model/BuildCondition.java.

          That's the code that seems to return a counter-intuitive result that mixes some stage-specific things with non-stage-specific things.

          I can create a global pipeline library method that basically pulls out the 3 components that BuildCondition uses, errorResult, execResult, and prevResult, and inspect and use them directly:

          @NonCPS
          Map<String,String> getStageConditionResults () {
              Map<String,String> results = [:]
              String stageName = env.STAGE_NAME
              WorkflowRun run = currentBuild.getRawBuild()
              Result errorResult = Result.SUCCESS;
              List<FlowNode> startAndEnd = CommonUtils.findPossiblyUnfinishedEndNodeForCurrentStage(stageName, run.getExecution());
              if (startAndEnd.size() == 2 && startAndEnd.get(0) != null && startAndEnd.get(1) != null) {
                  DepthFirstScanner scanner = new DepthFirstScanner();
                  if (scanner.setup(startAndEnd.get(1), Collections.singletonList(startAndEnd.get(0)))) {
                      WarningAction warningAction = StreamSupport.stream(scanner.spliterator(), false)
                          .map({node -> node.getPersistentAction(WarningAction.class)})
                          .filter(Objects.&nonNull)
                          .max(Comparator.comparing({warning -> warning.getResult().ordinal}))
                          .orElse(null);
                      if (warningAction != null) {
                          errorResult = warningAction.getResult();
                      }
                  }
              }
              Result execResult = run.getExecution().getResult()
              Result prevResult = run.getResult();
              if (prevResult == null) {
                  prevResult = Result.SUCCESS;
              }
              if (execResult == null) {
                  execResult = Result.SUCCESS;
              }
              results['errorResult'] = errorResult.toString()
              results['execResult'] = execResult.toString()
              results['prevResult'] = prevResult.toString()
              return results
          }
          

          What I notice is that unstable stages have UNSTABLE errorResult and successful stages have SUCCESS errorResult. But once one stage has UNSTABLE prevResult, all subsequent stages have UNSTABLE prevResult. It looks like in the BuildCondition code, that UNSTABLE prevResult will trump the SUCCESS errorResult and return UNSTABLE as the overall Result for a successful stage, firing stage post conditions like "unsuccessful". Really what would be good is a pair of stage post conditions like success/unsuccessful that ignores the UNSTABLE prevResult.

          In any case, I'm going to try using this pipeline library method in our stage post sections to detect when a stage is unstable and requires notifications.

          Is there any chance similar logic could be implemented by the existing post conditions, or added as new post conditions?

          Show
          jkd Jim D added a comment - Please note, right now I'm testing a workaround that's based on the code in pipeline-model-extensions/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/model/BuildCondition.java. That's the code that seems to return a counter-intuitive result that mixes some stage-specific things with non-stage-specific things. I can create a global pipeline library method that basically pulls out the 3 components that BuildCondition uses, errorResult, execResult, and prevResult, and inspect and use them directly: @NonCPS Map< String , String > getStageConditionResults () { Map< String , String > results = [:] String stageName = env.STAGE_NAME WorkflowRun run = currentBuild.getRawBuild() Result errorResult = Result.SUCCESS; List<FlowNode> startAndEnd = CommonUtils.findPossiblyUnfinishedEndNodeForCurrentStage(stageName, run.getExecution()); if (startAndEnd.size() == 2 && startAndEnd.get(0) != null && startAndEnd.get(1) != null ) { DepthFirstScanner scanner = new DepthFirstScanner(); if (scanner.setup(startAndEnd.get(1), Collections.singletonList(startAndEnd.get(0)))) { WarningAction warningAction = StreamSupport.stream(scanner.spliterator(), false ) .map({node -> node.getPersistentAction(WarningAction.class)}) .filter(Objects.&nonNull) .max(Comparator.comparing({warning -> warning.getResult().ordinal})) .orElse( null ); if (warningAction != null ) { errorResult = warningAction.getResult(); } } } Result execResult = run.getExecution().getResult() Result prevResult = run.getResult(); if (prevResult == null ) { prevResult = Result.SUCCESS; } if (execResult == null ) { execResult = Result.SUCCESS; } results[ 'errorResult' ] = errorResult.toString() results[ 'execResult' ] = execResult.toString() results[ 'prevResult' ] = prevResult.toString() return results } What I notice is that unstable stages have UNSTABLE errorResult and successful stages have SUCCESS errorResult. But once one stage has UNSTABLE prevResult, all subsequent stages have UNSTABLE prevResult. It looks like in the BuildCondition code, that UNSTABLE prevResult will trump the SUCCESS errorResult and return UNSTABLE as the overall Result for a successful stage, firing stage post conditions like "unsuccessful". Really what would be good is a pair of stage post conditions like success/unsuccessful that ignores the UNSTABLE prevResult. In any case, I'm going to try using this pipeline library method in our stage post sections to detect when a stage is unstable and requires notifications. Is there any chance similar logic could be implemented by the existing post conditions, or added as new post conditions?
          Hide
          famod Falko Modler added a comment -

          Just ran into the same problem today. Really annoying!

          Show
          famod Falko Modler added a comment - Just ran into the same problem today. Really annoying!

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            jkd Jim D
            Votes:
            4 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated: