Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-72575

Failed quality gate does not prevent whole pipeline execution

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Major Major
    • warnings-ng-plugin
    • Jenkins 2.426.2
      Warnings NG plugin 10.5.3

      While executing the recordIssues step with a defined quality gate to set stage and build results to "failed", it just sets status, but it doesn't prevent execution of the rest of the pipeline and I'm not sure if this is the expected behavior.

      In my case, I'm executing whole security checks before uploading built artifacts, and I would like to prevent this step execution for artifacts that do not meet specific security requirements (defined via quality gate). But even if the "Security scans" stage and even the whole build are set to "failed" due to the quality gate status, the next stages, including "Upload artifacts", are still executed.
       

      pipeline {
          agent {...}
      
          stages {
              stage("Earlier stages") {...}
              
              stage("Security scans") {
                  makeSecurityScans()
                  recordIssues tools: [...], qualityGates: [[threshold: 1, type: 'TOTAL_ERROR', unstable: false]]
              }
      
              stage("Upload artifacts") {
                  uploadArtifacts()
              }
          }
      }
      

          [JENKINS-72575] Failed quality gate does not prevent whole pipeline execution

          Ulli Hafner added a comment -

          Ulli Hafner added a comment - See https://github.com/jenkinsci/warnings-ng-plugin/pull/1649

          Łukasz Jackiewicz added a comment - - edited

          Before creating the issue, I was already looking through the open issues list and noticed JENKINS-72059, but unfortunately it's not a duplicate of my issue.

          Łukasz Jackiewicz added a comment - - edited Before creating the issue, I was already looking through the open issues list and noticed JENKINS-72059 , but unfortunately it's not a duplicate of my issue.

          The unexpected behavior here is the execution of subsequent stages of the pipeline even though the stage calling the recordIssue step has been marked as failed by it.

          And it doesn't make much sense to me.

          An example of the functionality I am writing about was implemented, among others, in the OWASP DependencyCheck plugin and is available there via the additional stopBuild flag of the dependencyCheckPublisher step.

          Łukasz Jackiewicz added a comment - The unexpected behavior here is the execution of subsequent stages of the pipeline even though the stage calling the recordIssue step has been marked as failed by it. And it doesn't make much sense to me. An example of the functionality I am writing about was implemented, among others, in the OWASP DependencyCheck plugin and is available there via the additional stopBuild flag of the dependencyCheckPublisher step.

          Ulli Hafner added a comment -

          Sorry, it seems that I misunderstood the problem.

          I am not aware that a plugin can stop a build. How can we achieve that? Shouldn't this be a Jenkins core or pipeline option how to handle unstable or failed results of a step?

          Ulli Hafner added a comment - Sorry, it seems that I misunderstood the problem. I am not aware that a plugin can stop a build. How can we achieve that? Shouldn't this be a Jenkins core or pipeline option how to handle unstable or failed results of a step?

          Unfortunately I don't know if there are better ways to achieve this, but referring to the OWASP DependencyCheck plugin, it seems to be achieved there by simply throwing an AbortException when the build status has previously been marked as failed in this process (due to set conditions, e.g. total amount of criticals equal or greater than 1) and the stopBuild flag has been set to true:
          https://github.com/jenkinsci/dependency-check-plugin/blob/master/src/main/java/org/jenkinsci/plugins/DependencyCheck/DependencyCheckPublisher.java#L119

          Example of execution of dependencyCheckPublisher step (within OWASP DependencyCheck stage):

          dependencyCheckPublisher(pattern: 'dependency-check-report.xml', failedTotalCritical: 1, stopBuild: true)
          

          Graph View for ODC:

          Stage View for ODC:

          Example of execution of recordIssues step (within Warnings NG: Trivy stage):

          recordIssues(tool: trivy(pattern: "reports/trivy-artifact-*.json"), qualityGates: [[threshold: 1, type: 'TOTAL_ERROR', unstable: false]])
          

          Graph View for Warnings NG:

          "SonarQube scan" and "Image build" stages here were skipped due to "when" condition set for these stages.

          Łukasz Jackiewicz added a comment - Unfortunately I don't know if there are better ways to achieve this, but referring to the OWASP DependencyCheck plugin, it seems to be achieved there by simply throwing an AbortException when the build status has previously been marked as failed in this process (due to set conditions, e.g. total amount of criticals equal or greater than 1) and the stopBuild flag has been set to true: https://github.com/jenkinsci/dependency-check-plugin/blob/master/src/main/java/org/jenkinsci/plugins/DependencyCheck/DependencyCheckPublisher.java#L119 Example of execution of dependencyCheckPublisher  step (within OWASP DependencyCheck stage): dependencyCheckPublisher(pattern: 'dependency-check-report.xml' , failedTotalCritical: 1, stopBuild: true ) Graph View for ODC: Stage View for ODC: Example of execution of recordIssues  step (within Warnings NG: Trivy stage): recordIssues(tool: trivy(pattern: "reports/trivy-artifact-*.json" ), qualityGates: [[threshold: 1, type: 'TOTAL_ERROR' , unstable: false ]]) Graph View for Warnings NG: "SonarQube scan" and "Image build" stages here were skipped due to "when" condition set for these stages.

          Ulli Hafner added a comment -

          Wouldn't it make sense to use different guards to the uploading step? Something like:

          stage('Third') {
                steps { 
                    recordIssues 
                }
                
                post {
                  success { 
                      uploadArtifacts() 
                  }
                  unstable { 
                      echo 'recordIssues was not successful' 
                  }
                }
              }
          

          When I throw such an exception by using an additional ABORT quality gate criticality (see https://github.com/jenkinsci/plugin-util-api-plugin/pull/309) then no warning results will be published in Jenkins, this is something I would like to avoid.

          Ulli Hafner added a comment - Wouldn't it make sense to use different guards to the uploading step? Something like: stage( 'Third' ) { steps { recordIssues } post { success { uploadArtifacts() } unstable { echo 'recordIssues was not successful' } } } When I throw such an exception by using an additional ABORT quality gate criticality (see https://github.com/jenkinsci/plugin-util-api-plugin/pull/309 ) then no warning results will be published in Jenkins, this is something I would like to avoid.

          I know that I could achieve this in many other ways, even keeping my current pipeline structure, what is important to me, and where I want to easily and clearly control under what conditions a specific stage should be executed, including the "Upload artifacts" stage, e.g. by checking the current build status in the following stages' "when" section, but it seems like unnecessary glue code to me.

          I also try to be aware that this may just be my point of view, but I don't know what benefit there is in marking a stage or build as failed (other than simply setting currentBuild.result = "FAILURE", but I expect it's more complicated underneath), if the entire pipeline execution is the same regardless of the quality gate status. I don't think it's a "gate" if it's always open and can't be closed.

          I trust that for various reasons we are not able to achieve this in this plugin, I have not delved into its code, but if I understand correctly what you mean by "warning results publishing", then other similar plugins handle this somehow.

          Additionally, I checked that the SonarQube plugin achieve this similarly to the OWASP DependencyCheck plugin (by throwing an exception):
          https://github.com/jenkinsci/sonarqube-plugin/blob/master/src/main/java/org/sonarsource/scanner/jenkins/pipeline/WaitForQualityGateStep.java#L323

          Łukasz Jackiewicz added a comment - I know that I could achieve this in many other ways, even keeping my current pipeline structure, what is important to me, and where I want to easily and clearly control under what conditions a specific stage should be executed, including the "Upload artifacts" stage, e.g. by checking the current build status in the following stages' "when" section, but it seems like unnecessary glue code to me. I also try to be aware that this may just be my point of view, but I don't know what benefit there is in marking a stage or build as failed (other than simply setting currentBuild.result = "FAILURE" , but I expect it's more complicated underneath), if the entire pipeline execution is the same regardless of the quality gate status. I don't think it's a "gate" if it's always open and can't be closed. I trust that for various reasons we are not able to achieve this in this plugin, I have not delved into its code, but if I understand correctly what you mean by "warning results publishing", then other similar plugins handle this somehow. Additionally, I checked that the SonarQube plugin achieve this similarly to the OWASP DependencyCheck plugin (by throwing an exception): https://github.com/jenkinsci/sonarqube-plugin/blob/master/src/main/java/org/sonarsource/scanner/jenkins/pipeline/WaitForQualityGateStep.java#L323

          Łukasz Jackiewicz added a comment - - edited

          Although this is still not the best solution, it is also possible to add a post failure section and call the error step there:

          stage("Security scans") {
              steps {
                  makeSecurityScans()
                  recordIssues tools: [...], qualityGates: [[threshold: 1, type: 'TOTAL_ERROR', unstable: false]]
              }
              post {
                  failure {
                      error("The project does not meet security rules.")
                  }
              }
          }
          

          The problem here is, probably due to the design of the post failure section itself, that in addition to our error itself, an error stacktrace is thrown in the log, which may be confusing for end users. So I can't recommend it, but I'm sharing it as an alternative workaround.

          Łukasz Jackiewicz added a comment - - edited Although this is still not the best solution, it is also possible to add a post failure section and call the error step there: stage( "Security scans" ) { steps { makeSecurityScans() recordIssues tools: [...], qualityGates: [[threshold: 1, type: 'TOTAL_ERROR' , unstable: false ]] } post { failure { error( "The project does not meet security rules." ) } } } The problem here is, probably due to the design of the post failure section itself, that in addition to our error itself, an error stacktrace is thrown in the log, which may be confusing for end users. So I can't recommend it, but I'm sharing it as an alternative workaround.

            Unassigned Unassigned
            ljackiewicz Łukasz Jackiewicz
            Votes:
            2 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: