Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-66696

Allow to set status UNSTABLE instead of FAILED

      In some pipelines we want to have result of SonarQube UNSTABLE instead of FAILED.
      Or at least get status of analysis instead of wrapping it in try/catch each time.

          [JENKINS-66696] Allow to set status UNSTABLE instead of FAILED

          Ian Williams added a comment -

          I thought a quality gate only had a pass/fail.
          What is the context for unstable?

          Ian Williams added a comment - I thought a quality gate only had a pass/fail. What is the context for unstable?

          ipleten added a comment -

          According to https://www.jenkins.io/doc/book/glossary/#unstable

          The Build had some errors but they were not fatal. A Build is unstable if it was built successfully and one or more publishers report it unstable. For example if the JUnit publisher is configured and a test fails then the Build will be marked unstable.
          

          We want to have a difference between pipeline status FAILED (in case something was wrong with agents or something really bad happened with build) and UNSTABLE (and skip all other stages so pipelines overall status would be UNSTABLE) to indicate that something is wrong with one of publishers.
          In case stage reported FAILED we can't demote it to UNSTABLE as Jenkins allows to promote statuses only to "the worse" direction not backward.
          Currently, agent gone during running and low test coverage both return FAILED.

          ipleten added a comment - According to https://www.jenkins.io/doc/book/glossary/#unstable The Build had some errors but they were not fatal. A Build is unstable if it was built successfully and one or more publishers report it unstable. For example if the JUnit publisher is configured and a test fails then the Build will be marked unstable. We want to have a difference between pipeline status FAILED (in case something was wrong with agents or something really bad happened with build) and UNSTABLE (and skip all other stages so pipelines overall status would be UNSTABLE) to indicate that something is wrong with one of publishers. In case stage reported FAILED we can't demote it to UNSTABLE as Jenkins allows to promote statuses only to "the worse" direction not backward. Currently, agent gone during running and low test coverage both return FAILED.

          Ian Williams added a comment -

          The glossary is reflective of the Jenkins "standalone" product. It does not consider what plugins may do.

          Also, the SonarQube PASS/FAIL criteria is build tool agnostic so is designed to report independently. The aggregate result is Code Coverage is being evaluated by Jenkins and by SonarQube (but only in the aggregate).

          If Coverage is the primary sematic concern, then simply omit Code Coverage from the SonarQube Quality Gate.

          The default (Sonar Way) is:

          Conditions on New Code
          - Metric -	Operator - 	Value
          - Coverage	is less than	80.0% 
          - Duplicated Lines (%)	is greater than	3.0% 
          - Maintainability Rating	is worse than	A 
          - Reliability Rating	is worse than	A 
          - Security Hotspots Reviewed	is less than	100% 
          - Security Rating	is worse than	A
          

          As the Quality Gate is a binary assessment of all Conditions, you really don't want to let one of the other criteria slip past (ie: a Vulnerability).

          I'm not a fan of the SonarQube binary option (but then never knew there was a 3rd state - Quality Gate condition - Warning). We have also created a Custom Quality Gate that addresses both "New Code" and "Overall Code" with criteria that meets our needs rather than Sonar's. As our legacy code base no longer has (or may not have had) complete code coverage and has significant duplication, those do not form part of the gate. We also have a higher threshold for Net New applications vs legacy.

          Ian Williams added a comment - The glossary is reflective of the Jenkins "standalone" product. It does not consider what plugins may do. Also, the SonarQube PASS/FAIL criteria is build tool agnostic so is designed to report independently. The aggregate result is Code Coverage is being evaluated by Jenkins and by SonarQube (but only in the aggregate). If Coverage is the primary sematic concern, then simply omit Code Coverage from the SonarQube Quality Gate . The default ( Sonar Way ) is: Conditions on New Code - Metric - Operator - Value - Coverage is less than 80.0% - Duplicated Lines (%) is greater than 3.0% - Maintainability Rating is worse than A - Reliability Rating is worse than A - Security Hotspots Reviewed is less than 100% - Security Rating is worse than A As the Quality Gate is a binary assessment of all Conditions , you really don't want to let one of the other criteria slip past (ie: a Vulnerability). I'm not a fan of the SonarQube binary option (but then never knew there was a 3rd state - Quality Gate condition - Warning ). We have also created a Custom Quality Gate that addresses both "New Code" and "Overall Code" with criteria that meets our needs rather than Sonar's. As our legacy code base no longer has (or may not have had) complete code coverage and has significant duplication, those do not form part of the gate. We also have a higher threshold for Net New applications vs legacy.

          Dan Alvizu added a comment -

          Hi Ian - I work with ipleten, so this is the same ask but allow me to clarify-

          The glossary is reflective of the Jenkins "standalone" product. It does not consider what plugins may do.

          The glossary directly references a plugin:

          For example if the JUnit publisher is configured [...]

          SonarQube has some pretty strong opinions on what is pass / fail. What we seek from the Jenkins sonar plugin is to tie these ideas into Jenkins own concepts. Sonarqube pass is pretty clear - pass sonar, pass the build.

          Sonarqube failure is not, as it is mapped to Jenkins failure today. There are three main use-cases where customizing this mapping is valuable:

          1) In this example above we want to distinguish between a quality failure and a CI infrastructure failure. We run Jenkins for hundreds of developers and thousands of builds a day, so being able to categorize these types of pipeline failures helps us understand where problems are and where Jenkins is doing its job of preventing quality leaks, like the Vulnerabilities you mention.

          2) This also allows us to hook into Jenkins post hook - if you are unstable you can do things like notify the correct people on what the exactly problem is: https://www.jenkins.io/doc/pipeline/tour/post/

          3) Another use-case is that during a commit build or a deploy to UAT, a SonarQube "failure" is not to be considered enough to blocking the pipeline and we actually desire it to continue to the next step, as it's still valuable to continue to the next steps.

          For #1 and #2 we still intend to stop the build with the skipStagesAfterUnstable() option, whereas in #3 the pipeline will continue.

          All of these use-cases make use of the Jenkins concept of 'unstable', and that is the feature request here.

           

          Dan Alvizu added a comment - Hi Ian - I work with ipleten, so this is the same ask but allow me to clarify- The glossary is reflective of the Jenkins "standalone" product. It does not consider what plugins may do. The glossary directly references a plugin : For example if the JUnit publisher is configured [...] SonarQube has some pretty strong opinions on what is pass / fail. What we seek from the Jenkins sonar plugin is to tie these ideas into Jenkins own concepts. Sonarqube pass is pretty clear - pass sonar, pass the build. Sonarqube failure is not, as it is mapped to Jenkins failure today. There are three main use-cases where customizing this mapping is valuable: 1) In this example above we want to distinguish between a quality failure and a CI infrastructure failure. We run Jenkins for hundreds of developers and thousands of builds a day, so being able to categorize these types of pipeline failures helps us understand where problems are and where Jenkins is doing its job of preventing quality leaks, like the Vulnerabilities you mention. 2) This also allows us to hook into Jenkins post hook - if you are unstable you can do things like notify the correct people on what the exactly problem is: https://www.jenkins.io/doc/pipeline/tour/post/ 3) Another use-case is that during a commit build or a deploy to UAT, a SonarQube "failure" is not to be considered enough to blocking the pipeline and we actually desire it to continue to the next step, as it's still valuable to continue to the next steps. For #1 and #2 we still intend to stop the build with the skipStagesAfterUnstable() option, whereas in #3 the pipeline will continue. All of these use-cases make use of the Jenkins concept of 'unstable', and that is the feature request here.  

          Ian Williams added a comment - - edited

          I get what you are saying .. I just did not understand how if the QualtiyGate returns only a PASS/FAIL that you can interpret that to be UNSTABLE under certain circumstances. I do not want to see the options be PASS/UNSTABLE as that would have major repercussions.

          They offer the option:

          // Parameter indicates whether to set pipeline to UNSTABLE if Quality Gate fails
          // true = set pipeline to UNSTABLE, false = don't
          waitForQualityGate abortPipeline: true

          I am simply suggesting you have an alternative to that, which is to have different Quality Gates configured in SonarQube.

          As noted in the SonarSource tracker, MMF-473 Simplify the definition of Quality Gate, the simplification (/dumbing down) of the Quality Gate has been on the roadmap for 5 years. I did not even know there used to a WARNING state, which could have mapped relatively cleanly to UNSTABLE request.

          I have no relation to the sonar plugin, other than as a user of it. We have well over 2000 projects under analysis across multiple Orgs w/1000's of Dev, we are now a user of SonarQube Enterprise. I agree SonarSource is rather/very opinionated, both in terms of pass/fail and in assumptions on how development happens and how you can use their tools, which don't necessarily integrate well with non-greenfield operations. I don't see them changing course (Immovable object).

          As for Jenkins glossary, I would not rely on that as definitive gospel. The explanation even says, "for example".

          If the actions of another plugin (sonar) don't align with the example given, that is a design decision the makers of the plugin have implemented (with an escape hatch); they decided the QualtiyGate FAIL should generally FAIL the build, and that makes sense. You have the ability to alter the FAIL criteria in SonarQube, otherwise if a user looked in SQ, they'd see it FAILS but the pipeline might continue with the failed artifact. That is is risky.

          Jenkins (Hudson) originated primarily as as a tool to build / CI Maven applications. JUnit is historically a universal step in the testing of maven projects and is used throughout the docs as an example (eg: here , here, here).

          The JUnit plugin is a publisher (post-build) step, while SonarScanner is a build step for analysis with a QualityGate step, which is also a build step.

          That's just my 2 cents. Thanks.

          Ian Williams added a comment - - edited I get what you are saying .. I just did not understand how if the QualtiyGate returns only a PASS/FAIL that you can interpret that to be UNSTABLE under certain circumstances. I do not want to see the options be PASS/UNSTABLE as that would have major repercussions. They offer the option : // Parameter indicates whether to set pipeline to UNSTABLE if Quality Gate fails // true = set pipeline to UNSTABLE, false = don't waitForQualityGate abortPipeline: true I am simply suggesting you have an alternative to that, which is to have different Quality Gates configured in SonarQube. As noted in the SonarSource tracker, MMF-473 Simplify the definition of Quality Gate , the simplification (/dumbing down) of the Quality Gate has been on the roadmap for 5 years. I did not even know there used to a WARNING state , which could have mapped relatively cleanly to UNSTABLE request. I have no relation to the sonar plugin, other than as a user of it. We have well over 2000 projects under analysis across multiple Orgs w/1000's of Dev, we are now a user of SonarQube Enterprise. I agree SonarSource is rather/very opinionated, both in terms of pass/fail and in assumptions on how development happens and how you can use their tools, which don't necessarily integrate well with non-greenfield operations. I don't see them changing course (Immovable object). As for Jenkins glossary, I would not rely on that as definitive gospel. The explanation even says, "for example". If the actions of another plugin (sonar) don't align with the example given, that is a design decision the makers of the plugin have implemented (with an escape hatch); they decided the QualtiyGate FAIL should generally FAIL the build, and that makes sense. You have the ability to alter the FAIL criteria in SonarQube, otherwise if a user looked in SQ, they'd see it FAILS but the pipeline might continue with the failed artifact. That is is risky. Jenkins (Hudson) originated primarily as as a tool to build / CI Maven applications. JUnit is historically a universal step in the testing of maven projects and is used throughout the docs as an example (eg: here , here , here ). The JUnit plugin is a publisher (post-build) step, while SonarScanner is a build step for analysis with a QualityGate step, which is also a build step. That's just my 2 cents. Thanks.

            sonarteam Sonar Team
            tamerlaha ipleten
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: