Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61130

"ignoring quality gate result for one build" feature seems not to work

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • warnings-ng-plugin
    • None

      Hello,

      I am using the plugin version 7.2.2 and Jenkins (LTS v2.190.3) reports
      "Resetting reference build, ignoring quality gate result for one build"
      after pressing the "Reset quality gate" button of the previous build, but it still prints
      "-> Some quality gates have been missed: overall result is FAILED".

      I have implemented at first my Jenkins pipeline without referenceJobName and then I set it to a fixed branch name "master", but it did not change anything.

      Log example:

      00:03:12.892  [Static Analysis] Attaching ResultAction with ID 'analysis' to run 'XYZ/aaaaaa/master #7'.
      00:03:12.917  [Static Analysis] Resetting reference build, ignoring quality gate result for one build
      00:03:12.917  [Static Analysis] Using reference build 'XYZ/aaaaaa/master #2' to compute new, fixed, and outstanding issues
      00:03:12.917  [Static Analysis] Issues delta (vs. reference build): outstanding: 61, new: 52, fixed: 1
      00:03:12.917  [Static Analysis] Evaluating quality gates
      00:03:12.917  [Static Analysis] -> PASSED - New (errors only): 0 - Quality QualityGate: 1
      00:03:12.917  [Static Analysis] -> PASSED - New (severity high only): 0 - Quality QualityGate: 1
      00:03:12.917  [Static Analysis] -> FAILED - New (severity normal only): 51 - Quality QualityGate: 4
      00:03:12.917  [Static Analysis] -> PASSED - New (severity low only): 1 - Quality QualityGate: 10
      00:03:12.917  [Static Analysis] -> Some quality gates have been missed: overall result is FAILED
      00:03:12.917  [Static Analysis] Enabling health report (Healthy=1, Unhealthy=12, Minimum Severity=LOW)
      00:03:12.965  [Static Analysis] Created analysis result for 113 issues (found 52 new issues, fixed 1 issues)
      

      With build no. 3 I started pressing the "Reset quality gate" button. But it has no effect.

      Any ideas or suggestions?

      Thank you.

          [JENKINS-61130] "ignoring quality gate result for one build" feature seems not to work

          Ulli Hafner added a comment -

          The button changes only the selection of the reference in the following way: as reference any previous build will be chosen that is not failing (regardless of the quality gate). So I am assuming your builds #3-#6 are failing?

          Or let me ask the other way around: what do you want to achieve?

          Ulli Hafner added a comment - The button changes only the selection of the reference in the following way: as reference any previous build will be chosen that is not failing (regardless of the quality gate). So I am assuming your builds #3-#6 are failing? Or let me ask the other way around: what do you want to achieve?

          R. Fitzner added a comment - - edited

          With build #2 the repository suppressed JavaDoc warnings/errors via Gradle options. With #3 I disabled suppressing it and off course now the "new" issues increased dramatically.
          See

          How can I make build #3 as a new baseline?

          PS. Yes build #3-#7 are failing because the quality gates insists that the "overall result is FAILED".

          R. Fitzner added a comment - - edited With build #2 the repository suppressed JavaDoc warnings/errors via Gradle options. With #3 I disabled suppressing it and off course now the "new" issues increased dramatically. See How can I make build #3 as a new baseline? PS. Yes build #3-#7 are failing because the quality gates insists that the "overall result is FAILED".

          R. Fitzner added a comment - - edited

          The official Pipeline Syntax documentation is:

          1. Ignore Quality Gate (Default for ignoreQualityGate: false):
          If checked, then the last finished build of the reference job will be used as baseline no matter if the quality gate of that build has been passed or not.

          " ... Jenkins will mark all builds as unstable until the issues have been resolved."

          2. Ignore Failed Builds (Default for ignoreFailedBuilds: true):
          If checked, then only successful or unstable builds will be considered as baseline.

           

          I have noticed that:

          1. It seems like that against the documentation "unstable builds" are treated as failed!
          Even if QualityGate is configured with unstable: true neither the "Reset quality gate" button is displayed nor after re-running the job the previous UNSTABLE build is concidered as new baseline.

          00:02:39.415  [Static Analysis] Attaching ResultAction with ID 'analysis' to run 'XYZ/aaaaaa/develop #12'.
          00:02:39.436  [Static Analysis] Using reference build 'XYZ/aaaaaa/develop #10' to compute new, fixed, and outstanding issues
          00:02:39.436  [Static Analysis] Issues delta (vs. reference build): outstanding: 61, new: 52, fixed: 1
          00:02:39.436  [Static Analysis] Evaluating quality gates
          00:02:39.436  [Static Analysis] -> PASSED - New (errors only): 0 - Quality QualityGate: 1
          00:02:39.436  [Static Analysis] -> PASSED - New (severity high only): 0 - Quality QualityGate: 1
          00:02:39.436  [Static Analysis] -> WARNING - New (severity normal only): 51 - Quality QualityGate: 4
          00:02:39.436  [Static Analysis] -> PASSED - New (severity low only): 1 - Quality QualityGate: 10
          00:02:39.436  [Static Analysis] -> Some quality gates have been missed: overall result is WARNING
          00:02:39.436  [Static Analysis] Enabling health report (Healthy=1, Unhealthy=12, Minimum Severity=LOW)
          00:02:39.483  [Static Analysis] Created analysis result for 113 issues (found 52 new issues, fixed 1 issues)
          

          This explains why adding "ignoreQualityGate: true" to my pipeline did not solve my problem because the parameter ignoreFailedBuilds was still true (default).

          2. Since "unstable" and "failed" builds are treated equally the plugin now would force me to set "ignoreFailedBuilds: false".
          If I would do this then defining a QualityGate does not realy make sense anymore because the baseline is always the previous build of the last one.
          Even if I would define a QualityGate and set it to unstable I cannot differenciate between UNSTABLE (publish SNAPSHOTS) and FAILED (abort and do not publish RELEASE) anymore because the plugin checks the build status and not its quality gate status.

          Or am I missing something?

          R. Fitzner added a comment - - edited The official Pipeline Syntax documentation is: 1. Ignore Quality Gate (Default for ignoreQualityGate: false): If checked, then the last finished build of the reference job will be used as baseline no matter if the quality gate of that build has been passed or not. " ... Jenkins will mark all builds as unstable until the issues have been resolved." 2. Ignore Failed Builds (Default for ignoreFailedBuilds: true): If checked, then only successful or unstable builds will be considered as baseline.   I have noticed that: 1. It seems like that against the documentation "unstable builds" are treated as failed! Even if QualityGate is configured with unstable: true neither the "Reset quality gate" button is displayed nor after re-running the job the previous UNSTABLE build is concidered as new baseline. 00:02:39.415 [Static Analysis] Attaching ResultAction with ID 'analysis' to run 'XYZ/aaaaaa/develop #12'. 00:02:39.436 [Static Analysis] Using reference build 'XYZ/aaaaaa/develop #10' to compute new, fixed, and outstanding issues 00:02:39.436 [Static Analysis] Issues delta (vs. reference build): outstanding: 61, new: 52, fixed: 1 00:02:39.436 [Static Analysis] Evaluating quality gates 00:02:39.436 [Static Analysis] -> PASSED - New (errors only): 0 - Quality QualityGate: 1 00:02:39.436 [Static Analysis] -> PASSED - New (severity high only): 0 - Quality QualityGate: 1 00:02:39.436 [Static Analysis] -> WARNING - New (severity normal only): 51 - Quality QualityGate: 4 00:02:39.436 [Static Analysis] -> PASSED - New (severity low only): 1 - Quality QualityGate: 10 00:02:39.436 [Static Analysis] -> Some quality gates have been missed: overall result is WARNING 00:02:39.436 [Static Analysis] Enabling health report (Healthy=1, Unhealthy=12, Minimum Severity=LOW) 00:02:39.483 [Static Analysis] Created analysis result for 113 issues (found 52 new issues, fixed 1 issues) This explains why adding "ignoreQualityGate: true" to my pipeline did not solve my problem because the parameter ignoreFailedBuilds was still true (default). 2. Since "unstable" and "failed" builds are treated equally the plugin now would force me to set "ignoreFailedBuilds: false". If I would do this then defining a QualityGate does not realy make sense anymore because the baseline is always the previous build of the last one. Even if I would define a QualityGate and set it to unstable I cannot differenciate between UNSTABLE (publish SNAPSHOTS) and FAILED (abort and do not publish RELEASE) anymore because the plugin checks the build status and not its quality gate status. Or am I missing something?

          Ulli Hafner added a comment -

          This is exactly what I mean: the reference build selection for quality gates that fail a build is not really working, see JENKINS-61140. The options currently available make only sense if you are using a quality gate that sets a build to unstable.

          Ulli Hafner added a comment - This is exactly what I mean: the reference build selection for quality gates that fail a build is not really working, see JENKINS-61140 . The options currently available make only sense if you are using a quality gate that sets a build to unstable.

          R. Fitzner added a comment - - edited

          A little correction on my side concerning my previous comment: " ... neither the "Reset quality gate" button is displayed ... " this is wrong. The button is displayed but only on the very latest build.

          R. Fitzner added a comment - - edited A little correction on my side concerning my previous comment: " ... neither the "Reset quality gate" button is displayed ... " this is wrong. The button is displayed but only on the very latest build.

          I hope I am not going off-topic with this, but it seems like a variation of the issue reported here.

          I have a repo with quality gates and the reference is "master". If I create a PR request it will fail if there are additional findings. I expected Reset quality gate to allow these additional findings and set a new reference. That does seems to be the case. When I start the build again it will again FAIL.

          The configuration looks like this:

          qualityGates: [[threshold: 1, type: 'NEW', unstable: false]]
          

          Oliver Stöneberg added a comment - I hope I am not going off-topic with this, but it seems like a variation of the issue reported here. I have a repo with quality gates and the reference is "master". If I create a PR request it will fail if there are additional findings. I expected Reset quality gate to allow these additional findings and set a new reference. That does seems to be the case. When I start the build again it will again FAIL. The configuration looks like this: qualityGates: [[threshold: 1, type: 'NEW', unstable: false]]

          Ulli Hafner added a comment -

          If you specify another branch as reference, then the reference build is always from that specified branch. In this case the option has no effect at all. Would you expect that the reference build is then chosen from the PR branch?

          Ulli Hafner added a comment - If you specify another branch as reference, then the reference build is always from that specified branch. In this case the option has no effect at all. Would you expect that the reference build is then chosen from the PR branch?

          As the button is shown I would expect it to do something useful. If it will never have any effect it should not be offered at all.

          It would also mean, that quality gates cannot be used with a branching/pull request model at all. If you would use the branch as reference the first build of the branch would create that reference. But the first push on the branch might already contain additional changes compared to the code you branched from. So even if you have automatic builds on that branch the reference might be different than the master. So after merging that branch to the master it would cause a build failure if there are additional issues and you need to reset the quality gate and run the build again. It might work if you are flagging the build as UNSTABLE but in that case issues would be introduced in master builds temporarily.

          Oliver Stöneberg added a comment - As the button is shown I would expect it to do something useful. If it will never have any effect it should not be offered at all. It would also mean, that quality gates cannot be used with a branching/pull request model at all. If you would use the branch as reference the first build of the branch would create that reference. But the first push on the branch might already contain additional changes compared to the code you branched from. So even if you have automatic builds on that branch the reference might be different than the master. So after merging that branch to the master it would cause a build failure if there are additional issues and you need to reset the quality gate and run the build again. It might work if you are flagging the build as UNSTABLE but in that case issues would be introduced in master builds temporarily.

          Ulli Hafner added a comment - - edited

          As the button is shown I would expect it to do something useful. If it will never have any effect it should not be offered at all.

          Yes, when the button has been added, we simply forgot this use case...
          Can you please create a followup Jira bug, so that we can see what to do in this case?

          It would also mean, that quality gates cannot be used with a branching/pull request model at all. If you would use the branch as reference the first build of the branch would create that reference. But the first push on the branch might already contain additional changes compared to the code you branched from. So even if you have automatic builds on that branch the reference might be different than the master. So after merging that branch to the master it would cause a build failure if there are additional issues and you need to reset the quality gate and run the build again. It might work if you are flagging the build as UNSTABLE but in that case issues would be introduced in master builds temporarily.

          Selecting the right baseline build is not trivial indeed. One of my students is currently developing an algorithm for that problem (see also /JENKINS-33274) in his master thesis.

          Ulli Hafner added a comment - - edited As the button is shown I would expect it to do something useful. If it will never have any effect it should not be offered at all. Yes, when the button has been added, we simply forgot this use case... Can you please create a followup Jira bug, so that we can see what to do in this case? It would also mean, that quality gates cannot be used with a branching/pull request model at all. If you would use the branch as reference the first build of the branch would create that reference. But the first push on the branch might already contain additional changes compared to the code you branched from. So even if you have automatic builds on that branch the reference might be different than the master. So after merging that branch to the master it would cause a build failure if there are additional issues and you need to reset the quality gate and run the build again. It might work if you are flagging the build as UNSTABLE but in that case issues would be introduced in master builds temporarily. Selecting the right baseline build is not trivial indeed. One of my students is currently developing an algorithm for that problem (see also / JENKINS-33274 ) in his master thesis.

            drulli Ulli Hafner
            rf R. Fitzner
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: