Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-24076

Incorrect builds used to generate deltas (No deltas for builds fixed after failure where errors were introduced)

    • Icon: New Feature New Feature
    • Resolution: Unresolved
    • Icon: Major Major
    • cppcheck-plugin
    • None

      As an example, consider the following:

      Build 1 - Stable (within CPPCheck threshold)
      Build 2 - Unstable (exceeds CPPCheck threshold)
      Build 3 - Unstable (no change)
      Build 4 - Unstable (no change)

      If you view the CPPCheck results for build 4, it shows the resolved issues since the dawn of time, which is arguably incorrect (see attached image for an example of solved errors showing when the delta is 0), but more crucially, generates its delta for new errors against the previous build, regardless of the build status.

      This creates the problem that, once build 2 drops off the history, you lose the deltas that help you identify the new errors that were introduced, because the deltas don't exist for build 3 or build 4 (they are zero).

      There are one of two solutions I can think of:

      1. Allow the previous build status for delta generation to be specified in the configuration: lastStableBuild, lastSuccessfulBuild

      2. Hard code it to use the lastStableBuild only.

          [JENKINS-24076] Incorrect builds used to generate deltas (No deltas for builds fixed after failure where errors were introduced)

          Craig Phillips created issue -

          Shows the solved issues against a build with a delta of zero.

          Craig Phillips added a comment - Shows the solved issues against a build with a delta of zero.
          Craig Phillips made changes -
          Attachment New: cppcheck-delta-incorrect.png [ 26475 ]

          Having used this feature for a while now, it simply looks like solved and new errors are being reported inverse to what they should be.

          The 'new' errors are listed in the view since the lastBuild tag. If there have been no new errors introduces since the last build, then there are no new errors displayed. But this is not logical, since the builds continue to be marked unstable because the error count exceeds the threshold, due to the errors introduced by the previous build.

          The 'solved' errors are listed since the lastStableBuild tag. If errors have been solved between the current build and the last build, which might be unstable, then you can't see them, because the solved errors includes all errors solved since the last stable build.

          What this ultimately means is that it is impossible to identify and fix new errors introduced between the last stable and the first unstable build, if the build history no longer exists. But you can see what errors have been resolved since the last stable build, which isn't much use. I'm not sure that I care what errors have been solved since the last stable build. What I want to see are the errors that have been introduced since the last stable build and what errors have been solved since the last build, so I know what errors I have left to resolve in order to restore the build to its original state.

          Craig Phillips added a comment - Having used this feature for a while now, it simply looks like solved and new errors are being reported inverse to what they should be. The 'new' errors are listed in the view since the lastBuild tag. If there have been no new errors introduces since the last build, then there are no new errors displayed. But this is not logical, since the builds continue to be marked unstable because the error count exceeds the threshold, due to the errors introduced by the previous build. The 'solved' errors are listed since the lastStableBuild tag. If errors have been solved between the current build and the last build, which might be unstable, then you can't see them, because the solved errors includes all errors solved since the last stable build. What this ultimately means is that it is impossible to identify and fix new errors introduced between the last stable and the first unstable build, if the build history no longer exists. But you can see what errors have been resolved since the last stable build, which isn't much use. I'm not sure that I care what errors have been solved since the last stable build. What I want to see are the errors that have been introduced since the last stable build and what errors have been solved since the last build, so I know what errors I have left to resolve in order to restore the build to its original state.

          I have a build at the moment with 1 stable build and 9 subsequent unstable builds because the errors haven't yet been resolved. If another build is triggered, the build history for the first unstable build will be lost and we lose the information about the new errors introduced by that build. The delta for the other 8 builds is 0, so there is no information about new errors. The only way to then resolve the "new" errors is to download the cppcheck.xml file from the stable build we have, download the second copy from the last build we have and diff them. This then shows us the new errors introduced since the last stable build, so that we can fix them and move on.

          Craig Phillips added a comment - I have a build at the moment with 1 stable build and 9 subsequent unstable builds because the errors haven't yet been resolved. If another build is triggered, the build history for the first unstable build will be lost and we lose the information about the new errors introduced by that build. The delta for the other 8 builds is 0, so there is no information about new errors. The only way to then resolve the "new" errors is to download the cppcheck.xml file from the stable build we have, download the second copy from the last build we have and diff them. This then shows us the new errors introduced since the last stable build, so that we can fix them and move on.

          The problem is worse if you have a failed build immediately preceding an unstable build, if the errors were introduced by the failed build. The delta is then zero, and the only way to identify the "new" errors, is to delete the build history back to the last stable build and run another build to regenerate the deltas. This is getting a bit silly now.

          Craig Phillips added a comment - The problem is worse if you have a failed build immediately preceding an unstable build, if the errors were introduced by the failed build. The delta is then zero, and the only way to identify the "new" errors, is to delete the build history back to the last stable build and run another build to regenerate the deltas. This is getting a bit silly now.
          Craig Phillips made changes -
          Attachment New: cppcheck2.png [ 26529 ]
          Craig Phillips made changes -
          Priority Original: Major [ 3 ] New: Critical [ 2 ]
          Summary Original: Incorrect builds used to generate deltas New: Incorrect builds used to generate deltas (No deltas for builds fixed after failure where errors were introduced)

          There hasn't been any movement on this issue and we are desperate for a fix. Any chance you could take a look. Thanks.

          Craig Phillips added a comment - There hasn't been any movement on this issue and we are desperate for a fix. Any chance you could take a look. Thanks.
          Craig Phillips made changes -
          Assignee Original: Gregory Boissinot [ gbois ] New: Michal Turek [ mixalturek ]

            iwonbigbro Craig Phillips
            iwonbigbro Craig Phillips
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: