Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21099

Don't give useless build time estimates by considering failed builds' durations

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • core
    • None

      We use the estimated build time mostly to determine 'by when will we have a result?'. There is some (±5-10%, the longer the duration the less variation) variation between durations of successful builds, but it's still good enough for an estimate (disregarding jobs where there are significant differences between builds due to e.g. parameters).

      Unfortunately, this commit changed computation of build duration estimates:
      https://github.com/jenkinsci/jenkins/commit/04d85eb476bdc57eb7ac3c2bb34e91be5b55c487

      Before: Use the average of the last up to three successful builds. Use fewer if no three successful builds exist. If there are no successful builds, don't attempt to provide an estimate.

      Now: Only look at the last six builds. Use the average of the last up to three successful builds of those. If there are fewer than three successful builds, also use the latest failed (but not aborted) builds to get three builds to base the estimate on. If three or fewer completed builds exist, use those.

      So, if there's only one successful build among the last six builds, it will also consider the latest two failed builds for the estimate. If there are no successful builds, use the build duration until previous build failures as estimate.

      Not going far back in the job history is a good idea to ensure computation is quick. Filling up the list of candidate builds with failed builds OTOH, is not.


      Taking into account failing builds makes the estimate completely meaningless. It doesn't tell you anything, because the data it uses is all over the place (and it uses too few data points for more sophisticated estimates).

      For a job that often fails after ~2-3 hours but takes 10 hours to complete successfully, this will often reduce the estimate to just ~5 hours. Even just considering one failed and two successful builds will result in an estimate of 7.5, which is way off the actual duration to get a successful build.


      Build failures should be considered exceptional conditions. For non-fatal issues, builds can be marked unstable. So failures should not be used in estimating build durations, as some unreliable component used in a build will completely distort build durations.

      Surprisingly, the change only considers completed builds and excludes aborted builds from the estimate. This respects the status of aborted builds as exceptional and unfit for build estimates (unless caused by the Build Timeout Plugin…). But failed builds are no more useful to build an estimate on!

      I'd rather have it use the average of the last N (e.g. 3 or 5) successful builds, and if there are fewer than M (e.g. 1, or 3), just give no estimate at all. For jobs that also have wildly varying durations for successful builds, it might actually be useful to not provide an estimated duration – this would make it explicit that Jenkins has no confidence in providing an estimate.

          [JENKINS-21099] Don't give useless build time estimates by considering failed builds' durations

          Daniel Beck created issue -
          Daniel Beck made changes -
          Description Original: Disregarding jobs where there are significant differences (e.g. parameters causing every other build to do twice the work) between builds, we use the estimated build time mostly to determine '_by when do we have a result?_'. There is some (±5-10%, the longer the duration the less variation) variation between durations of successful builds, but it's still good enough for an estimate based on how the UI works.

          Unfortunately, this commit changed computation of build duration estimates:
          https://github.com/jenkinsci/jenkins/commit/04d85eb476bdc57eb7ac3c2bb34e91be5b55c487

          *Before:* Use the average of the last up to three successful builds. Use fewer if no three successful builds exist. If there are no successful builds, don't attempt to provide an estimate.

          *Now:* Only look at the last six builds. Use the average of the last up to three successful builds of those. If there are fewer than three successful builds, also use the latest failed (but not aborted) builds to get three builds to base the estimate on. If three or fewer completed builds exist, use those.

          So, if there's only one successful build among the last six builds, it will also consider the latest two failed builds for the estimate. *If there are no successful builds, use the build duration until previous build failures as estimate*.

          ----

          *Taking into account failing builds makes the estimate completely meaningless.* It doesn't tell you anything, because the data it uses is all over the place (and it uses too few data points for more sophisticated estimates).

          For a job that often fails after ~2-3 hours but takes 10 hours to complete successfully, this will often reduce the estimate to just ~5 hours. Even just considering one failed and two successful builds will result in an estimate of 7.5, which is way off the actual duration to get a successful build.

          ----

          *Build failures should be considered exceptional conditions.* For non-fatal issues, builds can be marked unstable. *So failures should not be used in estimating build durations,* as some unreliable component used in a build will completely distort build durations.

          I'd rather have it use the average of the last N (e.g. 3 or 5) _successful_ builds, and if there are fewer than M (e.g. 1, or 3), *just give no estimate at all.* For jobs that also have wildly varying durations for successful builds, it might actually be useful to not provide an estimated duration -- this would make it *explicit that Jenkins has no confidence in providing an estimate.*
          New: Disregarding jobs where there are significant differences (e.g. parameters causing every other build to do twice the work) between builds, we use the estimated build time mostly to determine '_by when do we have a result?_'. There is some (±5-10%, the longer the duration the less variation) variation between durations of successful builds, but it's still good enough for an estimate based on how the UI works.

          Unfortunately, this commit changed computation of build duration estimates:
          https://github.com/jenkinsci/jenkins/commit/04d85eb476bdc57eb7ac3c2bb34e91be5b55c487

          *Before:* Use the average of the last up to three successful builds. Use fewer if no three successful builds exist. If there are no successful builds, don't attempt to provide an estimate.

          *Now:* Only look at the last six builds. Use the average of the last up to three successful builds of those. If there are fewer than three successful builds, also use the latest failed (but not aborted) builds to get three builds to base the estimate on. If three or fewer completed builds exist, use those.

          So, if there's only one successful build among the last six builds, it will also consider the latest two failed builds for the estimate. *If there are no successful builds, use the build duration until previous build failures as estimate*.

          Not going far back in the job history is a good idea to ensure computation is quick. Filling up the list of candidate builds with failed builds OTOH, is not.

          ----

          *Taking into account failing builds makes the estimate completely meaningless.* It doesn't tell you anything, because the data it uses is all over the place (and it uses too few data points for more sophisticated estimates).

          For a job that often fails after ~2-3 hours but takes 10 hours to complete successfully, this will often reduce the estimate to just ~5 hours. Even just considering one failed and two successful builds will result in an estimate of 7.5, which is way off the actual duration to get a successful build.

          ----

          *Build failures should be considered exceptional conditions.* For non-fatal issues, builds can be marked unstable. *So failures should not be used in estimating build durations,* as some unreliable component used in a build will completely distort build durations.

          Surprisingly, the change only considers _completed_ builds and excludes _aborted_ builds from the estimate. This respects the status of aborted builds as exceptional and unfit for build estimates (unless caused by the Build Timeout Plugin…). But failed builds are no more useful to build an estimate on!

          I'd rather have it use the average of the last N (e.g. 3 or 5) _successful_ builds, and if there are fewer than M (e.g. 1, or 3), *just give no estimate at all.* For jobs that also have wildly varying durations for successful builds, it might actually be useful to not provide an estimated duration -- this would make it *explicit that Jenkins has no confidence in providing an estimate.*
          Summary Original: Don't give useless build time estimates New: Don't give useless build time estimates by considering failed builds' durations
          Daniel Beck made changes -
          Description Original: Disregarding jobs where there are significant differences (e.g. parameters causing every other build to do twice the work) between builds, we use the estimated build time mostly to determine '_by when do we have a result?_'. There is some (±5-10%, the longer the duration the less variation) variation between durations of successful builds, but it's still good enough for an estimate based on how the UI works.

          Unfortunately, this commit changed computation of build duration estimates:
          https://github.com/jenkinsci/jenkins/commit/04d85eb476bdc57eb7ac3c2bb34e91be5b55c487

          *Before:* Use the average of the last up to three successful builds. Use fewer if no three successful builds exist. If there are no successful builds, don't attempt to provide an estimate.

          *Now:* Only look at the last six builds. Use the average of the last up to three successful builds of those. If there are fewer than three successful builds, also use the latest failed (but not aborted) builds to get three builds to base the estimate on. If three or fewer completed builds exist, use those.

          So, if there's only one successful build among the last six builds, it will also consider the latest two failed builds for the estimate. *If there are no successful builds, use the build duration until previous build failures as estimate*.

          Not going far back in the job history is a good idea to ensure computation is quick. Filling up the list of candidate builds with failed builds OTOH, is not.

          ----

          *Taking into account failing builds makes the estimate completely meaningless.* It doesn't tell you anything, because the data it uses is all over the place (and it uses too few data points for more sophisticated estimates).

          For a job that often fails after ~2-3 hours but takes 10 hours to complete successfully, this will often reduce the estimate to just ~5 hours. Even just considering one failed and two successful builds will result in an estimate of 7.5, which is way off the actual duration to get a successful build.

          ----

          *Build failures should be considered exceptional conditions.* For non-fatal issues, builds can be marked unstable. *So failures should not be used in estimating build durations,* as some unreliable component used in a build will completely distort build durations.

          Surprisingly, the change only considers _completed_ builds and excludes _aborted_ builds from the estimate. This respects the status of aborted builds as exceptional and unfit for build estimates (unless caused by the Build Timeout Plugin…). But failed builds are no more useful to build an estimate on!

          I'd rather have it use the average of the last N (e.g. 3 or 5) _successful_ builds, and if there are fewer than M (e.g. 1, or 3), *just give no estimate at all.* For jobs that also have wildly varying durations for successful builds, it might actually be useful to not provide an estimated duration -- this would make it *explicit that Jenkins has no confidence in providing an estimate.*
          New: We use the estimated build time mostly to determine '_by when will we have a result?_'. There is some (±5-10%, the longer the duration the less variation) variation between durations of successful builds, but it's still good enough for an estimate (disregarding jobs where there are significant differences between builds due to e.g. parameters).

          Unfortunately, this commit changed computation of build duration estimates:
          https://github.com/jenkinsci/jenkins/commit/04d85eb476bdc57eb7ac3c2bb34e91be5b55c487

          *Before:* Use the average of the last up to three successful builds. Use fewer if no three successful builds exist. If there are no successful builds, don't attempt to provide an estimate.

          *Now:* Only look at the last six builds. Use the average of the last up to three successful builds of those. If there are fewer than three successful builds, also use the latest failed (but not aborted) builds to get three builds to base the estimate on. If three or fewer completed builds exist, use those.

          So, if there's only one successful build among the last six builds, it will also consider the latest two failed builds for the estimate. *If there are no successful builds, use the build duration until previous build failures as estimate*.

          Not going far back in the job history is a good idea to ensure computation is quick. Filling up the list of candidate builds with failed builds OTOH, is not.

          ----

          *Taking into account failing builds makes the estimate completely meaningless.* It doesn't tell you anything, because the data it uses is all over the place (and it uses too few data points for more sophisticated estimates).

          For a job that often fails after ~2-3 hours but takes 10 hours to complete successfully, this will often reduce the estimate to just ~5 hours. Even just considering one failed and two successful builds will result in an estimate of 7.5, which is way off the actual duration to get a successful build.

          ----

          *Build failures should be considered exceptional conditions.* For non-fatal issues, builds can be marked unstable. *So failures should not be used in estimating build durations,* as some unreliable component used in a build will completely distort build durations.

          Surprisingly, the change only considers _completed_ builds and excludes _aborted_ builds from the estimate. This respects the status of aborted builds as exceptional and unfit for build estimates (unless caused by the Build Timeout Plugin…). But failed builds are no more useful to build an estimate on!

          I'd rather have it use the average of the last N (e.g. 3 or 5) _successful_ builds, and if there are fewer than M (e.g. 1, or 3), *just give no estimate at all.* For jobs that also have wildly varying durations for successful builds, it might actually be useful to not provide an estimated duration -- this would make it *explicit that Jenkins has no confidence in providing an estimate.*
          kutzi made changes -
          Priority Original: Major [ 3 ] New: Minor [ 4 ]
          R. Tyler Croy made changes -
          Workflow Original: JNJira [ 152961 ] New: JNJira + In-Review [ 178371 ]
          Jacob Svensson made changes -
          Link New: This issue relates to JENKINS-49425 [ JENKINS-49425 ]
          Martin made changes -
          Link New: This issue is related to JENKINS-63174 [ JENKINS-63174 ]
          Martin made changes -
          Comment [ To illustrate the current behaviour, I have included a section of our current display here. There are even three successful builds in the last 6 operations. However, an average of all available builds is calculated. In this case, there are 10 that we still know.
          It does not seem to be the result of the mentioned commit.

           
          |*Deploy application*|*Run tests*|
          |*11min 32s*|*4min 31s*|
          |{color:#00875a}6min 13s{color}|{color:#00875a}5min 30s{color}|
          |{color:#00875a}9min 9s{color}|{color:#00875a}6min 2s{color}|
          |{color:#de350b}41min 51s{color}|{color:#de350b}31ms{color}|
          |{color:#de350b}failed{color}|{color:#de350b}failed{color}|
          |28min 31s|{color:#172b4d}49ms{color}|
          |aborted|aborted|
          |{color:#00875a}4min 19s{color}|{color:#00875a}5min 53s{color}|
          |{color:#00875a}5min 42s{color}|{color:#00875a}6min 6s{color}| ]
          Dmitriy made changes -
          Attachment New: jenkins_mean_time.png [ 62069 ]

            Unassigned Unassigned
            danielbeck Daniel Beck
            Votes:
            20 Vote for this issue
            Watchers:
            19 Start watching this issue

              Created:
              Updated: