The fix of JENKINS-15652 merely downgrades the error but does not clean up build records. It would make sense to have the code detect anomalous conditions and initiate a builds folder cleanup. Take only the last build of a given number, and discard any earlier with higher number. I.e. just set <number> to 0 or something and ignore such builds, or rename build.xml to build-broken.xml.

          [JENKINS-18289] Detect & repair out-of-order build records

          Jesse Glick added a comment -

          A script like the following can be useful for diagnosis:

          CANONICAL_ID_FORMATTER = new java.text.SimpleDateFormat("yyyy-MM-dd_HH-mm-ss");
          NUMBER_XPATH = "/node()/number";
          xpath = javax.xml.xpath.XPathFactory.newInstance().newXPath();
          for (j in Jenkins.instance.getAllItems(AbstractProject)) {
            d = new File(j.rootDir, 'builds');
            m = new TreeMap();
            for (b in d.listFiles()) {
              xml = new File(b, 'build.xml');
              if (!xml.file) {
                if (!b.getName().startsWith('last')) {
                  println("not a build dir: ${b}");
                }
                continue;
              }
              try {
                CANONICAL_ID_FORMATTER.parse(b.name);
                xpath.reset();
                number = xpath.evaluate(NUMBER_XPATH, new org.xml.sax.InputSource(new java.io.FileReader(xml)), javax.xml.xpath.XPathConstants.NODE);
                if (number) {
                  m.put(b.name, number.textContent);
                } else {
                  println("cannot parse ${xml}");
                }
              } catch (java.text.ParseException e) {
                if (!Util.isSymlink(b)) {
                  println("${b} should be a symlink but is not");
                }
              }
            }
            original = m.values().collect({Integer.parseInt(it)});
            if (!original.equals(original.sort(false))) {
              println("out of order in ${d}: ${m}");
            }
          }
          

          Jesse Glick added a comment - A script like the following can be useful for diagnosis: CANONICAL_ID_FORMATTER = new java.text.SimpleDateFormat( "yyyy-MM-dd_HH-mm-ss" ); NUMBER_XPATH = "/node()/number" ; xpath = javax.xml.xpath.XPathFactory.newInstance().newXPath(); for (j in Jenkins.instance.getAllItems(AbstractProject)) { d = new File(j.rootDir, 'builds' ); m = new TreeMap(); for (b in d.listFiles()) { xml = new File(b, 'build.xml' ); if (!xml.file) { if (!b.getName().startsWith( 'last' )) { println( "not a build dir: ${b}" ); } continue ; } try { CANONICAL_ID_FORMATTER.parse(b.name); xpath.reset(); number = xpath.evaluate(NUMBER_XPATH, new org.xml.sax.InputSource( new java.io.FileReader(xml)), javax.xml.xpath.XPathConstants.NODE); if (number) { m.put(b.name, number.textContent); } else { println( "cannot parse ${xml}" ); } } catch (java.text.ParseException e) { if (!Util.isSymlink(b)) { println( "${b} should be a symlink but is not" ); } } } original = m.values().collect({ Integer .parseInt(it)}); if (!original.equals(original.sort( false ))) { println( "out of order in ${d}: ${m}" ); } }

          Daniel Beck added a comment -

          Jenkins should not destroy build data without administrator confirmation. Something like Manage Old Data for build records would be best.

          Daniel Beck added a comment - Jenkins should not destroy build data without administrator confirmation. Something like Manage Old Data for build records would be best.

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          changelog.html
          core/src/main/java/jenkins/diagnostics/ooom/BuildPtr.java
          core/src/main/java/jenkins/diagnostics/ooom/OutOfOrderBuildDetector.java
          core/src/main/java/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor.java
          core/src/main/java/jenkins/diagnostics/ooom/Problem.java
          core/src/main/java/jenkins/management/AsynchronousAdministrativeMonitor.java
          core/src/main/java/jenkins/model/lazy/AbstractLazyLoadRunMap.java
          core/src/main/java/jenkins/security/RekeySecretAdminMonitor.java
          core/src/main/resources/jenkins/diagnostics/ooom/Messages.properties
          core/src/main/resources/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor/message.groovy
          core/src/main/resources/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor/message.properties
          core/src/main/resources/jenkins/management/AsynchronousAdministrativeMonitor/log.groovy
          core/src/main/resources/jenkins/security/Messages.properties
          core/src/main/resources/jenkins/security/Messages_zh_TW.properties
          core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/log.groovy
          core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/log_zh_TW.properties
          core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/message.groovy
          test/src/test/groovy/jenkins/diagnostics/ooom/OutOfOrderBuildDetectorTest.groovy
          http://jenkins-ci.org/commit/jenkins/d90d47d1b76ebb5d753e919afefb5d2457664da2
          Log:
          [FIXED JENKINS-18289]

          The code is smart enough to compute a minimum set of builds to remove to
          restore order consistency.

          Compare: https://github.com/jenkinsci/jenkins/compare/0c6ab8491bff...d90d47d1b76e

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/src/main/java/jenkins/diagnostics/ooom/BuildPtr.java core/src/main/java/jenkins/diagnostics/ooom/OutOfOrderBuildDetector.java core/src/main/java/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor.java core/src/main/java/jenkins/diagnostics/ooom/Problem.java core/src/main/java/jenkins/management/AsynchronousAdministrativeMonitor.java core/src/main/java/jenkins/model/lazy/AbstractLazyLoadRunMap.java core/src/main/java/jenkins/security/RekeySecretAdminMonitor.java core/src/main/resources/jenkins/diagnostics/ooom/Messages.properties core/src/main/resources/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor/message.groovy core/src/main/resources/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor/message.properties core/src/main/resources/jenkins/management/AsynchronousAdministrativeMonitor/log.groovy core/src/main/resources/jenkins/security/Messages.properties core/src/main/resources/jenkins/security/Messages_zh_TW.properties core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/log.groovy core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/log_zh_TW.properties core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/message.groovy test/src/test/groovy/jenkins/diagnostics/ooom/OutOfOrderBuildDetectorTest.groovy http://jenkins-ci.org/commit/jenkins/d90d47d1b76ebb5d753e919afefb5d2457664da2 Log: [FIXED JENKINS-18289] The code is smart enough to compute a minimum set of builds to remove to restore order consistency. Compare: https://github.com/jenkinsci/jenkins/compare/0c6ab8491bff...d90d47d1b76e

          The code moves the offending build records into another directory.

          Kohsuke Kawaguchi added a comment - The code moves the offending build records into another directory.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #2766
          [FIXED JENKINS-18289] (Revision d90d47d1b76ebb5d753e919afefb5d2457664da2)

          Result = UNSTABLE
          kohsuke : d90d47d1b76ebb5d753e919afefb5d2457664da2
          Files :

          • core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/log_zh_TW.properties
          • core/src/main/resources/jenkins/security/Messages_zh_TW.properties
          • core/src/main/java/jenkins/diagnostics/ooom/Problem.java
          • core/src/main/java/jenkins/diagnostics/ooom/BuildPtr.java
          • core/src/main/resources/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor/message.groovy
          • test/src/test/groovy/jenkins/diagnostics/ooom/OutOfOrderBuildDetectorTest.groovy
          • core/src/main/resources/jenkins/diagnostics/ooom/Messages.properties
          • core/src/main/resources/jenkins/security/Messages.properties
          • core/src/main/java/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor.java
          • core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/log.groovy
          • changelog.html
          • core/src/main/resources/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor/message.properties
          • core/src/main/resources/jenkins/management/AsynchronousAdministrativeMonitor/log.groovy
          • core/src/main/java/jenkins/model/lazy/AbstractLazyLoadRunMap.java
          • core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/message.groovy
          • core/src/main/java/jenkins/management/AsynchronousAdministrativeMonitor.java
          • core/src/main/java/jenkins/security/RekeySecretAdminMonitor.java
          • core/src/main/java/jenkins/diagnostics/ooom/OutOfOrderBuildDetector.java

          dogfood added a comment - Integrated in jenkins_main_trunk #2766 [FIXED JENKINS-18289] (Revision d90d47d1b76ebb5d753e919afefb5d2457664da2) Result = UNSTABLE kohsuke : d90d47d1b76ebb5d753e919afefb5d2457664da2 Files : core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/log_zh_TW.properties core/src/main/resources/jenkins/security/Messages_zh_TW.properties core/src/main/java/jenkins/diagnostics/ooom/Problem.java core/src/main/java/jenkins/diagnostics/ooom/BuildPtr.java core/src/main/resources/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor/message.groovy test/src/test/groovy/jenkins/diagnostics/ooom/OutOfOrderBuildDetectorTest.groovy core/src/main/resources/jenkins/diagnostics/ooom/Messages.properties core/src/main/resources/jenkins/security/Messages.properties core/src/main/java/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor.java core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/log.groovy changelog.html core/src/main/resources/jenkins/diagnostics/ooom/OutOfOrderBuildMonitor/message.properties core/src/main/resources/jenkins/management/AsynchronousAdministrativeMonitor/log.groovy core/src/main/java/jenkins/model/lazy/AbstractLazyLoadRunMap.java core/src/main/resources/jenkins/security/RekeySecretAdminMonitor/message.groovy core/src/main/java/jenkins/management/AsynchronousAdministrativeMonitor.java core/src/main/java/jenkins/security/RekeySecretAdminMonitor.java core/src/main/java/jenkins/diagnostics/ooom/OutOfOrderBuildDetector.java

            kohsuke Kohsuke Kawaguchi
            jglick Jesse Glick
            Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: