Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-23130

nextBuildNumber keeps being set to previous numbers

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Major Major
    • core
    • None
    • Jenkins 1.554.1

      The nextBuildNumber seem to be getting reset to a previous number. This is especially evident on high traffic jobs that get run a lot but happens even on low traffic jobs.

      This causes Jenkins to build new jobs with the same build numbers and then cause the history to go crazy and to even lose builds (e.g. JENKINS-15156)

      Here is some output from jenkins-job-checker (a script I wrote to track these problems down):

      jobber.rb output
      $ jobber copyright-check-gerrit/
      Scanning: *
      
      **** PROBLEMS ****
      copyright-check-gerrit:
       * ORDER: The link copyright-check-gerrit/builds/10001 -> 2014-05-21_01-35-07 is out of order.
       * ORDER: The link copyright-check-gerrit/builds/10002 -> 2014-05-21_01-35-08 is out of order.
       * ORDER: The link copyright-check-gerrit/builds/10003 -> 2014-05-21_01-35-09 is out of order.
       * ORDER: The link copyright-check-gerrit/builds/10004 -> 2014-05-21_01-49-42 is out of order.
       * ORDER: The link copyright-check-gerrit/builds/10005 -> 2014-05-21_02-33-34 is out of order.
       * ORDER: The link copyright-check-gerrit/builds/10006 -> 2014-05-21_02-52-56 is out of order.
       * ORDER: The link copyright-check-gerrit/builds/10007 -> 2014-05-21_02-52-57 is out of order.
       * ORDER: The link copyright-check-gerrit/builds/10008 -> 2014-05-21_05-37-27 is out of order.
       * ORDER: The link copyright-check-gerrit/builds/10009 -> 2014-05-21_06-22-59 is out of order.
       * ORDER: The link copyright-check-gerrit/builds/10010 -> 2014-05-21_06-23-00 is out of order.
       * ORDER: The link copyright-check-gerrit/builds/10011 -> 2014-05-21_09-03-06 is out of order.
       * ORDER: The link copyright-check-gerrit/builds/10012 -> 2014-05-21_09-29-58 is out of order.
       * ORDER: The link copyright-check-gerrit/builds/10013 -> 2014-05-21_09-35-24 is out of order.
       * ORDER: The link copyright-check-gerrit/builds/10014 -> 2014-05-21_09-47-15 is out of order.
       * ORDER: The link copyright-check-gerrit/builds/10015 -> 2014-05-21_10-02-25 is out of order.
       * ORDER: The link copyright-check-gerrit/builds/10016 -> 2014-05-21_10-02-26 is out of order.
       * STOLEN: The date build copyright-check-gerrit/builds/2014-05-20_16-48-18 had its number stolen by copyright-check-gerrit/builds/10000 -> 2014-05-20_17-20-01
       * STOLEN: The date build copyright-check-gerrit/builds/2014-05-20_16-54-41 had its number stolen by copyright-check-gerrit/builds/10001 -> 2014-05-21_01-35-07
       * STOLEN: The date build copyright-check-gerrit/builds/2014-05-20_16-54-42 had its number stolen by copyright-check-gerrit/builds/10002 -> 2014-05-21_01-35-08
       * STOLEN: The date build copyright-check-gerrit/builds/2014-05-20_16-57-47 had its number stolen by copyright-check-gerrit/builds/10003 -> 2014-05-21_01-35-09
       * STOLEN: The date build copyright-check-gerrit/builds/2014-05-20_16-58-22 had its number stolen by copyright-check-gerrit/builds/10004 -> 2014-05-21_01-49-42
       * STOLEN: The date build copyright-check-gerrit/builds/2014-05-20_17-06-25 had its number stolen by copyright-check-gerrit/builds/10005 -> 2014-05-21_02-33-34
       * STOLEN: The date build copyright-check-gerrit/builds/2014-05-20_17-06-26 had its number stolen by copyright-check-gerrit/builds/10006 -> 2014-05-21_02-52-56
       * STOLEN: The date build copyright-check-gerrit/builds/2014-05-20_17-06-27 had its number stolen by copyright-check-gerrit/builds/10007 -> 2014-05-21_02-52-57
       * STOLEN: The date build copyright-check-gerrit/builds/2014-05-20_17-06-28 had its number stolen by copyright-check-gerrit/builds/10008 -> 2014-05-21_05-37-27
       * STOLEN: The date build copyright-check-gerrit/builds/2014-05-20_17-10-57 had its number stolen by copyright-check-gerrit/builds/10009 -> 2014-05-21_06-22-59
       * STOLEN: The date build copyright-check-gerrit/builds/2014-05-20_17-10-58 had its number stolen by copyright-check-gerrit/builds/10010 -> 2014-05-21_06-23-00
       * STOLEN: The date build copyright-check-gerrit/builds/2014-05-20_17-15-14 had its number stolen by copyright-check-gerrit/builds/10011 -> 2014-05-21_09-03-06
       * STOLEN: The date build copyright-check-gerrit/builds/2014-05-20_17-15-15 had its number stolen by copyright-check-gerrit/builds/10012 -> 2014-05-21_09-29-58
       * STOLEN: The date build copyright-check-gerrit/builds/2014-05-20_17-26-02 had its number stolen by copyright-check-gerrit/builds/10013 -> 2014-05-21_09-35-24
       * STOLEN: The date build copyright-check-gerrit/builds/2014-05-20_17-26-03 had its number stolen by copyright-check-gerrit/builds/10014 -> 2014-05-21_09-47-15
       * STOLEN: The date build copyright-check-gerrit/builds/2014-05-20_17-29-01 had its number stolen by copyright-check-gerrit/builds/10015 -> 2014-05-21_10-02-25
       * STOLEN: The date build copyright-check-gerrit/builds/2014-05-20_17-29-02 had its number stolen by copyright-check-gerrit/builds/10016 -> 2014-05-21_10-02-26
       * NEXT: The nextBuildNumber is set to 10017 but I expected at least 10042
      
      Found 34 problems.
      
      **** SOLUTIONS ****
      copyright-check-gerrit:
       * Archive out-of-order copyright-check-gerrit/builds/10001 -> 2014-05-21_01-35-07
       * Archive out-of-order copyright-check-gerrit/builds/10002 -> 2014-05-21_01-35-08
       * Archive out-of-order copyright-check-gerrit/builds/10003 -> 2014-05-21_01-35-09
       * Archive out-of-order copyright-check-gerrit/builds/10004 -> 2014-05-21_01-49-42
       * Archive out-of-order copyright-check-gerrit/builds/10005 -> 2014-05-21_02-33-34
       * Archive out-of-order copyright-check-gerrit/builds/10006 -> 2014-05-21_02-52-56
       * Archive out-of-order copyright-check-gerrit/builds/10007 -> 2014-05-21_02-52-57
       * Archive out-of-order copyright-check-gerrit/builds/10008 -> 2014-05-21_05-37-27
       * Archive out-of-order copyright-check-gerrit/builds/10009 -> 2014-05-21_06-22-59
       * Archive out-of-order copyright-check-gerrit/builds/10010 -> 2014-05-21_06-23-00
       * Archive out-of-order copyright-check-gerrit/builds/10011 -> 2014-05-21_09-03-06
       * Archive out-of-order copyright-check-gerrit/builds/10012 -> 2014-05-21_09-29-58
       * Archive out-of-order copyright-check-gerrit/builds/10013 -> 2014-05-21_09-35-24
       * Archive out-of-order copyright-check-gerrit/builds/10014 -> 2014-05-21_09-47-15
       * Archive out-of-order copyright-check-gerrit/builds/10015 -> 2014-05-21_10-02-25
       * Archive out-of-order copyright-check-gerrit/builds/10016 -> 2014-05-21_10-02-26
       * Relink 10000 to copyright-check-gerrit/builds/2014-05-20_16-48-18
       * Archive newer build copyright-check-gerrit/builds/2014-05-20_17-20-01
       * Relink 10001 to copyright-check-gerrit/builds/2014-05-20_16-54-41
       * Archive newer build copyright-check-gerrit/builds/2014-05-21_01-35-07
       * Relink 10002 to copyright-check-gerrit/builds/2014-05-20_16-54-42
       * Archive newer build copyright-check-gerrit/builds/2014-05-21_01-35-08
       * Relink 10003 to copyright-check-gerrit/builds/2014-05-20_16-57-47
       * Archive newer build copyright-check-gerrit/builds/2014-05-21_01-35-09
       * Relink 10004 to copyright-check-gerrit/builds/2014-05-20_16-58-22
       * Archive newer build copyright-check-gerrit/builds/2014-05-21_01-49-42
       * Relink 10005 to copyright-check-gerrit/builds/2014-05-20_17-06-25
       * Archive newer build copyright-check-gerrit/builds/2014-05-21_02-33-34
       * Relink 10006 to copyright-check-gerrit/builds/2014-05-20_17-06-26
       * Archive newer build copyright-check-gerrit/builds/2014-05-21_02-52-56
       * Relink 10007 to copyright-check-gerrit/builds/2014-05-20_17-06-27
       * Archive newer build copyright-check-gerrit/builds/2014-05-21_02-52-57
       * Relink 10008 to copyright-check-gerrit/builds/2014-05-20_17-06-28
       * Archive newer build copyright-check-gerrit/builds/2014-05-21_05-37-27
       * Relink 10009 to copyright-check-gerrit/builds/2014-05-20_17-10-57
       * Archive newer build copyright-check-gerrit/builds/2014-05-21_06-22-59
       * Relink 10010 to copyright-check-gerrit/builds/2014-05-20_17-10-58
       * Archive newer build copyright-check-gerrit/builds/2014-05-21_06-23-00
       * Relink 10011 to copyright-check-gerrit/builds/2014-05-20_17-15-14
       * Archive newer build copyright-check-gerrit/builds/2014-05-21_09-03-06
       * Relink 10012 to copyright-check-gerrit/builds/2014-05-20_17-15-15
       * Archive newer build copyright-check-gerrit/builds/2014-05-21_09-29-58
       * Relink 10013 to copyright-check-gerrit/builds/2014-05-20_17-26-02
       * Archive newer build copyright-check-gerrit/builds/2014-05-21_09-35-24
       * Relink 10014 to copyright-check-gerrit/builds/2014-05-20_17-26-03
       * Archive newer build copyright-check-gerrit/builds/2014-05-21_09-47-15
       * Relink 10015 to copyright-check-gerrit/builds/2014-05-20_17-29-01
       * Archive newer build copyright-check-gerrit/builds/2014-05-21_10-02-25
       * Relink 10016 to copyright-check-gerrit/builds/2014-05-20_17-29-02
       * Archive newer build copyright-check-gerrit/builds/2014-05-21_10-02-26
       * Reset nextBuildNumber
      

      You can see that at some point after build 10041 the nextBuildNumber got changed to 10000 and Jenkins merrily started creating duplicate builds.

          [JENKINS-23130] nextBuildNumber keeps being set to previous numbers

          So this looks like it is caused by two things:

          One, updateNextBuildNumber() is called with a too-low value. I'm not sure why this happens, yet.

          Two, updateNextBuildNumber() has a sanity check that fails if Jenkins doesn't know about all the builds on disk (as in JENKINS-15156).

          Jobs.java
              public synchronized void updateNextBuildNumber(int next) throws IOException {
                  RunT lb = getLastBuild();
                  if (lb!=null ?  next>lb.getNumber() : next>0) {
                      this.nextBuildNumber = next;
                      saveNextBuildNumber();
                  }
              }
          

          This check is only as good as the information in getLastBuild(). If the Job was unable to load everything from disk, then this number will be artificially low.

          Christian Höltje added a comment - So this looks like it is caused by two things: One, updateNextBuildNumber() is called with a too-low value. I'm not sure why this happens, yet. Two, updateNextBuildNumber() has a sanity check that fails if Jenkins doesn't know about all the builds on disk (as in JENKINS-15156 ). Jobs.java public synchronized void updateNextBuildNumber( int next) throws IOException { RunT lb = getLastBuild(); if (lb!= null ? next>lb.getNumber() : next>0) { this .nextBuildNumber = next; saveNextBuildNumber(); } } This check is only as good as the information in getLastBuild() . If the Job was unable to load everything from disk, then this number will be artificially low.

          ... except that virtually nothing uses it... On our system (after checking out all the plugin sources I could find) only maven-plugin and next-build-number-plugin call updateNextBuildNumber().

          Christian Höltje added a comment - ... except that virtually nothing uses it... On our system (after checking out all the plugin sources I could find) only maven-plugin and next-build-number-plugin call updateNextBuildNumber() .

          Jesse Glick added a comment -

          After JENKINS-24380 (1.597+), the disk format is simplified and there are some additional assertions that may help prevent this. Reopen if you continue to see the issue after 1.597.

          Jesse Glick added a comment - After JENKINS-24380 (1.597+), the disk format is simplified and there are some additional assertions that may help prevent this. Reopen if you continue to see the issue after 1.597.

            Unassigned Unassigned
            docwhat Christian Höltje
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: