Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-15439

Jenkins build records lazy-loading failed to load some of my jobs.

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core
    • Windows Server 2008 R2

      After upgraded to 1.485, following exception occurs while Jenkins starting up.

       
      2012/10/9 10:53:40AM jenkins.InitReactorRunner$1 onTaskFailed
      SEVERE: Failed Loading job prc-production
      java.lang.IndexOutOfBoundsException: Index: 20, Size: 20
      	at java.util.ArrayList.RangeCheck(Unknown Source)
      	at java.util.ArrayList.get(Unknown Source)
      	at jenkins.model.lazy.SortedList.get(SortedList.java:60)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:402)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:287)
      	at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:998)
      	at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184)
      	at hudson.maven.MavenModuleSet.createTransientActions(MavenModuleSet.java:365)
              ...
      

          [JENKINS-15439] Jenkins build records lazy-loading failed to load some of my jobs.

          I got something similar, exception below. This job didn't have any triggers as it is launched by another job. I managed to get this one to load by adding SCM trigger manually into config.xml.

          SEVERE: Failed Loading job jobX
          java.util.NoSuchElementException
          at java.util.AbstractList$Itr.next(AbstractList.java:364)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.all(AbstractLazyLoadRunMap.java:537)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.entrySet(AbstractLazyLoadRunMap.java:230)
          at java.util.AbstractMap$2$1.<init>(AbstractMap.java:378)
          at java.util.AbstractMap$2.iterator(AbstractMap.java:377)
          at hudson.util.RunList.iterator(RunList.java:102)
          at org.jvnet.hudson.plugins.DownStreamProjectActionFactory.createFor(DownStreamProjectActionFactory.java:59)
          at hudson.model.AbstractProject.createTransientActions(AbstractProject.java:675)
          at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:177)
          at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:665)
          at hudson.maven.MavenModule.updateTransientActions(MavenModule.java:411)
          at hudson.model.AbstractProject.onLoad(AbstractProject.java:299)
          at hudson.maven.MavenModule.onLoad(MavenModule.java:236)
          at hudson.model.Items.load(Items.java:221)
          at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:99)
          at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:669)
          at hudson.model.Items.load(Items.java:221)
          at jenkins.model.Jenkins$17.run(Jenkins.java:2507)
          at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146)
          at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
          at jenkins.model.Jenkins$7.runTask(Jenkins.java:883)
          at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
          at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
          at java.lang.Thread.run(Thread.java:722)

          Jyrki Puttonen added a comment - I got something similar, exception below. This job didn't have any triggers as it is launched by another job. I managed to get this one to load by adding SCM trigger manually into config.xml. SEVERE: Failed Loading job jobX java.util.NoSuchElementException at java.util.AbstractList$Itr.next(AbstractList.java:364) at jenkins.model.lazy.AbstractLazyLoadRunMap.all(AbstractLazyLoadRunMap.java:537) at jenkins.model.lazy.AbstractLazyLoadRunMap.entrySet(AbstractLazyLoadRunMap.java:230) at java.util.AbstractMap$2$1.<init>(AbstractMap.java:378) at java.util.AbstractMap$2.iterator(AbstractMap.java:377) at hudson.util.RunList.iterator(RunList.java:102) at org.jvnet.hudson.plugins.DownStreamProjectActionFactory.createFor(DownStreamProjectActionFactory.java:59) at hudson.model.AbstractProject.createTransientActions(AbstractProject.java:675) at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:177) at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:665) at hudson.maven.MavenModule.updateTransientActions(MavenModule.java:411) at hudson.model.AbstractProject.onLoad(AbstractProject.java:299) at hudson.maven.MavenModule.onLoad(MavenModule.java:236) at hudson.model.Items.load(Items.java:221) at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:99) at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:669) at hudson.model.Items.load(Items.java:221) at jenkins.model.Jenkins$17.run(Jenkins.java:2507) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259) at jenkins.model.Jenkins$7.runTask(Jenkins.java:883) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722)

          Jeff MAURY added a comment -

          I have the same stack track trace, seems to be related to the build history: my jobs are configured to keep only 50 builds, and I got the stack trace:

          java.lang.IndexOutOfBoundsException: Index: 50, Size: 50
          at java.util.ArrayList.rangeCheck(ArrayList.java:604)
          at java.util.ArrayList.get(ArrayList.java:382)
          at jenkins.model.lazy.SortedList.get(SortedList.java:60)
          at jenkins.model.lazy.SortedList.get(SortedList.java:36)
          at jenkins.model.lazy.SortedList.get(SortedList.java:60)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:402)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:287)
          at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:998)
          at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184)
          at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:665)
          at hudson.maven.MavenModule.updateTransientActions(MavenModule.java:411)
          at hudson.model.AbstractProject.onLoad(AbstractProject.java:299)
          at hudson.maven.MavenModule.onLoad(MavenModule.java:236)
          at hudson.model.Items.load(Items.java:221)
          at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:99)
          at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:669)
          at hudson.model.Items.load(Items.java:221)
          at jenkins.model.Jenkins$17.run(Jenkins.java:2507)

          Jeff MAURY added a comment - I have the same stack track trace, seems to be related to the build history: my jobs are configured to keep only 50 builds, and I got the stack trace: java.lang.IndexOutOfBoundsException: Index: 50, Size: 50 at java.util.ArrayList.rangeCheck(ArrayList.java:604) at java.util.ArrayList.get(ArrayList.java:382) at jenkins.model.lazy.SortedList.get(SortedList.java:60) at jenkins.model.lazy.SortedList.get(SortedList.java:36) at jenkins.model.lazy.SortedList.get(SortedList.java:60) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:402) at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:287) at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:998) at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184) at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:665) at hudson.maven.MavenModule.updateTransientActions(MavenModule.java:411) at hudson.model.AbstractProject.onLoad(AbstractProject.java:299) at hudson.maven.MavenModule.onLoad(MavenModule.java:236) at hudson.model.Items.load(Items.java:221) at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:99) at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:669) at hudson.model.Items.load(Items.java:221) at jenkins.model.Jenkins$17.run(Jenkins.java:2507)

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          core/src/main/java/jenkins/model/lazy/AbstractLazyLoadRunMap.java
          core/src/test/java/jenkins/model/lazy/AbstractLazyLoadRunMapTest.java
          http://jenkins-ci.org/commit/jenkins/ce6c6a91767b76167562b7c7235aa001608d1cac
          Log:
          JENKINS-15439

          still not really sure what's causing this, so adding more probes. Those who are seeing this problem, please run this build and send us the message you see.

          I tightened up the assertion pivot<=hi to pivot<hi, because lo<hi.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/jenkins/model/lazy/AbstractLazyLoadRunMap.java core/src/test/java/jenkins/model/lazy/AbstractLazyLoadRunMapTest.java http://jenkins-ci.org/commit/jenkins/ce6c6a91767b76167562b7c7235aa001608d1cac Log: JENKINS-15439 still not really sure what's causing this, so adding more probes. Those who are seeing this problem, please run this build and send us the message you see. I tightened up the assertion pivot<=hi to pivot<hi, because lo<hi.

          I'm creating a new build that has more debug probes for this problem (when it's ready, you should see the dogfood daemon posts an announcement with the pointer to the build.)

          If you are seeing this problem, please use that version and post the message you'll see.

          Kohsuke Kawaguchi added a comment - I'm creating a new build that has more debug probes for this problem (when it's ready, you should see the dogfood daemon posts an announcement with the pointer to the build.) If you are seeing this problem, please use that version and post the message you'll see.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #1982

          Result = SUCCESS

          dogfood added a comment - Integrated in jenkins_main_trunk #1982 Result = SUCCESS

          We experienced the same problem after an automatic update this morning.

          If it's any help figuring out why this was occurring, here's our experience:

          One of our projects was failing to load after the auto-update this morning, with the same exception as above except with the index/size of 3. That's because we have this project only keeping the last 3 builds.

          Now, we looked in the builds directory of this project and it contained the following directory listing:

          drwxr-xr-x. 2 jenkins jenkins 4096 Nov 20  2009 2009-11-20_11-28-12
          drwxr-xr-x  2 jenkins jenkins 4096 Oct  9 15:53 2012-10-09_15-52-05
          drwxr-xr-x  2 jenkins jenkins 4096 Oct  9 16:17 2012-10-09_16-17-07
          drwxr-xr-x  2 jenkins jenkins 4096 Oct  9 16:25 2012-10-09_16-24-49
          lrwxrwxrwx. 1 jenkins jenkins   19 Jan 15  2010 579 -> 2009-11-20_11-28-12
          lrwxrwxrwx  1 jenkins jenkins   19 Oct  9 15:52 7683 -> 2012-10-09_15-52-05
          lrwxrwxrwx  1 jenkins jenkins   19 Oct  9 16:17 7684 -> 2012-10-09_16-17-07
          lrwxrwxrwx  1 jenkins jenkins   19 Oct  9 16:24 7685 -> 2012-10-09_16-24-49

          Build 579 had not been appearing in the list of builds for this project, and as you can see it is quite old (well before we changed our server from using Hudson to Jenkins).

          We deleted the 579 link and it's associated directory and restarted Jenkins. The project then loaded fine without the error.

          Now what's weird is that we have a similar project which is also keeping 3 builds and again the build directory contains 4 builds, one of which is very old:

          lrwxrwxrwx. 1 jenkins jenkins   19 Mar 23  2010 1265 -> 2010-03-23_16-52-53
          drwxr-xr-x. 2 jenkins jenkins 4096 Mar 23  2010 2010-03-23_16-52-53
          drwxr-xr-x  2 jenkins jenkins 4096 Oct 10 10:38 2012-10-10_10-36-42
          drwxr-xr-x  2 jenkins jenkins 4096 Oct 10 10:42 2012-10-10_10-41-42
          drwxr-xr-x  2 jenkins jenkins 4096 Oct 10 12:12 2012-10-10_12-10-57
          lrwxrwxrwx  1 jenkins jenkins   19 Oct 10 10:36 6170 -> 2012-10-10_10-36-42
          lrwxrwxrwx  1 jenkins jenkins   19 Oct 10 10:41 6171 -> 2012-10-10_10-41-42
          lrwxrwxrwx  1 jenkins jenkins   19 Oct 10 12:10 6172 -> 2012-10-10_12-10-57

          Build 1265 does not show up in the list of builds for this project, but this project loads fine when Jenkins starts. Don't know what the difference is.

          If anyone needs to see the config files I can post them on request.

          Duncan Kinnear added a comment - We experienced the same problem after an automatic update this morning. If it's any help figuring out why this was occurring, here's our experience: One of our projects was failing to load after the auto-update this morning, with the same exception as above except with the index/size of 3. That's because we have this project only keeping the last 3 builds. Now, we looked in the builds directory of this project and it contained the following directory listing: drwxr-xr-x. 2 jenkins jenkins 4096 Nov 20  2009 2009-11-20_11-28-12 drwxr-xr-x  2 jenkins jenkins 4096 Oct  9 15:53 2012-10-09_15-52-05 drwxr-xr-x  2 jenkins jenkins 4096 Oct  9 16:17 2012-10-09_16-17-07 drwxr-xr-x  2 jenkins jenkins 4096 Oct  9 16:25 2012-10-09_16-24-49 lrwxrwxrwx. 1 jenkins jenkins   19 Jan 15  2010 579 -> 2009-11-20_11-28-12 lrwxrwxrwx  1 jenkins jenkins   19 Oct  9 15:52 7683 -> 2012-10-09_15-52-05 lrwxrwxrwx  1 jenkins jenkins   19 Oct  9 16:17 7684 -> 2012-10-09_16-17-07 lrwxrwxrwx  1 jenkins jenkins   19 Oct  9 16:24 7685 -> 2012-10-09_16-24-49 Build 579 had not been appearing in the list of builds for this project, and as you can see it is quite old (well before we changed our server from using Hudson to Jenkins). We deleted the 579 link and it's associated directory and restarted Jenkins. The project then loaded fine without the error. Now what's weird is that we have a similar project which is also keeping 3 builds and again the build directory contains 4 builds, one of which is very old: lrwxrwxrwx. 1 jenkins jenkins   19 Mar 23  2010 1265 -> 2010-03-23_16-52-53 drwxr-xr-x. 2 jenkins jenkins 4096 Mar 23  2010 2010-03-23_16-52-53 drwxr-xr-x  2 jenkins jenkins 4096 Oct 10 10:38 2012-10-10_10-36-42 drwxr-xr-x  2 jenkins jenkins 4096 Oct 10 10:42 2012-10-10_10-41-42 drwxr-xr-x  2 jenkins jenkins 4096 Oct 10 12:12 2012-10-10_12-10-57 lrwxrwxrwx  1 jenkins jenkins   19 Oct 10 10:36 6170 -> 2012-10-10_10-36-42 lrwxrwxrwx  1 jenkins jenkins   19 Oct 10 10:41 6171 -> 2012-10-10_10-41-42 lrwxrwxrwx  1 jenkins jenkins   19 Oct 10 12:10 6172 -> 2012-10-10_12-10-57 Build 1265 does not show up in the list of builds for this project, but this project loads fine when Jenkins starts. Don't know what the difference is. If anyone needs to see the config files I can post them on request.

          Tried jenkins_main_trunk #1982, got following errors in log

          Oct 10, 2012 6:53:19 AM jenkins.model.lazy.AbstractLazyLoadRunMap search
          WARNING: Assertion error: failing to load #2147483647 DESC: lo=101,hi=201,pivot=151,size=200 (initial:lo=0,hi=201)
          java.lang.Exception
          at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:414)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:293)
          at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:998)
          at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184)
          at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:665)
          at hudson.maven.MavenModule.updateTransientActions(MavenModule.java:411)
          at hudson.model.AbstractProject.onLoad(AbstractProject.java:299)
          at hudson.maven.MavenModule.onLoad(MavenModule.java:236)
          at hudson.model.Items.load(Items.java:221)
          at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:99)
          at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:669)
          at hudson.model.Items.load(Items.java:221)
          at jenkins.model.Jenkins$17.run(Jenkins.java:2507)
          at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146)
          at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
          at jenkins.model.Jenkins$7.runTask(Jenkins.java:883)
          at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
          at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
          at java.lang.Thread.run(Thread.java:722)

          Oct 10, 2012 6:53:19 AM jenkins.InitReactorRunner$1 onTaskFailed
          SEVERE: Failed Loading job ProjectX_Analysis
          java.lang.ArrayIndexOutOfBoundsException: Assertion error: failing to load #2147483647 DESC: lo=101,hi=201,pivot=151,size=200 (initial:lo=0,hi=201)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:415)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:293)
          at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:998)
          at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184)
          at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:665)
          at hudson.maven.MavenModule.updateTransientActions(MavenModule.java:411)
          at hudson.model.AbstractProject.onLoad(AbstractProject.java:299)
          at hudson.maven.MavenModule.onLoad(MavenModule.java:236)
          at hudson.model.Items.load(Items.java:221)
          at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:99)
          at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:669)
          at hudson.model.Items.load(Items.java:221)
          at jenkins.model.Jenkins$17.run(Jenkins.java:2507)
          at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146)
          at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
          at jenkins.model.Jenkins$7.runTask(Jenkins.java:883)
          at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
          at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
          at java.lang.Thread.run(Thread.java:722)

          Jyrki Puttonen added a comment - Tried jenkins_main_trunk #1982, got following errors in log Oct 10, 2012 6:53:19 AM jenkins.model.lazy.AbstractLazyLoadRunMap search WARNING: Assertion error: failing to load #2147483647 DESC: lo=101,hi=201,pivot=151,size=200 (initial:lo=0,hi=201) java.lang.Exception at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:414) at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:293) at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:998) at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184) at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:665) at hudson.maven.MavenModule.updateTransientActions(MavenModule.java:411) at hudson.model.AbstractProject.onLoad(AbstractProject.java:299) at hudson.maven.MavenModule.onLoad(MavenModule.java:236) at hudson.model.Items.load(Items.java:221) at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:99) at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:669) at hudson.model.Items.load(Items.java:221) at jenkins.model.Jenkins$17.run(Jenkins.java:2507) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259) at jenkins.model.Jenkins$7.runTask(Jenkins.java:883) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Oct 10, 2012 6:53:19 AM jenkins.InitReactorRunner$1 onTaskFailed SEVERE: Failed Loading job ProjectX_Analysis java.lang.ArrayIndexOutOfBoundsException: Assertion error: failing to load #2147483647 DESC: lo=101,hi=201,pivot=151,size=200 (initial:lo=0,hi=201) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:415) at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:293) at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:998) at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184) at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:665) at hudson.maven.MavenModule.updateTransientActions(MavenModule.java:411) at hudson.model.AbstractProject.onLoad(AbstractProject.java:299) at hudson.maven.MavenModule.onLoad(MavenModule.java:236) at hudson.model.Items.load(Items.java:221) at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:99) at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:669) at hudson.model.Items.load(Items.java:221) at jenkins.model.Jenkins$17.run(Jenkins.java:2507) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259) at jenkins.model.Jenkins$7.runTask(Jenkins.java:883) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722)

          Pei-Tang Huang added a comment - - edited

          After upgrading to the debug build, stack traces like jyrkip posted appeared:

          java.lang.ArrayIndexOutOfBoundsException: Assertion error: failing to load #2147483647 DESC: lo=14,hi=27,pivot=20,size=20 (initial:lo=0,hi=27)
          

          The "builds" directory within the job contains 20 valid build directories plus 7 outdateds (6 of them are empty, and the oldest one contains a "log" file):

          +---2011-11-04_16-35-12
          |       log
          |       
          +---2011-11-10_15-19-14
          +---2012-02-14_16-46-20
          +---2012-02-20_20-03-43
          +---2012-02-21_14-21-11
          +---2012-02-23_13-06-07
          +---2012-02-29_13-02-39
          +---2012-10-04_20-34-15
          |       build.xml
          |       changelog.xml
          |       log
          |       revision.txt
          |       
          +---2012-10-04_21-50-35
          |       build.xml
          |       changelog.xml
          |       log
          |       revision.txt
          |       
          +--- and 18 more build seems all valid
          

          I tried to remove some of those builds, all with no luck:

          • Round 1: do nothing to the builds dir (20 valid + 7 outdated)
            Assertion error: failing to load #2147483647 DESC: lo=14,hi=27,pivot=20,size=20 (initial:lo=0,hi=27)
          • Round 2: remove the oldest build which contains only one "log" file (20 valid + 6 outdated)
            Assertion error: failing to load #2147483647 DESC: lo=12,hi=23,pivot=17,size=20 (initial:lo=0,hi=23)
          • Round 3: remove all outdated builds (20 valid)
            Assertion error: failing to load #2147483647 DESC: lo=11,hi=21,pivot=16,size=20 (initial:lo=0,hi=21)
          • Round 4: remove the oldest valid build (19 valid)
            Assertion error: failing to load #2147483647 DESC: lo=11,hi=21,pivot=16,size=20 (initial:lo=0,hi=21)

          Pei-Tang Huang added a comment - - edited After upgrading to the debug build, stack traces like jyrkip posted appeared: java.lang.ArrayIndexOutOfBoundsException: Assertion error: failing to load #2147483647 DESC: lo=14,hi=27,pivot=20,size=20 (initial:lo=0,hi=27) The "builds" directory within the job contains 20 valid build directories plus 7 outdateds (6 of them are empty, and the oldest one contains a "log" file): +---2011-11-04_16-35-12 | log | +---2011-11-10_15-19-14 +---2012-02-14_16-46-20 +---2012-02-20_20-03-43 +---2012-02-21_14-21-11 +---2012-02-23_13-06-07 +---2012-02-29_13-02-39 +---2012-10-04_20-34-15 | build.xml | changelog.xml | log | revision.txt | +---2012-10-04_21-50-35 | build.xml | changelog.xml | log | revision.txt | +--- and 18 more build seems all valid I tried to remove some of those builds, all with no luck: Round 1: do nothing to the builds dir (20 valid + 7 outdated) Assertion error: failing to load #2147483647 DESC: lo=14,hi=27,pivot=20,size=20 (initial:lo=0,hi=27) Round 2: remove the oldest build which contains only one "log" file (20 valid + 6 outdated) Assertion error: failing to load #2147483647 DESC: lo=12,hi=23,pivot=17,size=20 (initial:lo=0,hi=23) Round 3: remove all outdated builds (20 valid) Assertion error: failing to load #2147483647 DESC: lo=11,hi=21,pivot=16,size=20 (initial:lo=0,hi=21) Round 4: remove the oldest valid build (19 valid) Assertion error: failing to load #2147483647 DESC: lo=11,hi=21,pivot=16,size=20 (initial:lo=0,hi=21)

          nicusorb added a comment -

          I have the same problem. Here is the error from the log file:

          SEVERE: Failed Loading job my_job_name
          java.lang.IndexOutOfBoundsException: Index: 48, Size: 48
          at java.util.ArrayList.RangeCheck(ArrayList.java:547)
          at java.util.ArrayList.get(ArrayList.java:322)
          at jenkins.model.lazy.SortedList.get(SortedList.java:60)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:402)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:287)
          at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:998)
          at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184)
          at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:665)
          at hudson.maven.MavenModule.updateTransientActions(MavenModule.java:411)
          at hudson.model.AbstractProject.onLoad(AbstractProject.java:299)
          at hudson.maven.MavenModule.onLoad(MavenModule.java:236)
          at hudson.model.Items.load(Items.java:221)
          at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:99)
          at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:669)
          at hudson.model.Items.load(Items.java:221)
          at jenkins.model.Jenkins$17.run(Jenkins.java:2507)
          at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146)
          at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
          at jenkins.model.Jenkins$7.runTask(Jenkins.java:883)
          at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
          at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:662)

          nicusorb added a comment - I have the same problem. Here is the error from the log file: SEVERE: Failed Loading job my_job_name java.lang.IndexOutOfBoundsException: Index: 48, Size: 48 at java.util.ArrayList.RangeCheck(ArrayList.java:547) at java.util.ArrayList.get(ArrayList.java:322) at jenkins.model.lazy.SortedList.get(SortedList.java:60) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:402) at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:287) at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:998) at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184) at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:665) at hudson.maven.MavenModule.updateTransientActions(MavenModule.java:411) at hudson.model.AbstractProject.onLoad(AbstractProject.java:299) at hudson.maven.MavenModule.onLoad(MavenModule.java:236) at hudson.model.Items.load(Items.java:221) at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:99) at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:669) at hudson.model.Items.load(Items.java:221) at jenkins.model.Jenkins$17.run(Jenkins.java:2507) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259) at jenkins.model.Jenkins$7.runTask(Jenkins.java:883) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)

          With the nightly build I have this stacktrace:

          Oct 10, 2012 9:51:59 AM jenkins.InitReactorRunner$1 onTaskFailed
          SEVERE: Failed Loading job JAPI-Sonar
          java.lang.ArrayIndexOutOfBoundsException: Assertion error: failing to load #2147483647 DESC: lo=4,hi=7,pivot=5,size=6 (initial:lo=0,hi=7)
          	at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:415)
          	at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:293)
          	at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:998)
          	at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:137)
          	at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184)
          	at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:665)
          	at hudson.maven.MavenModule.updateTransientActions(MavenModule.java:412)
          	at hudson.model.AbstractProject.onLoad(AbstractProject.java:299)
          	at hudson.maven.MavenModule.onLoad(MavenModule.java:237)
          	at hudson.model.Items.load(Items.java:221)
          	at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:99)
          	at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:671)
          	at hudson.model.Items.load(Items.java:221)
          	at jenkins.model.Jenkins$17.run(Jenkins.java:2507)
          	at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146)
          	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
          	at jenkins.model.Jenkins$7.runTask(Jenkins.java:883)
          	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
          	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          	at java.lang.Thread.run(Thread.java:619)
          

          This job has 7 saved builds in its history.

          Jean-Christophe Sirot added a comment - With the nightly build I have this stacktrace: Oct 10, 2012 9:51:59 AM jenkins.InitReactorRunner$1 onTaskFailed SEVERE: Failed Loading job JAPI-Sonar java.lang.ArrayIndexOutOfBoundsException: Assertion error: failing to load #2147483647 DESC: lo=4,hi=7,pivot=5,size=6 (initial:lo=0,hi=7) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:415) at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:293) at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:998) at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:137) at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184) at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:665) at hudson.maven.MavenModule.updateTransientActions(MavenModule.java:412) at hudson.model.AbstractProject.onLoad(AbstractProject.java:299) at hudson.maven.MavenModule.onLoad(MavenModule.java:237) at hudson.model.Items.load(Items.java:221) at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:99) at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:671) at hudson.model.Items.load(Items.java:221) at jenkins.model.Jenkins$17.run(Jenkins.java:2507) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259) at jenkins.model.Jenkins$7.runTask(Jenkins.java:883) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) This job has 7 saved builds in its history.

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          core/src/main/java/jenkins/model/lazy/AbstractLazyLoadRunMap.java
          core/src/test/java/jenkins/model/lazy/AbstractLazyLoadRunMapTest.java
          core/src/test/java/jenkins/model/lazy/FakeMapBuilder.java
          core/src/test/java/jenkins/model/lazy/SortedIntListTest.java
          http://jenkins-ci.org/commit/jenkins/ff2d15aced231bdf2c8955a4e9f554c0631f38c3
          Log:
          JENKINS-15439

          adding more probe.

          Compare: https://github.com/jenkinsci/jenkins/compare/d372f71a0bd7...ff2d15aced23

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/jenkins/model/lazy/AbstractLazyLoadRunMap.java core/src/test/java/jenkins/model/lazy/AbstractLazyLoadRunMapTest.java core/src/test/java/jenkins/model/lazy/FakeMapBuilder.java core/src/test/java/jenkins/model/lazy/SortedIntListTest.java http://jenkins-ci.org/commit/jenkins/ff2d15aced231bdf2c8955a4e9f554c0631f38c3 Log: JENKINS-15439 adding more probe. Compare: https://github.com/jenkinsci/jenkins/compare/d372f71a0bd7...ff2d15aced23

          Thanks everyone for trying a snapshot build.

          I'm still having trouble understanding how the assertion problem happens, so I added more probes in the code. If you are seeing this problem, please try the newer build and report back what you see.

          It should be in #1990 if I didn't mess up anything, but the dogfood daemon will tell us exactly which build it should be.

          Kohsuke Kawaguchi added a comment - Thanks everyone for trying a snapshot build. I'm still having trouble understanding how the assertion problem happens, so I added more probes in the code. If you are seeing this problem, please try the newer build and report back what you see. It should be in #1990 if I didn't mess up anything, but the dogfood daemon will tell us exactly which build it should be.

          This is a very critical bug that I'd like to fix quickly, so I really appreciate everyone's help.

          Kohsuke Kawaguchi added a comment - This is a very critical bug that I'd like to fix quickly, so I really appreciate everyone's help.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #1990
          JENKINS-15439 (Revision ff2d15aced231bdf2c8955a4e9f554c0631f38c3)

          Result = SUCCESS
          kohsuke : ff2d15aced231bdf2c8955a4e9f554c0631f38c3
          Files :

          • core/src/test/java/jenkins/model/lazy/SortedIntListTest.java
          • core/src/test/java/jenkins/model/lazy/AbstractLazyLoadRunMapTest.java
          • core/src/main/java/jenkins/model/lazy/AbstractLazyLoadRunMap.java
          • core/src/test/java/jenkins/model/lazy/FakeMapBuilder.java

          dogfood added a comment - Integrated in jenkins_main_trunk #1990 JENKINS-15439 (Revision ff2d15aced231bdf2c8955a4e9f554c0631f38c3) Result = SUCCESS kohsuke : ff2d15aced231bdf2c8955a4e9f554c0631f38c3 Files : core/src/test/java/jenkins/model/lazy/SortedIntListTest.java core/src/test/java/jenkins/model/lazy/AbstractLazyLoadRunMapTest.java core/src/main/java/jenkins/model/lazy/AbstractLazyLoadRunMap.java core/src/test/java/jenkins/model/lazy/FakeMapBuilder.java

          Mark Wielaard added a comment -

          Oct 10, 2012 10:33:43 PM jenkins.model.lazy.AbstractLazyLoadRunMap search
          WARNING: Assertion error: failing to load #2147483647 DESC: lo=127,hi=253,pivot=190,size=252 (initial:lo=0,hi=253,size=253)
          java.lang.Exception
          at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:425)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:293)
          at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:998)
          at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184)
          at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:665)
          at hudson.maven.MavenModule.updateTransientActions(MavenModule.java:411)
          at hudson.model.AbstractProject.onLoad(AbstractProject.java:299)
          at hudson.maven.MavenModule.onLoad(MavenModule.java:236)
          at hudson.model.Items.load(Items.java:221)
          at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:99)
          at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:669)
          at hudson.model.Items.load(Items.java:221)
          at jenkins.model.Jenkins$17.run(Jenkins.java:2507)
          at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146)
          at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
          at jenkins.model.Jenkins$7.runTask(Jenkins.java:883)
          at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
          at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
          at java.lang.Thread.run(Thread.java:636)
          Oct 10, 2012 10:33:43 PM jenkins.InitReactorRunner$1 onTaskFailed
          SEVERE: Failed Loading job Thermostat
          java.lang.ArrayIndexOutOfBoundsException: Assertion error: failing to load #2147483647 DESC: lo=127,hi=253,pivot=190,size=252 (initial:lo=0,hi=253,size=253)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:426)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:293)
          at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:998)
          at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184)
          at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:665)
          at hudson.maven.MavenModule.updateTransientActions(MavenModule.java:411)
          at hudson.model.AbstractProject.onLoad(AbstractProject.java:299)
          at hudson.maven.MavenModule.onLoad(MavenModule.java:236)
          at hudson.model.Items.load(Items.java:221)
          at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:99)
          at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:669)
          at hudson.model.Items.load(Items.java:221)
          at jenkins.model.Jenkins$17.run(Jenkins.java:2507)
          at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146)
          at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
          at jenkins.model.Jenkins$7.runTask(Jenkins.java:883)
          at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
          at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
          at java.lang.Thread.run(Thread.java:636)
          Oct 10, 2012 10:33:43 PM jenkins.InitReactorRunner$1 onAttained
          INFO: Loaded all jobs

          Mark Wielaard added a comment - Oct 10, 2012 10:33:43 PM jenkins.model.lazy.AbstractLazyLoadRunMap search WARNING: Assertion error: failing to load #2147483647 DESC: lo=127,hi=253,pivot=190,size=252 (initial:lo=0,hi=253,size=253) java.lang.Exception at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:425) at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:293) at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:998) at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184) at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:665) at hudson.maven.MavenModule.updateTransientActions(MavenModule.java:411) at hudson.model.AbstractProject.onLoad(AbstractProject.java:299) at hudson.maven.MavenModule.onLoad(MavenModule.java:236) at hudson.model.Items.load(Items.java:221) at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:99) at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:669) at hudson.model.Items.load(Items.java:221) at jenkins.model.Jenkins$17.run(Jenkins.java:2507) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259) at jenkins.model.Jenkins$7.runTask(Jenkins.java:883) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Oct 10, 2012 10:33:43 PM jenkins.InitReactorRunner$1 onTaskFailed SEVERE: Failed Loading job Thermostat java.lang.ArrayIndexOutOfBoundsException: Assertion error: failing to load #2147483647 DESC: lo=127,hi=253,pivot=190,size=252 (initial:lo=0,hi=253,size=253) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:426) at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:293) at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:998) at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184) at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:665) at hudson.maven.MavenModule.updateTransientActions(MavenModule.java:411) at hudson.model.AbstractProject.onLoad(AbstractProject.java:299) at hudson.maven.MavenModule.onLoad(MavenModule.java:236) at hudson.model.Items.load(Items.java:221) at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:99) at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:669) at hudson.model.Items.load(Items.java:221) at jenkins.model.Jenkins$17.run(Jenkins.java:2507) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259) at jenkins.model.Jenkins$7.runTask(Jenkins.java:883) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Oct 10, 2012 10:33:43 PM jenkins.InitReactorRunner$1 onAttained INFO: Loaded all jobs

          Just looking at the code of the 'search' method in AbstractLazyLoadRunMap.java

          Shouldn't the line which does 'idOnDisk.remove(pivot)' (currently line 436) be accompanied by a reduction of 'hi' by one? If you remove an element from 'idOnDisk', then 'hi' should be reduced by one as well.

          Or am I barking up the wrong tree?

          The fact that, in our case, our fourth (old) build was not appearing in the list of builds indicates that it was an invalid build and the 'search' method will be trying to ignore it.

          For those experiencing this error, it might pay you to check that ALL the directories under the 'builds' directory of your project are valid (compare the contents with one-another) as it looks like the code is not coping with build directories that are not valid.

          Duncan Kinnear added a comment - Just looking at the code of the 'search' method in AbstractLazyLoadRunMap.java Shouldn't the line which does 'idOnDisk.remove(pivot)' (currently line 436) be accompanied by a reduction of 'hi' by one? If you remove an element from 'idOnDisk', then 'hi' should be reduced by one as well. Or am I barking up the wrong tree? The fact that, in our case, our fourth (old) build was not appearing in the list of builds indicates that it was an invalid build and the 'search' method will be trying to ignore it. For those experiencing this error, it might pay you to check that ALL the directories under the 'builds' directory of your project are valid (compare the contents with one-another) as it looks like the code is not coping with build directories that are not valid.

          See also investigations on JENKINS-15465

          But the symptom/fix is more in line with what is reported here.

          I added some logging to AbstractLazyLoadRunMap to watch how lo, hi, pivot and size changed when high ended up being greater than size. I also added logging to trigger when r was detected as null at line 724.

          11-Oct-2012 01:21:42 jenkins.model.lazy.AbstractLazyLoadRunMap search
          WARNING: lo=0, hi=22, pivot=11, size=22
          11-Oct-2012 01:21:42 jenkins.model.lazy.AbstractLazyLoadRunMap search
          WARNING: lo=12, hi=22, pivot=17, size=22
          11-Oct-2012 01:21:42 jenkins.model.lazy.AbstractLazyLoadRunMap search
          WARNING: lo=18, hi=22, pivot=20, size=21
          11-Oct-2012 01:21:42 jenkins.InitReactorRunner$1 onTaskFailed
          SEVERE: Failed Loading job tcserv
          java.lang.ArrayIndexOutOfBoundsException: lo=18, hi=22, pivot=20, size=21
          

          In this failing scenario the debug for r==null never triggers but size reduces.

          I've tracked this down to SortedList doing a shallow clone another list and sharing the same underlying data list. Thus if another thread is accessing the list and removes an item the clone will see the item removed too.

              public SortedList(List<T> data) {
                  this.data = data;
                  assert isSorted();
              }
          

          This can be fixed using

          this.data = new ArrayList(data);
          

          Richard Mortimer added a comment - See also investigations on JENKINS-15465 But the symptom/fix is more in line with what is reported here. I added some logging to AbstractLazyLoadRunMap to watch how lo, hi, pivot and size changed when high ended up being greater than size. I also added logging to trigger when r was detected as null at line 724. 11-Oct-2012 01:21:42 jenkins.model.lazy.AbstractLazyLoadRunMap search WARNING: lo=0, hi=22, pivot=11, size=22 11-Oct-2012 01:21:42 jenkins.model.lazy.AbstractLazyLoadRunMap search WARNING: lo=12, hi=22, pivot=17, size=22 11-Oct-2012 01:21:42 jenkins.model.lazy.AbstractLazyLoadRunMap search WARNING: lo=18, hi=22, pivot=20, size=21 11-Oct-2012 01:21:42 jenkins.InitReactorRunner$1 onTaskFailed SEVERE: Failed Loading job tcserv java.lang.ArrayIndexOutOfBoundsException: lo=18, hi=22, pivot=20, size=21 In this failing scenario the debug for r==null never triggers but size reduces. I've tracked this down to SortedList doing a shallow clone another list and sharing the same underlying data list. Thus if another thread is accessing the list and removes an item the clone will see the item removed too. public SortedList(List<T> data) { this .data = data; assert isSorted(); } This can be fixed using this .data = new ArrayList(data);

          Here is the exception I have with build #1990 (same job as yesterday).

          java.lang.ArrayIndexOutOfBoundsException: Assertion error: failing to load #2147483647 DESC: lo=4,hi=7,pivot=5,size=6 (initial:lo=0,hi=7,size=7)
          	at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:426)
          	at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:293)
          	at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:998)
          	at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:137)
          	at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184)
          	at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:665)
          	at hudson.maven.MavenModule.updateTransientActions(MavenModule.java:412)
          	at hudson.model.AbstractProject.onLoad(AbstractProject.java:299)
          	at hudson.maven.MavenModule.onLoad(MavenModule.java:237)
          	at hudson.model.Items.load(Items.java:221)
          	at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:99)
          	at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:671)
          	at hudson.model.Items.load(Items.java:221)
          	at jenkins.model.Jenkins$17.run(Jenkins.java:2507)
          	at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146)
          	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
          	at jenkins.model.Jenkins$7.runTask(Jenkins.java:883)
          	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
          	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          	at java.lang.Thread.run(Thread.java:619)
          

          Jean-Christophe Sirot added a comment - Here is the exception I have with build #1990 (same job as yesterday). java.lang.ArrayIndexOutOfBoundsException: Assertion error: failing to load #2147483647 DESC: lo=4,hi=7,pivot=5,size=6 (initial:lo=0,hi=7,size=7) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:426) at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:293) at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:998) at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:137) at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184) at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:665) at hudson.maven.MavenModule.updateTransientActions(MavenModule.java:412) at hudson.model.AbstractProject.onLoad(AbstractProject.java:299) at hudson.maven.MavenModule.onLoad(MavenModule.java:237) at hudson.model.Items.load(Items.java:221) at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:99) at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:671) at hudson.model.Items.load(Items.java:221) at jenkins.model.Jenkins$17.run(Jenkins.java:2507) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259) at jenkins.model.Jenkins$7.runTask(Jenkins.java:883) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619)

          I submitted the following pull request with my suggested fix.

          https://github.com/jenkinsci/jenkins/pull/585

          Richard Mortimer added a comment - I submitted the following pull request with my suggested fix. https://github.com/jenkinsci/jenkins/pull/585

          Code changed in jenkins
          User: Richard Mortimer
          Path:
          changelog.html
          core/src/main/java/jenkins/model/lazy/SortedList.java
          core/src/test/java/jenkins/model/lazy/SortedListTest.java
          http://jenkins-ci.org/commit/jenkins/761c437983ad2b08399efa5e69e848764fcff4de
          Log:
          [FIXED JENKINS-15439] Jenkins build records lazy-loading failed to load some of my jobs

          Ensure that SortedList clones the underlying data list.
          Without this changes to a clone also change the original list.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Richard Mortimer Path: changelog.html core/src/main/java/jenkins/model/lazy/SortedList.java core/src/test/java/jenkins/model/lazy/SortedListTest.java http://jenkins-ci.org/commit/jenkins/761c437983ad2b08399efa5e69e848764fcff4de Log: [FIXED JENKINS-15439] Jenkins build records lazy-loading failed to load some of my jobs Ensure that SortedList clones the underlying data list. Without this changes to a clone also change the original list.

          Richard, you are THE MAN!

          I think I'm going to backport this to the RC branch.

          Kohsuke Kawaguchi added a comment - Richard, you are THE MAN! I think I'm going to backport this to the RC branch.

          Code changed in jenkins
          User: Richard Mortimer
          Path:
          changelog.html
          core/src/main/java/jenkins/model/lazy/SortedList.java
          core/src/test/java/jenkins/model/lazy/SortedListTest.java
          http://jenkins-ci.org/commit/jenkins/6fca1f08d20fcb9d8639ce6e3ff83aa63e145fde
          Log:
          [FIXED JENKINS-15439] Jenkins build records lazy-loading failed to load some of my jobs

          Ensure that SortedList clones the underlying data list.
          Without this changes to a clone also change the original list.
          (cherry picked from commit 761c437983ad2b08399efa5e69e848764fcff4de)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Richard Mortimer Path: changelog.html core/src/main/java/jenkins/model/lazy/SortedList.java core/src/test/java/jenkins/model/lazy/SortedListTest.java http://jenkins-ci.org/commit/jenkins/6fca1f08d20fcb9d8639ce6e3ff83aa63e145fde Log: [FIXED JENKINS-15439] Jenkins build records lazy-loading failed to load some of my jobs Ensure that SortedList clones the underlying data list. Without this changes to a clone also change the original list. (cherry picked from commit 761c437983ad2b08399efa5e69e848764fcff4de)

          Backported to 1.486.

          Kohsuke Kawaguchi added a comment - Backported to 1.486.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #1994
          [FIXED JENKINS-15439] Jenkins build records lazy-loading failed to load some of my jobs (Revision 6fca1f08d20fcb9d8639ce6e3ff83aa63e145fde)

          Result = SUCCESS
          kohsuke : 6fca1f08d20fcb9d8639ce6e3ff83aa63e145fde
          Files :

          • changelog.html
          • core/src/test/java/jenkins/model/lazy/SortedListTest.java
          • core/src/main/java/jenkins/model/lazy/SortedList.java

          dogfood added a comment - Integrated in jenkins_main_trunk #1994 [FIXED JENKINS-15439] Jenkins build records lazy-loading failed to load some of my jobs (Revision 6fca1f08d20fcb9d8639ce6e3ff83aa63e145fde) Result = SUCCESS kohsuke : 6fca1f08d20fcb9d8639ce6e3ff83aa63e145fde Files : changelog.html core/src/test/java/jenkins/model/lazy/SortedListTest.java core/src/main/java/jenkins/model/lazy/SortedList.java

          Jose Sa added a comment -

          I'm still getting problems loading jobs after LazyLoading feature introduction. I've added my symptoms in JENKINS-15533

          Jose Sa added a comment - I'm still getting problems loading jobs after LazyLoading feature introduction. I've added my symptoms in JENKINS-15533

          Ben McDonie added a comment -

          I am running version 1.488 and am still having issues with LazyLoading. My stack trace is in JENKINS-15642. It happens near the end of Maven builds that were triggered by SCM changes.

          Ben McDonie added a comment - I am running version 1.488 and am still having issues with LazyLoading. My stack trace is in JENKINS-15642 . It happens near the end of Maven builds that were triggered by SCM changes.

            kohsuke Kohsuke Kawaguchi
            beta Pei-Tang Huang
            Votes:
            5 Vote for this issue
            Watchers:
            25 Start watching this issue

              Created:
              Updated:
              Resolved: