Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-15439

Jenkins build records lazy-loading failed to load some of my jobs.

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core
    • Windows Server 2008 R2

      After upgraded to 1.485, following exception occurs while Jenkins starting up.

       
      2012/10/9 10:53:40AM jenkins.InitReactorRunner$1 onTaskFailed
      SEVERE: Failed Loading job prc-production
      java.lang.IndexOutOfBoundsException: Index: 20, Size: 20
      	at java.util.ArrayList.RangeCheck(Unknown Source)
      	at java.util.ArrayList.get(Unknown Source)
      	at jenkins.model.lazy.SortedList.get(SortedList.java:60)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:402)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:287)
      	at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:998)
      	at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184)
      	at hudson.maven.MavenModuleSet.createTransientActions(MavenModuleSet.java:365)
              ...
      

          [JENKINS-15439] Jenkins build records lazy-loading failed to load some of my jobs.

          See also investigations on JENKINS-15465

          But the symptom/fix is more in line with what is reported here.

          I added some logging to AbstractLazyLoadRunMap to watch how lo, hi, pivot and size changed when high ended up being greater than size. I also added logging to trigger when r was detected as null at line 724.

          11-Oct-2012 01:21:42 jenkins.model.lazy.AbstractLazyLoadRunMap search
          WARNING: lo=0, hi=22, pivot=11, size=22
          11-Oct-2012 01:21:42 jenkins.model.lazy.AbstractLazyLoadRunMap search
          WARNING: lo=12, hi=22, pivot=17, size=22
          11-Oct-2012 01:21:42 jenkins.model.lazy.AbstractLazyLoadRunMap search
          WARNING: lo=18, hi=22, pivot=20, size=21
          11-Oct-2012 01:21:42 jenkins.InitReactorRunner$1 onTaskFailed
          SEVERE: Failed Loading job tcserv
          java.lang.ArrayIndexOutOfBoundsException: lo=18, hi=22, pivot=20, size=21
          

          In this failing scenario the debug for r==null never triggers but size reduces.

          I've tracked this down to SortedList doing a shallow clone another list and sharing the same underlying data list. Thus if another thread is accessing the list and removes an item the clone will see the item removed too.

              public SortedList(List<T> data) {
                  this.data = data;
                  assert isSorted();
              }
          

          This can be fixed using

          this.data = new ArrayList(data);
          

          Richard Mortimer added a comment - See also investigations on JENKINS-15465 But the symptom/fix is more in line with what is reported here. I added some logging to AbstractLazyLoadRunMap to watch how lo, hi, pivot and size changed when high ended up being greater than size. I also added logging to trigger when r was detected as null at line 724. 11-Oct-2012 01:21:42 jenkins.model.lazy.AbstractLazyLoadRunMap search WARNING: lo=0, hi=22, pivot=11, size=22 11-Oct-2012 01:21:42 jenkins.model.lazy.AbstractLazyLoadRunMap search WARNING: lo=12, hi=22, pivot=17, size=22 11-Oct-2012 01:21:42 jenkins.model.lazy.AbstractLazyLoadRunMap search WARNING: lo=18, hi=22, pivot=20, size=21 11-Oct-2012 01:21:42 jenkins.InitReactorRunner$1 onTaskFailed SEVERE: Failed Loading job tcserv java.lang.ArrayIndexOutOfBoundsException: lo=18, hi=22, pivot=20, size=21 In this failing scenario the debug for r==null never triggers but size reduces. I've tracked this down to SortedList doing a shallow clone another list and sharing the same underlying data list. Thus if another thread is accessing the list and removes an item the clone will see the item removed too. public SortedList(List<T> data) { this .data = data; assert isSorted(); } This can be fixed using this .data = new ArrayList(data);

          Here is the exception I have with build #1990 (same job as yesterday).

          java.lang.ArrayIndexOutOfBoundsException: Assertion error: failing to load #2147483647 DESC: lo=4,hi=7,pivot=5,size=6 (initial:lo=0,hi=7,size=7)
          	at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:426)
          	at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:293)
          	at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:998)
          	at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:137)
          	at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184)
          	at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:665)
          	at hudson.maven.MavenModule.updateTransientActions(MavenModule.java:412)
          	at hudson.model.AbstractProject.onLoad(AbstractProject.java:299)
          	at hudson.maven.MavenModule.onLoad(MavenModule.java:237)
          	at hudson.model.Items.load(Items.java:221)
          	at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:99)
          	at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:671)
          	at hudson.model.Items.load(Items.java:221)
          	at jenkins.model.Jenkins$17.run(Jenkins.java:2507)
          	at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146)
          	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
          	at jenkins.model.Jenkins$7.runTask(Jenkins.java:883)
          	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
          	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          	at java.lang.Thread.run(Thread.java:619)
          

          Jean-Christophe Sirot added a comment - Here is the exception I have with build #1990 (same job as yesterday). java.lang.ArrayIndexOutOfBoundsException: Assertion error: failing to load #2147483647 DESC: lo=4,hi=7,pivot=5,size=6 (initial:lo=0,hi=7,size=7) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:426) at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:293) at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:998) at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:137) at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184) at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:665) at hudson.maven.MavenModule.updateTransientActions(MavenModule.java:412) at hudson.model.AbstractProject.onLoad(AbstractProject.java:299) at hudson.maven.MavenModule.onLoad(MavenModule.java:237) at hudson.model.Items.load(Items.java:221) at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:99) at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:671) at hudson.model.Items.load(Items.java:221) at jenkins.model.Jenkins$17.run(Jenkins.java:2507) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259) at jenkins.model.Jenkins$7.runTask(Jenkins.java:883) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619)

          I submitted the following pull request with my suggested fix.

          https://github.com/jenkinsci/jenkins/pull/585

          Richard Mortimer added a comment - I submitted the following pull request with my suggested fix. https://github.com/jenkinsci/jenkins/pull/585

          Code changed in jenkins
          User: Richard Mortimer
          Path:
          changelog.html
          core/src/main/java/jenkins/model/lazy/SortedList.java
          core/src/test/java/jenkins/model/lazy/SortedListTest.java
          http://jenkins-ci.org/commit/jenkins/761c437983ad2b08399efa5e69e848764fcff4de
          Log:
          [FIXED JENKINS-15439] Jenkins build records lazy-loading failed to load some of my jobs

          Ensure that SortedList clones the underlying data list.
          Without this changes to a clone also change the original list.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Richard Mortimer Path: changelog.html core/src/main/java/jenkins/model/lazy/SortedList.java core/src/test/java/jenkins/model/lazy/SortedListTest.java http://jenkins-ci.org/commit/jenkins/761c437983ad2b08399efa5e69e848764fcff4de Log: [FIXED JENKINS-15439] Jenkins build records lazy-loading failed to load some of my jobs Ensure that SortedList clones the underlying data list. Without this changes to a clone also change the original list.

          Richard, you are THE MAN!

          I think I'm going to backport this to the RC branch.

          Kohsuke Kawaguchi added a comment - Richard, you are THE MAN! I think I'm going to backport this to the RC branch.

          Code changed in jenkins
          User: Richard Mortimer
          Path:
          changelog.html
          core/src/main/java/jenkins/model/lazy/SortedList.java
          core/src/test/java/jenkins/model/lazy/SortedListTest.java
          http://jenkins-ci.org/commit/jenkins/6fca1f08d20fcb9d8639ce6e3ff83aa63e145fde
          Log:
          [FIXED JENKINS-15439] Jenkins build records lazy-loading failed to load some of my jobs

          Ensure that SortedList clones the underlying data list.
          Without this changes to a clone also change the original list.
          (cherry picked from commit 761c437983ad2b08399efa5e69e848764fcff4de)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Richard Mortimer Path: changelog.html core/src/main/java/jenkins/model/lazy/SortedList.java core/src/test/java/jenkins/model/lazy/SortedListTest.java http://jenkins-ci.org/commit/jenkins/6fca1f08d20fcb9d8639ce6e3ff83aa63e145fde Log: [FIXED JENKINS-15439] Jenkins build records lazy-loading failed to load some of my jobs Ensure that SortedList clones the underlying data list. Without this changes to a clone also change the original list. (cherry picked from commit 761c437983ad2b08399efa5e69e848764fcff4de)

          Backported to 1.486.

          Kohsuke Kawaguchi added a comment - Backported to 1.486.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #1994
          [FIXED JENKINS-15439] Jenkins build records lazy-loading failed to load some of my jobs (Revision 6fca1f08d20fcb9d8639ce6e3ff83aa63e145fde)

          Result = SUCCESS
          kohsuke : 6fca1f08d20fcb9d8639ce6e3ff83aa63e145fde
          Files :

          • changelog.html
          • core/src/test/java/jenkins/model/lazy/SortedListTest.java
          • core/src/main/java/jenkins/model/lazy/SortedList.java

          dogfood added a comment - Integrated in jenkins_main_trunk #1994 [FIXED JENKINS-15439] Jenkins build records lazy-loading failed to load some of my jobs (Revision 6fca1f08d20fcb9d8639ce6e3ff83aa63e145fde) Result = SUCCESS kohsuke : 6fca1f08d20fcb9d8639ce6e3ff83aa63e145fde Files : changelog.html core/src/test/java/jenkins/model/lazy/SortedListTest.java core/src/main/java/jenkins/model/lazy/SortedList.java

          Jose Sa added a comment -

          I'm still getting problems loading jobs after LazyLoading feature introduction. I've added my symptoms in JENKINS-15533

          Jose Sa added a comment - I'm still getting problems loading jobs after LazyLoading feature introduction. I've added my symptoms in JENKINS-15533

          Ben McDonie added a comment -

          I am running version 1.488 and am still having issues with LazyLoading. My stack trace is in JENKINS-15642. It happens near the end of Maven builds that were triggered by SCM changes.

          Ben McDonie added a comment - I am running version 1.488 and am still having issues with LazyLoading. My stack trace is in JENKINS-15642 . It happens near the end of Maven builds that were triggered by SCM changes.

            kohsuke Kohsuke Kawaguchi
            beta Pei-Tang Huang
            Votes:
            5 Vote for this issue
            Watchers:
            25 Start watching this issue

              Created:
              Updated:
              Resolved: