Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-6435

ReactorException: ConcurrentModificationException

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core
    • None

      I updated to 1.356 and now all page views simply display:

      org.jvnet.hudson.reactor.ReactorException: java.util.ConcurrentModificationException
      	at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:246)
      	at hudson.model.Hudson.executeReactor(Hudson.java:683)
      	at hudson.model.Hudson.(Hudson.java:601)
      	at hudson.model.Hudson.(Hudson.java:541)
      	at hudson.WebAppMain$2.run(WebAppMain.java:222)
      Caused by: java.util.ConcurrentModificationException
      	at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
      	at java.util.AbstractList$Itr.next(AbstractList.java:343)
      	at hudson.PluginManager$1$3$2$1.addTo(PluginManager.java:224)
      	at hudson.PluginManager$1$3$2$1.getEdges(PluginManager.java:218)
      	at hudson.PluginManager$1$3$2$1.getEdges(PluginManager.java:214)
      	at hudson.util.CyclicGraphDetector.visit(CyclicGraphDetector.java:38)
      	at hudson.util.CyclicGraphDetector.run(CyclicGraphDetector.java:22)
      	at hudson.PluginManager$1$3$2.run(PluginManager.java:214)
      	at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146)
      	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
      	at hudson.model.Hudson$3.runTask(Hudson.java:665)
      	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
      	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:619)
      

          [JENKINS-6435] ReactorException: ConcurrentModificationException

          Jesse Glick added a comment -

          Downgrading to 1.354 works around the issue. (I had been running 1.355 which would probably work as well.)

          Jesse Glick added a comment - Downgrading to 1.354 works around the issue. (I had been running 1.355 which would probably work as well.)

          Jesse Glick added a comment -

          The log under 1.354 shows some apparently nonfatal exceptions:

          WARNING: Failed to load hudson.plugins.warnings.dashboard.WarningsPriorityGraphPortlet$WarningsGraphDescriptor
          java.lang.InstantiationException: java.lang.reflect.InvocationTargetException
          	at net.java.sezpoz.IndexItem.instance(IndexItem.java:159)
          	at hudson.ExtensionFinder$Sezpoz.findExtensions(ExtensionFinder.java:101)
          	at hudson.ExtensionList.load(ExtensionList.java:209)
          	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:192)
          	at hudson.ExtensionList.iterator(ExtensionList.java:112)
          	at hudson.model.Hudson.getDescriptor(Hudson.java:891)
          	at hudson.model.Hudson.getDescriptorOrDie(Hudson.java:905)
          	at hudson.tasks.BuildWrapper.getDescriptor(BuildWrapper.java:216)
          	at hudson.model.Descriptor.toMap(Descriptor.java:663)
          	at hudson.util.DescribableList.toMap(DescribableList.java:125)
          	at hudson.model.Project.getBuildWrappers(Project.java:120)
          	at hudson.model.Project.updateTransientActions(Project.java:210)
          	at hudson.model.AbstractProject.onLoad(AbstractProject.java:246)
          	at hudson.model.Project.onLoad(Project.java:87)
          	at hudson.model.Items.load(Items.java:107)
          	at hudson.model.Hudson$12.run(Hudson.java:2097)
          	at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146)
          	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
          	at hudson.model.Hudson$3.runTask(Hudson.java:647)
          	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
          	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          	at java.lang.Thread.run(Thread.java:619)
          Caused by: java.lang.reflect.InvocationTargetException
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at net.java.sezpoz.IndexItem.instance(IndexItem.java:151)
          	... 23 more
          Caused by: java.lang.NoClassDefFoundError: hudson/plugins/view/dashboard/DashboardPortlet
          	at java.lang.ClassLoader.defineClass1(Native Method)
          	at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
          	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
          	at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
          	at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
          	at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
          	at hudson.ClassicPluginStrategy$DependencyClassLoader.findClass(ClassicPluginStrategy.java:410)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:300)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
          	at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
          	at hudson.plugins.warnings.dashboard.WarningsPriorityGraphPortlet$WarningsGraphDescriptor.newInstance(WarningsPriorityGraphPortlet.java:69)
          	... 28 more
          Caused by: java.lang.ClassNotFoundException: hudson.plugins.view.dashboard.DashboardPortlet
          	at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
          	at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
          	... 44 more
          

          Jesse Glick added a comment - The log under 1.354 shows some apparently nonfatal exceptions: WARNING: Failed to load hudson.plugins.warnings.dashboard.WarningsPriorityGraphPortlet$WarningsGraphDescriptor java.lang.InstantiationException: java.lang.reflect.InvocationTargetException at net.java.sezpoz.IndexItem.instance(IndexItem.java:159) at hudson.ExtensionFinder$Sezpoz.findExtensions(ExtensionFinder.java:101) at hudson.ExtensionList.load(ExtensionList.java:209) at hudson.ExtensionList.ensureLoaded(ExtensionList.java:192) at hudson.ExtensionList.iterator(ExtensionList.java:112) at hudson.model.Hudson.getDescriptor(Hudson.java:891) at hudson.model.Hudson.getDescriptorOrDie(Hudson.java:905) at hudson.tasks.BuildWrapper.getDescriptor(BuildWrapper.java:216) at hudson.model.Descriptor.toMap(Descriptor.java:663) at hudson.util.DescribableList.toMap(DescribableList.java:125) at hudson.model.Project.getBuildWrappers(Project.java:120) at hudson.model.Project.updateTransientActions(Project.java:210) at hudson.model.AbstractProject.onLoad(AbstractProject.java:246) at hudson.model.Project.onLoad(Project.java:87) at hudson.model.Items.load(Items.java:107) at hudson.model.Hudson$12.run(Hudson.java:2097) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259) at hudson.model.Hudson$3.runTask(Hudson.java:647) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at net.java.sezpoz.IndexItem.instance(IndexItem.java:151) ... 23 more Caused by: java.lang.NoClassDefFoundError: hudson/plugins/view/dashboard/DashboardPortlet at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:621) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) at java.net.URLClassLoader.defineClass(URLClassLoader.java:260) at java.net.URLClassLoader.access$000(URLClassLoader.java:56) at java.net.URLClassLoader$1.run(URLClassLoader.java:195) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:252) at hudson.ClassicPluginStrategy$DependencyClassLoader.findClass(ClassicPluginStrategy.java:410) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:300) at java.lang.ClassLoader.loadClass(ClassLoader.java:252) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) at hudson.plugins.warnings.dashboard.WarningsPriorityGraphPortlet$WarningsGraphDescriptor.newInstance(WarningsPriorityGraphPortlet.java:69) ... 28 more Caused by: java.lang.ClassNotFoundException: hudson.plugins.view.dashboard.DashboardPortlet at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:252) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) ... 44 more

          Jesse Glick added a comment -

          Looks like PluginWrapper.get

          {,Optional}

          Dependencies() fails to make a defensive copy of its fields?

          Jesse Glick added a comment - Looks like PluginWrapper.get {,Optional} Dependencies() fails to make a defensive copy of its fields?

          Alan Harder added a comment -

          "all page views".. meaning this was a startup error, so any page just shows the exception that occurred during startup, correct?
          Not seeing the problem here.. if you try a new install of 1.356 in your environment do you see the issue, or is it something specific in your existing HUDSON_HOME triggering this?

          Alan Harder added a comment - "all page views".. meaning this was a startup error, so any page just shows the exception that occurred during startup, correct? Not seeing the problem here.. if you try a new install of 1.356 in your environment do you see the issue, or is it something specific in your existing HUDSON_HOME triggering this?

          Alan Harder added a comment -

          You must have some plugins with optional dependencies. I guess this hudson.reactor stuff may run different tasks concurrently since you're seeing this error. I think the p.resolvePluginDependencies() call at PluginManager.java:265 is the call modifying the dependencies list and creating the CME.

          Alan Harder added a comment - You must have some plugins with optional dependencies. I guess this hudson.reactor stuff may run different tasks concurrently since you're seeing this error. I think the p.resolvePluginDependencies() call at PluginManager.java:265 is the call modifying the dependencies list and creating the CME.

          Jesse Glick added a comment -

          Yes, this seems to have been a startup error. With a 1.355 installation, I agreed to install updates to all installed plugins as well as the 1.356 update, then agreed to a safe restart. When I visited the master page later after the restart, I just saw this stack trace and nothing else.

          Later today I should be able to shut down the restored 1.354 installation. (For now it is running useful jobs so I can't just experiment on it.) Given the stack trace it seems likely that it is random - a race condition.

          Jesse Glick added a comment - Yes, this seems to have been a startup error. With a 1.355 installation, I agreed to install updates to all installed plugins as well as the 1.356 update, then agreed to a safe restart. When I visited the master page later after the restart, I just saw this stack trace and nothing else. Later today I should be able to shut down the restored 1.354 installation. (For now it is running useful jobs so I can't just experiment on it.) Given the stack trace it seems likely that it is random - a race condition.

          Jesse Glick added a comment -

          Indeed after a restart it looks fine, so probably a race condition.

          (The NCDFE's still show in the log; probably a problem in the release of the warnings plugins.)

          Jesse Glick added a comment - Indeed after a restart it looks fine, so probably a race condition. (The NCDFE's still show in the log; probably a problem in the release of the warnings plugins.)

          Code changed in hudson
          User: : mindless
          Path:
          trunk/hudson/main/core/src/main/java/hudson/PluginManager.java
          trunk/www/changelog.html
          http://jenkins-ci.org/commit/31481
          Log:
          [FIXED JENKINS-6435] move "Checking cyclic dependencies" step to PLUGINS_PREPARED
          to avoid possible ConcurrentModificationException when optional dependencies are
          being processed in PLUGINS_LISTED phase.

          SCM/JIRA link daemon added a comment - Code changed in hudson User: : mindless Path: trunk/hudson/main/core/src/main/java/hudson/PluginManager.java trunk/www/changelog.html http://jenkins-ci.org/commit/31481 Log: [FIXED JENKINS-6435] move "Checking cyclic dependencies" step to PLUGINS_PREPARED to avoid possible ConcurrentModificationException when optional dependencies are being processed in PLUGINS_LISTED phase.

            mindless Alan Harder
            jglick Jesse Glick
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: