Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19244

jenkins random hang during startup - Solaris and Linux

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Hi,

      We are having intermittent issues starting Jenkins on our Solaris machine. About 3 times out of 5 restarts, Jenkins does not go passed the 'INFO Augmented all extensions' line and just hangs. I have attached 2 different jstack dumps hoping that this can help troubleshoot what is going on. It looks as if hudson.Util.resolveSymlink is involved, but no idea what to look for further. If needed i can truss the process but that gives a lot of output so you'll need to tell me a bit more in detail what to look for.

      jmap -help does not seem to indicate a problem:

      Attaching to process ID 20951, please wait...
      Debugger attached successfully.
      Server compiler detected.
      JVM version is 20.2-b06
      
      using thread-local object allocation.
      Parallel GC with 43 thread(s)
      
      Heap Configuration:
         MinHeapFreeRatio = 40
         MaxHeapFreeRatio = 70
         MaxHeapSize      = 2147483648 (2048.0MB)
         NewSize          = 1048576 (1.0MB)
         MaxNewSize       = 4294901760 (4095.9375MB)
         OldSize          = 4194304 (4.0MB)
         NewRatio         = 2
         SurvivorRatio    = 8
         PermSize         = 134217728 (128.0MB)
         MaxPermSize      = 268435456 (256.0MB)
      
      Heap Usage:
      PS Young Generation
      Eden Space:
         capacity = 537919488 (513.0MB)
         used     = 270585256 (258.05020904541016MB)
         free     = 267334232 (254.94979095458984MB)
         50.30218499910529% used
      From Space:
         capacity = 89653248 (85.5MB)
         used     = 51440608 (49.057586669921875MB)
         free     = 38212640 (36.442413330078125MB)
         57.37729435078582% used
      To Space:
         capacity = 89653248 (85.5MB)
         used     = 0 (0.0MB)
         free     = 89653248 (85.5MB)
         0.0% used
      PS Old Generation
         capacity = 1434451968 (1368.0MB)
         used     = 0 (0.0MB)
         free     = 1434451968 (1368.0MB)
         0.0% used
      PS Perm Generation
         capacity = 134217728 (128.0MB)
         used     = 94797032 (90.4054946899414MB)
         free     = 39420696 (37.594505310058594MB)
         70.62929272651672% used
      

        Attachments

          Activity

          heymjo Jorg Heymans created issue -
          Hide
          dantheperson daniel carter added a comment -

          Same problem here! Sysadmins took our Solaris server down for patching, now jenkins wont startup unless i remove all the jobs from the jobs directory.

          Show
          dantheperson daniel carter added a comment - Same problem here! Sysadmins took our Solaris server down for patching, now jenkins wont startup unless i remove all the jobs from the jobs directory.
          Hide
          sstack S Stack added a comment -

          Jenkins 1.511/Solaris

          We see precisely the same problem on one of our Solaris servers. It's the only one running 1.511, the others are on an earlier release.

          A thread dump (kill -3) shows we are always BLOCKED here:

          "UpdateCenter.init" daemon prio=3 tid=0x000000000105f000 nid=0x1a waiting for monitor entry [0xfffffd7ed7d550
          00]
             java.lang.Thread.State: BLOCKED (on object monitor)
                  at hudson.ExtensionList.ensureLoaded(ExtensionList.java:247)
                  - waiting to lock <0xfffffd7fb0eb9650> (a hudson.ExtensionList$Lock)
                  at hudson.ExtensionList.iterator(ExtensionList.java:138)
                  at jenkins.model.Jenkins.getAdministrativeMonitor(Jenkins.java:1748)
                  at hudson.diagnosis.OldDataMonitor.remove(OldDataMonitor.java:88)
                  at hudson.diagnosis.OldDataMonitor.access$000(OldDataMonitor.java:64)
                  at hudson.diagnosis.OldDataMonitor$1.onChange(OldDataMonitor.java:104)
                  at hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:78)
                  at hudson.model.UpdateCenter.save(UpdateCenter.java:514)
                  - locked <0xfffffd7fa95cd218> (a hudson.model.UpdateCenter)
                  at hudson.util.PersistedList.onModified(PersistedList.java:167)
                  at hudson.util.PersistedList.replaceBy(PersistedList.java:79)
                  at hudson.model.UpdateCenter.load(UpdateCenter.java:528)
                  - locked <0xfffffd7fa95cd218> (a hudson.model.UpdateCenter)
                  at hudson.model.UpdateCenter.init(UpdateCenter.java:1538)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                  at java.lang.reflect.Method.invoke(Method.java:597)
                  at hudson.init.InitializerFinder.invoke(InitializerFinder.java:120)
                  at hudson.init.InitializerFinder$TaskImpl.run(InitializerFinder.java:184)
                  at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
                  at jenkins.model.Jenkins$7.runTask(Jenkins.java:888)
                  at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
                  at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                  at java.lang.Thread.run(Thread.java:619)
          

          Troubleshooting

          • We aren't firewall-blocked to updates.jenkins-ci.org (199.193.196.24)
          • Adding -Dhudson.model.Hudson.logStartupPerformance=true is somewhat helpful. We can see one job has loaded before the hang. Perhaps, as the reporter has mentioned, this problem is really related to traversing symlinks and not the UpdateCenter.
          Show
          sstack S Stack added a comment - Jenkins 1.511/Solaris We see precisely the same problem on one of our Solaris servers. It's the only one running 1.511 , the others are on an earlier release. JDK 1.6.0 u21 Our service manifest is based on: http://pauloswald.com/blog/article/29/hudson-solaris-smf-manifest Manifest is actually here: https://gist.github.com/poswald/762379 322 jobs including a dozen or so matrix jobs Many plugins, too many to list here A thread dump (kill -3) shows we are always BLOCKED here: "UpdateCenter.init" daemon prio=3 tid=0x000000000105f000 nid=0x1a waiting for monitor entry [0xfffffd7ed7d550 00] java.lang. Thread .State: BLOCKED (on object monitor) at hudson.ExtensionList.ensureLoaded(ExtensionList.java:247) - waiting to lock <0xfffffd7fb0eb9650> (a hudson.ExtensionList$Lock) at hudson.ExtensionList.iterator(ExtensionList.java:138) at jenkins.model.Jenkins.getAdministrativeMonitor(Jenkins.java:1748) at hudson.diagnosis.OldDataMonitor.remove(OldDataMonitor.java:88) at hudson.diagnosis.OldDataMonitor.access$000(OldDataMonitor.java:64) at hudson.diagnosis.OldDataMonitor$1.onChange(OldDataMonitor.java:104) at hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:78) at hudson.model.UpdateCenter.save(UpdateCenter.java:514) - locked <0xfffffd7fa95cd218> (a hudson.model.UpdateCenter) at hudson.util.PersistedList.onModified(PersistedList.java:167) at hudson.util.PersistedList.replaceBy(PersistedList.java:79) at hudson.model.UpdateCenter.load(UpdateCenter.java:528) - locked <0xfffffd7fa95cd218> (a hudson.model.UpdateCenter) at hudson.model.UpdateCenter.init(UpdateCenter.java:1538) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at hudson.init.InitializerFinder.invoke(InitializerFinder.java:120) at hudson.init.InitializerFinder$TaskImpl.run(InitializerFinder.java:184) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259) at jenkins.model.Jenkins$7.runTask(Jenkins.java:888) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang. Thread .run( Thread .java:619) Troubleshooting We aren't firewall-blocked to updates.jenkins-ci.org (199.193.196.24) Adding -Dhudson.model.Hudson.logStartupPerformance=true is somewhat helpful. We can see one job has loaded before the hang. Perhaps, as the reporter has mentioned, this problem is really related to traversing symlinks and not the UpdateCenter .
          Hide
          dantheperson daniel carter added a comment -

          Ok i have the workaround

          mkdir jobs.bak
          mv jobs/* jobs.bak

          <now start jenkins>

          <wait for it to start>

          mv jobs.bak/* jobs

          <now go into the web-ui and Manage Jenkins -> Reload Configuration from Disk>

          Show
          dantheperson daniel carter added a comment - Ok i have the workaround mkdir jobs.bak mv jobs/* jobs.bak <now start jenkins> <wait for it to start> mv jobs.bak/* jobs <now go into the web-ui and Manage Jenkins -> Reload Configuration from Disk>
          dantheperson daniel carter made changes -
          Field Original Value New Value
          Priority Major [ 3 ] Critical [ 2 ]
          Hide
          dantheperson daniel carter added a comment -

          Bumped to Critical as this makes Jenkins effectively unusable on Solaris.

          Show
          dantheperson daniel carter added a comment - Bumped to Critical as this makes Jenkins effectively unusable on Solaris.
          Hide
          dantheperson daniel carter added a comment -

          Have updated to the latest Jenkins ver. 1.528 here, same problem.

          Show
          dantheperson daniel carter added a comment - Have updated to the latest Jenkins ver. 1.528 here, same problem.
          Hide
          sstack S Stack added a comment - - edited

          Solaris 5.10 (VM)

          We upgraded to Jenkins from 1.511 to 1.518 and continue to see bad startup behavior.
          One problem resolved:
          The BLOCK at UpdateCenter.init was due to old IP's in the server's /etc/hosts. For reasons I won't go into here, this server does not use DNS.

          Old IP's are commented out here:

          ###63.246.20.93 jenkins-ci.org
          ###63.246.20.93 downloads.jenkins-ci.org
          199.193.196.24  jenkins-ci.org
          199.193.196.24  updates.jenkins-ci.org
          199.193.196.24  downloads.jenkins-ci.org
          199.193.196.24  mirrors.jenkins-ci.org
          

          We get further now... but startup still hangs:

          "GitSCM.onLoaded" daemon prio=3 tid=0x0000000001393000 nid=0x1c waiting for monitor entry [0xfffffd7ed7b03000]
             java.lang.Thread.State: BLOCKED (on object monitor)
          

          or

          "PluginImpl.init" daemon prio=3 tid=0x00000000012f9000 nid=0x16 waiting for monitor entry [0xfffffd7ed8159000]
             java.lang.Thread.State: BLOCKED (on object monitor)
          

          Our solution:

          • In one terminal session, tail the Jenkins log (tail -f jenkins.log)
          • In another terminal session, repeat a 'kill -9' until Jenkins starts.
            It now takes about 3 attempts to start which is a slight improvement.

          Note: In the command below, Jenkins runs as a service owned by svc_jenk.

          kill -9 `ptree svc_jenk | grep jdk | grep -v grep | awk '{print $1}'`}
          
          Show
          sstack S Stack added a comment - - edited Solaris 5.10 (VM) We upgraded to Jenkins from 1.511 to 1.518 and continue to see bad startup behavior. One problem resolved: The BLOCK at UpdateCenter.init was due to old IP's in the server's /etc/hosts. For reasons I won't go into here, this server does not use DNS. Old IP's are commented out here: ###63.246.20.93 jenkins-ci.org ###63.246.20.93 downloads.jenkins-ci.org 199.193.196.24 jenkins-ci.org 199.193.196.24 updates.jenkins-ci.org 199.193.196.24 downloads.jenkins-ci.org 199.193.196.24 mirrors.jenkins-ci.org We get further now... but startup still hangs: "GitSCM.onLoaded" daemon prio=3 tid=0x0000000001393000 nid=0x1c waiting for monitor entry [0xfffffd7ed7b03000] java.lang. Thread .State: BLOCKED (on object monitor) or "PluginImpl.init" daemon prio=3 tid=0x00000000012f9000 nid=0x16 waiting for monitor entry [0xfffffd7ed8159000] java.lang. Thread .State: BLOCKED (on object monitor) Our solution: In one terminal session, tail the Jenkins log (tail -f jenkins.log) In another terminal session, repeat a 'kill -9' until Jenkins starts. It now takes about 3 attempts to start which is a slight improvement. Note: In the command below, Jenkins runs as a service owned by svc_jenk . kill -9 `ptree svc_jenk | grep jdk | grep -v grep | awk '{print $1}' `}
          Hide
          sstack S Stack added a comment -

          Adding Solaris to issue summary.

          Show
          sstack S Stack added a comment - Adding Solaris to issue summary.
          sstack S Stack made changes -
          Summary jenkins random hang during startup jenkins random hang during startup - Solaris
          Hide
          dantheperson daniel carter added a comment -

          For me it pretty much always hangs at startup. I have only once seen it startup successfully without having first deleted all the jobs

          Show
          dantheperson daniel carter added a comment - For me it pretty much always hangs at startup. I have only once seen it startup successfully without having first deleted all the jobs
          Hide
          dantheperson daniel carter added a comment -

          Here's the logs up to the point it hangs. Always hangs at the same point. CPU stays busy for a little while after it logs "Augmented all extensions" then goes dead.

          6/09/2013 5:00:10 PM org.apache.catalina.core.AprLifecycleListener init
          INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /opt/jdk1.6.0_37/jre/lib/sparc/
          server:/opt/jdk1.6.0_37/jre/lib/sparc:/opt/jdk1.6.0_37/jre/../lib/sparc:/usr/jdk/packages/lib/sparc:/lib:/usr/lib
          6/09/2013 5:00:11 PM org.apache.coyote.http11.Http11Protocol init
          INFO: Initializing Coyote HTTP/1.1 on http-11080
          6/09/2013 5:00:11 PM org.apache.catalina.startup.Catalina load
          INFO: Initialization processed in 3401 ms
          6/09/2013 5:00:11 PM org.apache.catalina.core.StandardService start
          INFO: Starting service Catalina
          6/09/2013 5:00:11 PM org.apache.catalina.core.StandardEngine start
          INFO: Starting Servlet Engine: Apache Tomcat/6.0.29
          6/09/2013 5:00:11 PM org.apache.catalina.startup.HostConfig deployDescriptor
          INFO: Deploying configuration descriptor host-manager.xml
          6/09/2013 5:00:14 PM org.apache.catalina.startup.HostConfig deployWAR
          INFO: Deploying web application archive hudson.war
          Jenkins home directory: /opt/tst/hudson/.hudson found at: $user.home/.hudson
          6/09/2013 5:00:22 PM org.apache.catalina.startup.HostConfig deployDirectory
          INFO: Deploying web application directory ROOT
          6/09/2013 5:00:23 PM org.apache.coyote.http11.Http11Protocol start
          INFO: Starting Coyote HTTP/1.1 on http-11080
          6/09/2013 5:00:23 PM org.apache.jk.common.ChannelSocket init
          INFO: JK: ajp13 listening on /0.0.0.0:11009
          6/09/2013 5:00:23 PM org.apache.jk.server.JkMain start
          INFO: Jk running ID=0 time=1/148  config=null
          6/09/2013 5:00:23 PM org.apache.catalina.startup.Catalina start
          INFO: Server startup in 12601 ms
          6/09/2013 5:00:27 PM jenkins.InitReactorRunner$1 onAttained
          INFO: Started initialization
          6/09/2013 5:00:29 PM jenkins.InitReactorRunner$1 onAttained
          INFO: Listed all plugins
          6/09/2013 5:00:29 PM jenkins.InitReactorRunner$1 onAttained
          INFO: Prepared all plugins
          6/09/2013 5:00:33 PM jenkins.InitReactorRunner$1 onAttained
          INFO: Started all plugins
          6/09/2013 5:00:33 PM jenkins.InitReactorRunner$1 onAttained
          INFO: Augmented all extensions
          
          Show
          dantheperson daniel carter added a comment - Here's the logs up to the point it hangs. Always hangs at the same point. CPU stays busy for a little while after it logs "Augmented all extensions" then goes dead. 6/09/2013 5:00:10 PM org.apache.catalina.core.AprLifecycleListener init INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /opt/jdk1.6.0_37/jre/lib/sparc/ server:/opt/jdk1.6.0_37/jre/lib/sparc:/opt/jdk1.6.0_37/jre/../lib/sparc:/usr/jdk/packages/lib/sparc:/lib:/usr/lib 6/09/2013 5:00:11 PM org.apache.coyote.http11.Http11Protocol init INFO: Initializing Coyote HTTP/1.1 on http-11080 6/09/2013 5:00:11 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 3401 ms 6/09/2013 5:00:11 PM org.apache.catalina.core.StandardService start INFO: Starting service Catalina 6/09/2013 5:00:11 PM org.apache.catalina.core.StandardEngine start INFO: Starting Servlet Engine: Apache Tomcat/6.0.29 6/09/2013 5:00:11 PM org.apache.catalina.startup.HostConfig deployDescriptor INFO: Deploying configuration descriptor host-manager.xml 6/09/2013 5:00:14 PM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive hudson.war Jenkins home directory: /opt/tst/hudson/.hudson found at: $user.home/.hudson 6/09/2013 5:00:22 PM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory ROOT 6/09/2013 5:00:23 PM org.apache.coyote.http11.Http11Protocol start INFO: Starting Coyote HTTP/1.1 on http-11080 6/09/2013 5:00:23 PM org.apache.jk.common.ChannelSocket init INFO: JK: ajp13 listening on /0.0.0.0:11009 6/09/2013 5:00:23 PM org.apache.jk.server.JkMain start INFO: Jk running ID=0 time=1/148 config= null 6/09/2013 5:00:23 PM org.apache.catalina.startup.Catalina start INFO: Server startup in 12601 ms 6/09/2013 5:00:27 PM jenkins.InitReactorRunner$1 onAttained INFO: Started initialization 6/09/2013 5:00:29 PM jenkins.InitReactorRunner$1 onAttained INFO: Listed all plugins 6/09/2013 5:00:29 PM jenkins.InitReactorRunner$1 onAttained INFO: Prepared all plugins 6/09/2013 5:00:33 PM jenkins.InitReactorRunner$1 onAttained INFO: Started all plugins 6/09/2013 5:00:33 PM jenkins.InitReactorRunner$1 onAttained INFO: Augmented all extensions
          Hide
          heymjo Jorg Heymans added a comment -

          I got some threaddumps indicating it is a problem with the parallel loading of job configurations at startup. So adding -Dhudson.model.Hudson.parallelLoad=false to our weblogic startup script makes the server startup successfully (but a lot more slower ofcourse) each time

          "PluginImpl.init" daemon prio=3 tid=0x01794400 nid=0x6e waiting for monitor entry [0x63efe000]
             java.lang.Thread.State: BLOCKED (on object monitor)
          	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:247)
          	- waiting to lock <0xf1108090> (a hudson.ExtensionList$Lock)
          	at hudson.ExtensionList.get(ExtensionList.java:153)
          	at hudson.PluginManager$PluginUpdateMonitor.getInstance(PluginManager.java:1066)
          	at hudson.maven.PluginImpl.init(PluginImpl.java:54)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at hudson.init.InitializerFinder.invoke(InitializerFinder.java:120)
          	at hudson.init.InitializerFinder$TaskImpl.run(InitializerFinder.java:184)
          	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
          	at jenkins.model.Jenkins$7.runTask(Jenkins.java:888)
          	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
          	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          	at java.lang.Thread.run(Thread.java:662)
          
          "UpdateCenter.init" daemon prio=3 tid=0x01053800 nid=0x65 waiting for monitor entry [0x647fe000]
             java.lang.Thread.State: BLOCKED (on object monitor)
          	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:247)
          	- waiting to lock <0xf1108090> (a hudson.ExtensionList$Lock)
          	at hudson.ExtensionList.iterator(ExtensionList.java:138)
          	at jenkins.model.Jenkins.getAdministrativeMonitor(Jenkins.java:1748)
          	at hudson.diagnosis.OldDataMonitor.remove(OldDataMonitor.java:88)
          	at hudson.diagnosis.OldDataMonitor.access$000(OldDataMonitor.java:64)
          	at hudson.diagnosis.OldDataMonitor$1.onChange(OldDataMonitor.java:104)
          	at hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:78)
          	at hudson.model.UpdateCenter.save(UpdateCenter.java:514)
          	- locked <0xf1208df0> (a hudson.model.UpdateCenter)
          	at hudson.util.PersistedList.onModified(PersistedList.java:167)
          	at hudson.util.PersistedList.replaceBy(PersistedList.java:79)
          	at hudson.model.UpdateCenter.load(UpdateCenter.java:528)
          	- locked <0xf1208df0> (a hudson.model.UpdateCenter)
          	at hudson.model.UpdateCenter.init(UpdateCenter.java:1538)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at hudson.init.InitializerFinder.invoke(InitializerFinder.java:120)
          	at hudson.init.InitializerFinder$TaskImpl.run(InitializerFinder.java:184)
          	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
          	at jenkins.model.Jenkins$7.runTask(Jenkins.java:888)
          	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
          	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          	at java.lang.Thread.run(Thread.java:662)
          
          "[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x00571400 nid=0x44 waiting for monitor entry [0x66d7d000]
             java.lang.Thread.State: BLOCKED (on object monitor)
          	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:247)
          	- waiting to lock <0xf1108090> (a hudson.ExtensionList$Lock)
          	at hudson.ExtensionList.iterator(ExtensionList.java:138)
          	at jenkins.model.Jenkins.cleanUp(Jenkins.java:2647)
          	at hudson.WebAppMain.contextDestroyed(WebAppMain.java:337)
          	at weblogic.servlet.internal.EventsManager$FireContextListenerAction.run(EventsManager.java:482)
          	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
          	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
          	at weblogic.servlet.internal.EventsManager.notifyContextDestroyedEvent(EventsManager.java:200)
          	at weblogic.servlet.internal.WebAppServletContext.destroy(WebAppServletContext.java:3224)
          	- locked <0xf11de8f0> (a weblogic.servlet.internal.WebAppServletContext)
          	at weblogic.servlet.internal.ServletContextManager.destroyContext(ServletContextManager.java:240)
          	at weblogic.servlet.internal.HttpServer.unloadWebApp(HttpServer.java:461)
          	- locked <0xf11deb30> (a weblogic.servlet.internal.HttpServer)
          	at weblogic.servlet.internal.WebAppModule.destroyContexts(WebAppModule.java:1535)
          	at weblogic.servlet.internal.WebAppModule.deactivate(WebAppModule.java:507)
          	at weblogic.application.internal.flow.ModuleStateDriver$2.previous(ModuleStateDriver.java:387)
          	at weblogic.application.utils.StateMachineDriver.previousState(StateMachineDriver.java:223)
          	at weblogic.application.utils.StateMachineDriver.previousState(StateMachineDriver.java:215)
          	at weblogic.application.internal.flow.ModuleStateDriver.deactivate(ModuleStateDriver.java:141)
          	at weblogic.application.internal.flow.ScopedModuleDriver.deactivate(ScopedModuleDriver.java:206)
          	at weblogic.application.internal.flow.ModuleListenerInvoker.deactivate(ModuleListenerInvoker.java:261)
          	at weblogic.application.internal.flow.DeploymentCallbackFlow$2.previous(DeploymentCallbackFlow.java:547)
          	at weblogic.application.utils.StateMachineDriver.previousState(StateMachineDriver.java:223)
          	at weblogic.application.utils.StateMachineDriver.previousState(StateMachineDriver.java:215)
          	at weblogic.application.internal.flow.DeploymentCallbackFlow.deactivate(DeploymentCallbackFlow.java:192)
          	at weblogic.application.internal.flow.DeploymentCallbackFlow.deactivate(DeploymentCallbackFlow.java:184)
          	at weblogic.application.internal.BaseDeployment$2.previous(BaseDeployment.java:642)
          	at weblogic.application.utils.StateMachineDriver.previousState(StateMachineDriver.java:223)
          	at weblogic.application.utils.StateMachineDriver.previousState(StateMachineDriver.java:215)
          	at weblogic.application.internal.BaseDeployment.deactivate(BaseDeployment.java:227)
          	at weblogic.application.internal.EarDeployment.deactivate(EarDeployment.java:58)
          	at weblogic.application.internal.DeploymentStateChecker.deactivate(DeploymentStateChecker.java:198)
          	at weblogic.deploy.internal.targetserver.AppContainerInvoker.deactivate(AppContainerInvoker.java:98)
          	at weblogic.deploy.internal.targetserver.BasicDeployment.deactivate(BasicDeployment.java:263)
          	at weblogic.deploy.internal.targetserver.BasicDeployment.deactivateFromServerLifecycle(BasicDeployment.java:458)
          	at weblogic.management.deploy.internal.DeploymentAdapter$1.doDeactivate(DeploymentAdapter.java:73)
          	at weblogic.management.deploy.internal.DeploymentAdapter.deactivate(DeploymentAdapter.java:215)
          	at weblogic.management.deploy.internal.AppTransition$6.transitionApp(AppTransition.java:66)
          	at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:240)
          	at weblogic.management.deploy.internal.ConfiguredDeployments.deactivate(ConfiguredDeployments.java:198)
          	at weblogic.management.deploy.internal.ConfiguredDeployments.undeploy(ConfiguredDeployments.java:191)
          	at weblogic.management.deploy.internal.DeploymentServerService.shutdownApps(DeploymentServerService.java:195)
          	at weblogic.management.deploy.internal.DeploymentServerService.shutdownHelper(DeploymentServerService.java:127)
          	at weblogic.application.ApplicationShutdownService.halt(ApplicationShutdownService.java:142)
          	at weblogic.t3.srvr.ServerServicesManager.haltInternal(ServerServicesManager.java:504)
          	at weblogic.t3.srvr.ServerServicesManager.halt(ServerServicesManager.java:336)
          	at weblogic.t3.srvr.T3Srvr.shutdown(T3Srvr.java:1117)
          	at weblogic.t3.srvr.T3Srvr.forceShutdown(T3Srvr.java:1022)
          	at weblogic.t3.srvr.ServerRuntime.forceShutdown(ServerRuntime.java:351)
          	at weblogic.server.RemoteLifeCycleOperationsImpl.forceShutdown(RemoteLifeCycleOperationsImpl.java:57)
          	at weblogic.server.ServerLifeCycleRuntime$ShutdownRequest.run(ServerLifeCycleRuntime.java:877)
          	at weblogic.work.ContextWrap.run(ContextWrap.java:41)
          	at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
          	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
          	at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
          
          "ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" daemon prio=3 tid=0x0234b000 nid=0x3e waiting for monitor entry [0x672ff000]
             java.lang.Thread.State: BLOCKED (on object monitor)
          	at weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:94)
          	- waiting to lock <0xf1367198> (a java.lang.String)
          	at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
          	at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
          	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
          	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
          
          "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=3 tid=0x0234a400 nid=0x3d waiting for monitor entry [0x673ff000]
             java.lang.Thread.State: BLOCKED (on object monitor)
          	at weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:94)
          	- waiting to lock <0xf1367198> (a java.lang.String)
          	at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
          	at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
          	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
          	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
          
          "Handling GET /jenkins/tcpSlaveAgentListener/ : [ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x02408c00 nid=0x37 waiting for monitor entry [0x679fb000]
             java.lang.Thread.State: BLOCKED (on object monitor)
          	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:247)
          	- waiting to lock <0xf1108090> (a hudson.ExtensionList$Lock)
          	at hudson.ExtensionList.iterator(ExtensionList.java:138)
          	at hudson.model.DownloadService.generateFragment(DownloadService.java:72)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258)
          	at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
          	at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
          	at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
          	at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
          	at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
          	at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
          	at org.apache.commons.jelly.impl.ExpressionScript.run(ExpressionScript.java:66)
          	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
          	at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
          	at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146)
          	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
          	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
          	at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
          	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
          	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
          	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
          	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
          	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
          	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
          	at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
          	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
          	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
          	at org.kohsuke.stapler.jelly.JellyRequestDispatcher.forward(JellyRequestDispatcher.java:55)
          	at hudson.util.HudsonIsLoading.doDynamic(HudsonIsLoading.java:45)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:288)
          	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:151)
          	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:90)
          	at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:363)
          	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677)
          	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)
          	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:583)
          	at org.kohsuke.stapler.Stapler.service(Stapler.java:214)
          	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
          	at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
          	at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
          	at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
          	at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
          	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
          	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
          	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:208)
          	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:181)
          	at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86)
          	at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84)
          	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
          	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
          	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
          	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
          	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
          	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:162)
          	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
          	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
          	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
          	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
          	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
          	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)
          	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
          	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
          	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
          	at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
          	at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)
          	at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454)
          	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
          	at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
          
          
          Show
          heymjo Jorg Heymans added a comment - I got some threaddumps indicating it is a problem with the parallel loading of job configurations at startup. So adding -Dhudson.model.Hudson.parallelLoad=false to our weblogic startup script makes the server startup successfully (but a lot more slower ofcourse) each time "PluginImpl.init" daemon prio=3 tid=0x01794400 nid=0x6e waiting for monitor entry [0x63efe000] java.lang. Thread .State: BLOCKED (on object monitor) at hudson.ExtensionList.ensureLoaded(ExtensionList.java:247) - waiting to lock <0xf1108090> (a hudson.ExtensionList$Lock) at hudson.ExtensionList.get(ExtensionList.java:153) at hudson.PluginManager$PluginUpdateMonitor.getInstance(PluginManager.java:1066) at hudson.maven.PluginImpl.init(PluginImpl.java:54) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at hudson.init.InitializerFinder.invoke(InitializerFinder.java:120) at hudson.init.InitializerFinder$TaskImpl.run(InitializerFinder.java:184) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259) at jenkins.model.Jenkins$7.runTask(Jenkins.java:888) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang. Thread .run( Thread .java:662) "UpdateCenter.init" daemon prio=3 tid=0x01053800 nid=0x65 waiting for monitor entry [0x647fe000] java.lang. Thread .State: BLOCKED (on object monitor) at hudson.ExtensionList.ensureLoaded(ExtensionList.java:247) - waiting to lock <0xf1108090> (a hudson.ExtensionList$Lock) at hudson.ExtensionList.iterator(ExtensionList.java:138) at jenkins.model.Jenkins.getAdministrativeMonitor(Jenkins.java:1748) at hudson.diagnosis.OldDataMonitor.remove(OldDataMonitor.java:88) at hudson.diagnosis.OldDataMonitor.access$000(OldDataMonitor.java:64) at hudson.diagnosis.OldDataMonitor$1.onChange(OldDataMonitor.java:104) at hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:78) at hudson.model.UpdateCenter.save(UpdateCenter.java:514) - locked <0xf1208df0> (a hudson.model.UpdateCenter) at hudson.util.PersistedList.onModified(PersistedList.java:167) at hudson.util.PersistedList.replaceBy(PersistedList.java:79) at hudson.model.UpdateCenter.load(UpdateCenter.java:528) - locked <0xf1208df0> (a hudson.model.UpdateCenter) at hudson.model.UpdateCenter.init(UpdateCenter.java:1538) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at hudson.init.InitializerFinder.invoke(InitializerFinder.java:120) at hudson.init.InitializerFinder$TaskImpl.run(InitializerFinder.java:184) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259) at jenkins.model.Jenkins$7.runTask(Jenkins.java:888) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang. Thread .run( Thread .java:662) "[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)' " daemon prio=3 tid=0x00571400 nid=0x44 waiting for monitor entry [0x66d7d000] java.lang. Thread .State: BLOCKED (on object monitor) at hudson.ExtensionList.ensureLoaded(ExtensionList.java:247) - waiting to lock <0xf1108090> (a hudson.ExtensionList$Lock) at hudson.ExtensionList.iterator(ExtensionList.java:138) at jenkins.model.Jenkins.cleanUp(Jenkins.java:2647) at hudson.WebAppMain.contextDestroyed(WebAppMain.java:337) at weblogic.servlet.internal.EventsManager$FireContextListenerAction.run(EventsManager.java:482) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service. SecurityManager .runAs( SecurityManager .java:120) at weblogic.servlet.internal.EventsManager.notifyContextDestroyedEvent(EventsManager.java:200) at weblogic.servlet.internal.WebAppServletContext.destroy(WebAppServletContext.java:3224) - locked <0xf11de8f0> (a weblogic.servlet.internal.WebAppServletContext) at weblogic.servlet.internal.ServletContextManager.destroyContext(ServletContextManager.java:240) at weblogic.servlet.internal.HttpServer.unloadWebApp(HttpServer.java:461) - locked <0xf11deb30> (a weblogic.servlet.internal.HttpServer) at weblogic.servlet.internal.WebAppModule.destroyContexts(WebAppModule.java:1535) at weblogic.servlet.internal.WebAppModule.deactivate(WebAppModule.java:507) at weblogic.application.internal.flow.ModuleStateDriver$2.previous(ModuleStateDriver.java:387) at weblogic.application.utils.StateMachineDriver.previousState(StateMachineDriver.java:223) at weblogic.application.utils.StateMachineDriver.previousState(StateMachineDriver.java:215) at weblogic.application.internal.flow.ModuleStateDriver.deactivate(ModuleStateDriver.java:141) at weblogic.application.internal.flow.ScopedModuleDriver.deactivate(ScopedModuleDriver.java:206) at weblogic.application.internal.flow.ModuleListenerInvoker.deactivate(ModuleListenerInvoker.java:261) at weblogic.application.internal.flow.DeploymentCallbackFlow$2.previous(DeploymentCallbackFlow.java:547) at weblogic.application.utils.StateMachineDriver.previousState(StateMachineDriver.java:223) at weblogic.application.utils.StateMachineDriver.previousState(StateMachineDriver.java:215) at weblogic.application.internal.flow.DeploymentCallbackFlow.deactivate(DeploymentCallbackFlow.java:192) at weblogic.application.internal.flow.DeploymentCallbackFlow.deactivate(DeploymentCallbackFlow.java:184) at weblogic.application.internal.BaseDeployment$2.previous(BaseDeployment.java:642) at weblogic.application.utils.StateMachineDriver.previousState(StateMachineDriver.java:223) at weblogic.application.utils.StateMachineDriver.previousState(StateMachineDriver.java:215) at weblogic.application.internal.BaseDeployment.deactivate(BaseDeployment.java:227) at weblogic.application.internal.EarDeployment.deactivate(EarDeployment.java:58) at weblogic.application.internal.DeploymentStateChecker.deactivate(DeploymentStateChecker.java:198) at weblogic.deploy.internal.targetserver.AppContainerInvoker.deactivate(AppContainerInvoker.java:98) at weblogic.deploy.internal.targetserver.BasicDeployment.deactivate(BasicDeployment.java:263) at weblogic.deploy.internal.targetserver.BasicDeployment.deactivateFromServerLifecycle(BasicDeployment.java:458) at weblogic.management.deploy.internal.DeploymentAdapter$1.doDeactivate(DeploymentAdapter.java:73) at weblogic.management.deploy.internal.DeploymentAdapter.deactivate(DeploymentAdapter.java:215) at weblogic.management.deploy.internal.AppTransition$6.transitionApp(AppTransition.java:66) at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:240) at weblogic.management.deploy.internal.ConfiguredDeployments.deactivate(ConfiguredDeployments.java:198) at weblogic.management.deploy.internal.ConfiguredDeployments.undeploy(ConfiguredDeployments.java:191) at weblogic.management.deploy.internal.DeploymentServerService.shutdownApps(DeploymentServerService.java:195) at weblogic.management.deploy.internal.DeploymentServerService.shutdownHelper(DeploymentServerService.java:127) at weblogic.application.ApplicationShutdownService.halt(ApplicationShutdownService.java:142) at weblogic.t3.srvr.ServerServicesManager.haltInternal(ServerServicesManager.java:504) at weblogic.t3.srvr.ServerServicesManager.halt(ServerServicesManager.java:336) at weblogic.t3.srvr.T3Srvr.shutdown(T3Srvr.java:1117) at weblogic.t3.srvr.T3Srvr.forceShutdown(T3Srvr.java:1022) at weblogic.t3.srvr.ServerRuntime.forceShutdown(ServerRuntime.java:351) at weblogic.server.RemoteLifeCycleOperationsImpl.forceShutdown(RemoteLifeCycleOperationsImpl.java:57) at weblogic.server.ServerLifeCycleRuntime$ShutdownRequest.run(ServerLifeCycleRuntime.java:877) at weblogic.work.ContextWrap.run(ContextWrap.java:41) at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207) at weblogic.work.ExecuteThread.run(ExecuteThread.java:176) "ExecuteThread: '3' for queue: 'weblogic.socket.Muxer' " daemon prio=3 tid=0x0234b000 nid=0x3e waiting for monitor entry [0x672ff000] java.lang. Thread .State: BLOCKED (on object monitor) at weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:94) - waiting to lock <0xf1367198> (a java.lang. String ) at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29) at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42) at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117) "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer' " daemon prio=3 tid=0x0234a400 nid=0x3d waiting for monitor entry [0x673ff000] java.lang. Thread .State: BLOCKED (on object monitor) at weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:94) - waiting to lock <0xf1367198> (a java.lang. String ) at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29) at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42) at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117) "Handling GET /jenkins/tcpSlaveAgentListener/ : [ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)' " daemon prio=3 tid=0x02408c00 nid=0x37 waiting for monitor entry [0x679fb000] java.lang. Thread .State: BLOCKED (on object monitor) at hudson.ExtensionList.ensureLoaded(ExtensionList.java:247) - waiting to lock <0xf1108090> (a hudson.ExtensionList$Lock) at hudson.ExtensionList.iterator(ExtensionList.java:138) at hudson.model.DownloadService.generateFragment(DownloadService.java:72) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258) at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104) at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83) at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57) at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51) at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80) at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74) at org.apache.commons.jelly.impl.ExpressionScript.run(ExpressionScript.java:66) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53) at org.kohsuke.stapler.jelly.JellyRequestDispatcher.forward(JellyRequestDispatcher.java:55) at hudson.util.HudsonIsLoading.doDynamic(HudsonIsLoading.java:45) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:288) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:151) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:90) at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:363) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:583) at org.kohsuke.stapler.Stapler.service(Stapler.java:214) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227) at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125) at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300) at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:208) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:181) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:162) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service. SecurityManager .runAs( SecurityManager .java:120) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207) at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
          Hide
          sstack S Stack added a comment -

          @heymjo, your comment was very helpful.
          I added -Dhudson.model.Hudson.parallelLoad=false to our Solaris service manifest and Jenkins started up on the first attempt. It has been a long time since the startup went this smoothly – probably going back to Jenkins 1.496. By the way, the startup did not seem slower at all. We use apache not WebLogic for Jenkins.
          Many thanks for your post.

          Show
          sstack S Stack added a comment - @heymjo, your comment was very helpful. I added -Dhudson.model.Hudson.parallelLoad=false to our Solaris service manifest and Jenkins started up on the first attempt. It has been a long time since the startup went this smoothly – probably going back to Jenkins 1.496 . By the way, the startup did not seem slower at all. We use apache not WebLogic for Jenkins. Many thanks for your post .
          Hide
          heymjo Jorg Heymans added a comment -

          No problem, glad it helped your case as well. It would be nice to still get the problem fixed in Jenkins itself though. I tried pinging Kohsuke on twitter but I realize now that this was probably not the most effective way to draw some attention to this issue.

          Show
          heymjo Jorg Heymans added a comment - No problem, glad it helped your case as well. It would be nice to still get the problem fixed in Jenkins itself though. I tried pinging Kohsuke on twitter but I realize now that this was probably not the most effective way to draw some attention to this issue.
          Hide
          dantheperson daniel carter added a comment -

          I can confirm -Dhudson.model.Hudson.parallelLoad=false fixes the problem here also.

          Possibly drop the priority now that there is an easy work around?

          Show
          dantheperson daniel carter added a comment - I can confirm -Dhudson.model.Hudson.parallelLoad=false fixes the problem here also. Possibly drop the priority now that there is an easy work around?
          Hide
          sstack S Stack added a comment -

          More proof

          • Yesterday, 2 more Solaris Jenkins servers exhibited the same hang on startup. Both were fixed by adding -Dhudson.model.parallelLoad=false to their service manifest.
          • Retracting my initial observation. Startup without parallel job loading is indeed slower.

          @danielcarter, agree this is no longer Critical priority but the workaround does impact startup performance.

          Show
          sstack S Stack added a comment - More proof Yesterday, 2 more Solaris Jenkins servers exhibited the same hang on startup. Both were fixed by adding -Dhudson.model.parallelLoad=false to their service manifest. Retracting my initial observation. Startup without parallel job loading is indeed slower. @danielcarter, agree this is no longer Critical priority but the workaround does impact startup performance.
          dantheperson daniel carter made changes -
          Priority Critical [ 2 ] Major [ 3 ]
          Hide
          mfriedenhagen Mirko Friedenhagen added a comment - - edited

          I have seen the same issue while running integration tests for the jobConfigHistory-plugin, see https://huschteguzzel.de/hudson/job/jenkinsci-jobConfigHistory-plugin/1596/. This is on Linux with JDK 1.7.0_40 on a virtual instance:

          Java stack information for the threads listed above:
          ===================================================
          "Trigger.init":
          	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:246)
          	- waiting to lock <0x00000000f9a01760> (a hudson.DescriptorExtensionList)
          	at hudson.ExtensionList.iterator(ExtensionList.java:138)
          	at hudson.model.User.load(User.java:151)
          	- locked <0x00000000f7e44988> (a hudson.model.User)
          	at hudson.model.User.<init>(User.java:122)
          	at hudson.model.User.getOrCreate(User.java:343)
          	at hudson.model.User.get(User.java:332)
          	at hudson.model.User.get(User.java:300)
          	at hudson.model.User.get(User.java:355)
          	at hudson.model.User.current(User.java:367)
          	at hudson.plugins.jobConfigHistory.PluginUtils.getHistoryDao(PluginUtils.java:78)
          	at hudson.plugins.jobConfigHistory.PluginUtils.getHistoryDao(PluginUtils.java:62)
          	at hudson.plugins.jobConfigHistory.JobConfigHistoryPurger.<init>(JobConfigHistoryPurger.java:45)
          	at hudson.plugins.jobConfigHistory.JobConfigHistoryPurger$$FastClassByGuice$$737d44b4.newInstance(<generated>)
          	at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
          	at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61)
          	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:108)
          	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:87)
          	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:259)
          	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1018)
          	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          	at com.google.inject.Scopes$1$1.get(Scopes.java:59)
          	- locked <0x00000000f95cccc8> (a com.google.inject.Scopes$1$1)
          	at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:429)
          	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
          	at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:965)
          	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1011)
          	at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:961)
          	at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:391)
          	at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:382)
          	at hudson.ExtensionFinder._find(ExtensionFinder.java:151)
          	at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:326)
          	at hudson.ExtensionList.load(ExtensionList.java:295)
          	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:248)
          	- locked <0x00000000f7c4ee30> (a hudson.ExtensionList$Lock)
          	at hudson.ExtensionList.iterator(ExtensionList.java:138)
          	at hudson.triggers.Trigger.init(Trigger.java:293)
          	at sun.reflect.GeneratedMethodAccessor328.invoke(Unknown Source)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:606)
          	at hudson.init.InitializerFinder.invoke(InitializerFinder.java:120)
          	at hudson.init.InitializerFinder$TaskImpl.run(InitializerFinder.java:184)
          	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
          	at jenkins.model.Jenkins$7.runTask(Jenkins.java:893)
          	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
          	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          	at java.lang.Thread.run(Thread.java:724)
          "UpdateCenter.init":
          	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:246)
          	- waiting to lock <0x00000000f7c4ee30> (a hudson.ExtensionList$Lock)
          	at hudson.ExtensionList.getComponents(ExtensionList.java:149)
          	at hudson.DescriptorExtensionList.load(DescriptorExtensionList.java:182)
          	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:248)
          	- locked <0x00000000f9a01760> (a hudson.DescriptorExtensionList)
          	at hudson.ExtensionList.iterator(ExtensionList.java:138)
          	at hudson.model.User.load(User.java:151)
          	- locked <0x00000000f9a01330> (a hudson.model.User)
          	at hudson.model.User.<init>(User.java:122)
          	at hudson.model.User.getOrCreate(User.java:343)
          	at hudson.model.User.get(User.java:332)
          	at hudson.model.User.get(User.java:300)
          	at hudson.model.User.get(User.java:355)
          	at hudson.model.User.current(User.java:367)
          	at hudson.plugins.jobConfigHistory.PluginUtils.getHistoryDao(PluginUtils.java:78)
          	at hudson.plugins.jobConfigHistory.PluginUtils.getHistoryDao(PluginUtils.java:62)
          	at hudson.plugins.jobConfigHistory.JobConfigHistorySaveableListener.getHistoryDao(JobConfigHistorySaveableListener.java:49)
          	at hudson.plugins.jobConfigHistory.JobConfigHistorySaveableListener.onChange(JobConfigHistorySaveableListener.java:28)
          	at hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:78)
          	at hudson.model.UpdateCenter.save(UpdateCenter.java:519)
          	- locked <0x00000000f77b0c90> (a hudson.model.UpdateCenter)
          	at hudson.util.PersistedList.onModified(PersistedList.java:167)
          	at hudson.util.PersistedList.add(PersistedList.java:69)
          	at hudson.model.UpdateCenter.load(UpdateCenter.java:549)
          	- locked <0x00000000f77b0c90> (a hudson.model.UpdateCenter)
          	at hudson.model.UpdateCenter.init(UpdateCenter.java:1546)
          	at sun.reflect.GeneratedMethodAccessor319.invoke(Unknown Source)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:606)
          	at hudson.init.InitializerFinder.invoke(InitializerFinder.java:120)
          	at hudson.init.InitializerFinder$TaskImpl.run(InitializerFinder.java:184)
          	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
          	at jenkins.model.Jenkins$7.runTask(Jenkins.java:893)
          	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
          	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          	at java.lang.Thread.run(Thread.java:724)
          
          Found 1 deadlock.
          

          This is not predictable, every 4th or 5th run of the integration tests just succeeds. Same problem on https://jenkins.ci.cloudbees.com/job/plugins/job/jobConfigHistory-plugin/, only runs on https://buildhive.cloudbees.com/job/jenkinsci/job/jobConfigHistory-plugin/ seem to have no problem.

          Show
          mfriedenhagen Mirko Friedenhagen added a comment - - edited I have seen the same issue while running integration tests for the jobConfigHistory-plugin, see https://huschteguzzel.de/hudson/job/jenkinsci-jobConfigHistory-plugin/1596/ . This is on Linux with JDK 1.7.0_40 on a virtual instance: Java stack information for the threads listed above: =================================================== "Trigger.init" : at hudson.ExtensionList.ensureLoaded(ExtensionList.java:246) - waiting to lock <0x00000000f9a01760> (a hudson.DescriptorExtensionList) at hudson.ExtensionList.iterator(ExtensionList.java:138) at hudson.model.User.load(User.java:151) - locked <0x00000000f7e44988> (a hudson.model.User) at hudson.model.User.<init>(User.java:122) at hudson.model.User.getOrCreate(User.java:343) at hudson.model.User.get(User.java:332) at hudson.model.User.get(User.java:300) at hudson.model.User.get(User.java:355) at hudson.model.User.current(User.java:367) at hudson.plugins.jobConfigHistory.PluginUtils.getHistoryDao(PluginUtils.java:78) at hudson.plugins.jobConfigHistory.PluginUtils.getHistoryDao(PluginUtils.java:62) at hudson.plugins.jobConfigHistory.JobConfigHistoryPurger.<init>(JobConfigHistoryPurger.java:45) at hudson.plugins.jobConfigHistory.JobConfigHistoryPurger$$FastClassByGuice$$737d44b4.newInstance(<generated>) at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61) at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:108) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:87) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:259) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1018) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.Scopes$1$1.get(Scopes.java:59) - locked <0x00000000f95cccc8> (a com.google.inject.Scopes$1$1) at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:429) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:965) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1011) at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:961) at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:391) at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:382) at hudson.ExtensionFinder._find(ExtensionFinder.java:151) at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:326) at hudson.ExtensionList.load(ExtensionList.java:295) at hudson.ExtensionList.ensureLoaded(ExtensionList.java:248) - locked <0x00000000f7c4ee30> (a hudson.ExtensionList$Lock) at hudson.ExtensionList.iterator(ExtensionList.java:138) at hudson.triggers.Trigger.init(Trigger.java:293) at sun.reflect.GeneratedMethodAccessor328.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at hudson.init.InitializerFinder.invoke(InitializerFinder.java:120) at hudson.init.InitializerFinder$TaskImpl.run(InitializerFinder.java:184) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259) at jenkins.model.Jenkins$7.runTask(Jenkins.java:893) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:724) "UpdateCenter.init" : at hudson.ExtensionList.ensureLoaded(ExtensionList.java:246) - waiting to lock <0x00000000f7c4ee30> (a hudson.ExtensionList$Lock) at hudson.ExtensionList.getComponents(ExtensionList.java:149) at hudson.DescriptorExtensionList.load(DescriptorExtensionList.java:182) at hudson.ExtensionList.ensureLoaded(ExtensionList.java:248) - locked <0x00000000f9a01760> (a hudson.DescriptorExtensionList) at hudson.ExtensionList.iterator(ExtensionList.java:138) at hudson.model.User.load(User.java:151) - locked <0x00000000f9a01330> (a hudson.model.User) at hudson.model.User.<init>(User.java:122) at hudson.model.User.getOrCreate(User.java:343) at hudson.model.User.get(User.java:332) at hudson.model.User.get(User.java:300) at hudson.model.User.get(User.java:355) at hudson.model.User.current(User.java:367) at hudson.plugins.jobConfigHistory.PluginUtils.getHistoryDao(PluginUtils.java:78) at hudson.plugins.jobConfigHistory.PluginUtils.getHistoryDao(PluginUtils.java:62) at hudson.plugins.jobConfigHistory.JobConfigHistorySaveableListener.getHistoryDao(JobConfigHistorySaveableListener.java:49) at hudson.plugins.jobConfigHistory.JobConfigHistorySaveableListener.onChange(JobConfigHistorySaveableListener.java:28) at hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:78) at hudson.model.UpdateCenter.save(UpdateCenter.java:519) - locked <0x00000000f77b0c90> (a hudson.model.UpdateCenter) at hudson.util.PersistedList.onModified(PersistedList.java:167) at hudson.util.PersistedList.add(PersistedList.java:69) at hudson.model.UpdateCenter.load(UpdateCenter.java:549) - locked <0x00000000f77b0c90> (a hudson.model.UpdateCenter) at hudson.model.UpdateCenter.init(UpdateCenter.java:1546) at sun.reflect.GeneratedMethodAccessor319.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at hudson.init.InitializerFinder.invoke(InitializerFinder.java:120) at hudson.init.InitializerFinder$TaskImpl.run(InitializerFinder.java:184) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259) at jenkins.model.Jenkins$7.runTask(Jenkins.java:893) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:724) Found 1 deadlock. This is not predictable, every 4th or 5th run of the integration tests just succeeds. Same problem on https://jenkins.ci.cloudbees.com/job/plugins/job/jobConfigHistory-plugin/ , only runs on https://buildhive.cloudbees.com/job/jenkinsci/job/jobConfigHistory-plugin/ seem to have no problem.
          mfriedenhagen Mirko Friedenhagen made changes -
          Summary jenkins random hang during startup - Solaris jenkins random hang during startup - Solaris and Linux
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Mirko Friedenhagen
          Path:
          src/test/java/hudson/plugins/jobConfigHistory/AbstractHudsonTestCaseDeletingInstanceDir.java
          http://jenkins-ci.org/commit/jobConfigHistory-plugin/ae6f95f768644539649aed7bf77cfe04ec19deb5
          Log:
          Possible workaround for https://issues.jenkins-ci.org/browse/JENKINS-19244
          Jenkins random hang during startup - Solaris and Linux

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mirko Friedenhagen Path: src/test/java/hudson/plugins/jobConfigHistory/AbstractHudsonTestCaseDeletingInstanceDir.java http://jenkins-ci.org/commit/jobConfigHistory-plugin/ae6f95f768644539649aed7bf77cfe04ec19deb5 Log: Possible workaround for https://issues.jenkins-ci.org/browse/JENKINS-19244 Jenkins random hang during startup - Solaris and Linux
          Hide
          gcummings Geoff Cummings added a comment -

          Might not be related but...

          Our jenkins server used to take 15 minutes to load jobs on startup after INFO: Augmented all extensions
          Last week it failed to start at all.

          I read the the Downstream build view plugin causes jenkins to load all jobs at startup so i disabled it.

          I then found that Jenkins Javascript Widgets Plugin seems to load all jobs and build on startup so I disabled it

          Jenkins restart is a lot quicker now!

          Show
          gcummings Geoff Cummings added a comment - Might not be related but... Our jenkins server used to take 15 minutes to load jobs on startup after INFO: Augmented all extensions Last week it failed to start at all. I read the the Downstream build view plugin causes jenkins to load all jobs at startup so i disabled it. I then found that Jenkins Javascript Widgets Plugin seems to load all jobs and build on startup so I disabled it Jenkins restart is a lot quicker now!
          Hide
          mfriedenhagen Mirko Friedenhagen added a comment -

          Just for the record: -Dhudson.model.parallelLoad=false fixed this for the integration-tests of jobConfigHistory-plugin.

          Show
          mfriedenhagen Mirko Friedenhagen added a comment - Just for the record: -Dhudson.model.parallelLoad=false fixed this for the integration-tests of jobConfigHistory-plugin.
          Hide
          jglick Jesse Glick added a comment -

          @mfriedenhagen your deadlock is reported as JENKINS-20988; not related I think.

          Show
          jglick Jesse Glick added a comment - @mfriedenhagen your deadlock is reported as JENKINS-20988 ; not related I think.
          jglick Jesse Glick made changes -
          Labels threads
          jglick Jesse Glick made changes -
          Labels threads startup threads
          Hide
          xivo_dev_team XiVO DeV TEAM added a comment -

          We had the same problem: Jenkins randomly (but very often) hangs at startup during the loading of jobs.
          Adding the option -Dhudson.model.parallelLoad=false did not solve the problem.
          Looking at the jtrace, we saw one thread blocking on the plugin "Throttle Concurrent Builds Plugin".
          We fixed the problem by downgrading the plugin "Throttle Concurrent Builds Plugin" from 1.8.1 to 1.8.

          Show
          xivo_dev_team XiVO DeV TEAM added a comment - We had the same problem: Jenkins randomly (but very often) hangs at startup during the loading of jobs. Adding the option -Dhudson.model.parallelLoad=false did not solve the problem. Looking at the jtrace, we saw one thread blocking on the plugin "Throttle Concurrent Builds Plugin". We fixed the problem by downgrading the plugin "Throttle Concurrent Builds Plugin" from 1.8.1 to 1.8.
          Hide
          sylvain_mougenot Sylvain Mougenot added a comment -

          Same problem but jenkins hangs every times.
          Jekins : 1.553, 1.552
          throttle-concurrents : 1.8.1
          #jobs : ~300
          JDK: 1.7 or 1.6

          Thank's for all the comments that helped a lot.

          I followed the advises here :

          • disable throttle-concurrents (touch jenkins/plugins/throttle-concurrents.jpi.disabled)
          • downgrade jenkins to the latest (1.549) I ran successfully (thanks to the backups in /usr/lib/jenkins)
          • move the "jobs" directory so that jenkins startup without jobs (mv /usr/lib/jenkins/jobs /usr/lib/jenkins/jobs_)
          • move back the "jobs" directory after jenkins is up and running (mv /usr/lib/jenkins/jobs_ /usr/lib/jenkins/jobs)
          • go to admin and trigger the "reload config" action
          Show
          sylvain_mougenot Sylvain Mougenot added a comment - Same problem but jenkins hangs every times. Jekins : 1.553, 1.552 throttle-concurrents : 1.8.1 #jobs : ~300 JDK: 1.7 or 1.6 Thank's for all the comments that helped a lot. I followed the advises here : disable throttle-concurrents (touch jenkins/plugins/throttle-concurrents.jpi.disabled) downgrade jenkins to the latest (1.549) I ran successfully (thanks to the backups in /usr/lib/jenkins) move the "jobs" directory so that jenkins startup without jobs (mv /usr/lib/jenkins/jobs /usr/lib/jenkins/jobs_) move back the "jobs" directory after jenkins is up and running (mv /usr/lib/jenkins/jobs_ /usr/lib/jenkins/jobs) go to admin and trigger the "reload config" action
          Hide
          benk Ben Ketteridge added a comment -

          We have the same problem on 1.554. We don't use throttle-concurrents.
          Moving the jobs to a backup directory and back after start-up works.
          Adding -Dhudson.model.Hudson.parallelLoad=false did not.
          Our configuration has 80 maven builds, none of which are matrix jobs.

          Show
          benk Ben Ketteridge added a comment - We have the same problem on 1.554. We don't use throttle-concurrents. Moving the jobs to a backup directory and back after start-up works. Adding -Dhudson.model.Hudson.parallelLoad=false did not. Our configuration has 80 maven builds, none of which are matrix jobs.
          Hide
          mwhidden Michael Whidden added a comment -

          Having the same issue. Jenkins startup on Solaris hangs after 'Augmented all extensions'. Removing all jobs works (103 jobs). I tried adding back the jobs one at a time, and it only seems to be an issue when there are more than 5 or 6 jobs.

          -Dhudson.model.Hudson.parallelLoad=false did solve the problem, so it seems to be an issue with the parallel load when there are more than a small number of jobs.

          Show
          mwhidden Michael Whidden added a comment - Having the same issue. Jenkins startup on Solaris hangs after 'Augmented all extensions'. Removing all jobs works (103 jobs). I tried adding back the jobs one at a time, and it only seems to be an issue when there are more than 5 or 6 jobs. -Dhudson.model.Hudson.parallelLoad=false did solve the problem, so it seems to be an issue with the parallel load when there are more than a small number of jobs.
          Hide
          danielbeck Daniel Beck added a comment -

          Is this still an issue on recent Jenkins versions?

          If so, please post comments with updated information. Be specific, and include at least the following:

          • Which Solaris version
          • Which JRE in which version
          • Which Jenkins version, which plugins in which versions
          • If you tried removing plugins one by one, which did you find to be the culprit?
          • Please post your thread dumps when it hangs. (https://wiki.jenkins-ci.org/display/JENKINS/Obtaining+a+thread+dump)
          • Try all of the workarounds posted in the comments (so far: uninstall/downgrade Throwttle Concurrent Builds, -Dhudson.model.parallelLoad=false) and report their results
          Show
          danielbeck Daniel Beck added a comment - Is this still an issue on recent Jenkins versions? If so, please post comments with updated information. Be specific, and include at least the following: Which Solaris version Which JRE in which version Which Jenkins version, which plugins in which versions If you tried removing plugins one by one, which did you find to be the culprit? Please post your thread dumps when it hangs. ( https://wiki.jenkins-ci.org/display/JENKINS/Obtaining+a+thread+dump ) Try all of the workarounds posted in the comments (so far: uninstall/downgrade Throwttle Concurrent Builds, -Dhudson.model.parallelLoad=false) and report their results
          Hide
          danielbeck Daniel Beck added a comment -

          No response to comment asking for updated and additional information in over a month, so resolving as Cannot Reproduce.

          Please file a new issue when something like this happens to you on recent Jenkins versions, and link back to this one as possibly related. Make sure to follow the advice on: https://wiki.jenkins-ci.org/display/JENKINS/How+to+report+an+issue

          Show
          danielbeck Daniel Beck added a comment - No response to comment asking for updated and additional information in over a month, so resolving as Cannot Reproduce. Please file a new issue when something like this happens to you on recent Jenkins versions, and link back to this one as possibly related. Make sure to follow the advice on: https://wiki.jenkins-ci.org/display/JENKINS/How+to+report+an+issue
          danielbeck Daniel Beck made changes -
          Resolution Cannot Reproduce [ 5 ]
          Status Open [ 1 ] Resolved [ 5 ]
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 150692 ] JNJira + In-Review [ 193632 ]

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            heymjo Jorg Heymans
            Votes:
            8 Vote for this issue
            Watchers:
            16 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: