Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49038

Jenkins does not start due to a deadlock

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • None
    • Windows Server 2012 R2
      Jenkins 2.102
      Oracle Java HotSpot(TM) 64-Bit Server VM version 25.144-b01

      Recent changes on the affected server:
      Jenkins-Update to 2.102, including all installed plugins
      Update for WIndows 2012 R2

      Jenkins refuses to start due to a thread deadlock during the startup process. This deadlock also affects HTTP request workers; so the server does not respond to HTTP requests anymore.

      The stacks below show the deadlock between thread pool-6-thread-16 (a.k.a. GitSCM.onLoaded) and pool-6-thread-11. The last stack shows how RequestHandlerThread1 is blocked by the lock owned by GitSCM.onLoaded.

      ------------------------------------------------------------------------------
      pool-6-thread-16

      Name: GitSCM.onLoaded
      State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@29986f0 owned by: PreventRefreshFilter.initAutoRefreshFilter
      Total blocked: 91 Total waited: 39

      Stack trace:
      sun.misc.Unsafe.park(Native Method)
      java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
      java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
      java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
      com.google.inject.internal.CycleDetectingLock$CycleDetectingLockFactory$ReentrantCycleDetectingLock.lockOrDetectPotentialLocksCycle(CycleDetectingLock.java:160)
      com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:136)
      hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:424)
      com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
      com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
      com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
      com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
      hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:386)
      hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:377)
      hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:482)
      hudson.ExtensionList.load(ExtensionList.java:366)
      hudson.ExtensionList.ensureLoaded(ExtensionList.java:304)

      • locked hudson.ExtensionList$Lock@5cfdc450
        hudson.ExtensionList.iterator(ExtensionList.java:158)
        jenkins.model.Jenkins.getDescriptorByType(Jenkins.java:1519)
        hudson.plugins.git.GitSCM.onLoaded(GitSCM.java:1859)
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        java.lang.reflect.Method.invoke(Method.java:498)
        hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
        hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
        org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
        jenkins.model.Jenkins$5.runTask(Jenkins.java:1063)
        org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
        org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:748)

      ------------------------------------------------------------------------------
      pool-6-thread-11

      Name: PreventRefreshFilter.initAutoRefreshFilter
      State: BLOCKED on hudson.ExtensionList$Lock@5cfdc450 owned by: GitSCM.onLoaded
      Total blocked: 10 Total waited: 41

      Stack trace:
      hudson.ExtensionList.ensureLoaded(ExtensionList.java:303)
      hudson.ExtensionList.iterator(ExtensionList.java:158)
      hudson.ExtensionList.get(ExtensionList.java:149)
      org.jenkinsci.plugins.scriptsecurity.scripts.ScriptApproval.get(ScriptApproval.java:97)
      org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScript.configuring(SecureGroovyScript.java:126)
      org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScript.readResolve(SecureGroovyScript.java:102)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:498)
      com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callReadResolve(SerializationMethodInvoker.java:66)
      hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:271)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393)
      hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331)
      hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
      com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
      com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
      hudson.util.XStream2.unmarshal(XStream2.java:147)
      hudson.util.XStream2.unmarshal(XStream2.java:118)
      com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
      hudson.XmlFile.unmarshal(XmlFile.java:179)
      hudson.XmlFile.unmarshal(XmlFile.java:162)
      hudson.model.Descriptor.load(Descriptor.java:893)

      • locked hudson.plugins.claim.ClaimConfig@1d46c412
        hudson.plugins.claim.ClaimConfig.<init>(ClaimConfig.java:24)
        hudson.plugins.claim.ClaimConfig$$FastClassByGuice$$74f9760b.newInstance(<generated>)
        com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
        com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61)
        com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105)
        com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
        com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
        com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
        com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
        hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:424)
        com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
        com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
        com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1051)
        jenkins.ProxyInjector.getInstance(ProxyInjector.java:98)
        hudson.init.TaskMethodFinder.lookUp(TaskMethodFinder.java:124)
        hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:102)
        hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
        org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
        jenkins.model.Jenkins$5.runTask(Jenkins.java:1063)
        org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
        org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:748)

      ------------------------------------------------------------------------------
      RequestHandlerThread1

      Name: Handling GET / from 10.10.15.31 : RequestHandlerThread1 HudsonIsLoading/index.jelly
      State: BLOCKED on hudson.ExtensionList$Lock@5cfdc450 owned by: GitSCM.onLoaded
      Total blocked: 1 Total waited: 0

      Stack trace:
      hudson.ExtensionList.ensureLoaded(ExtensionList.java:303)
      hudson.ExtensionList.iterator(ExtensionList.java:158)
      jenkins.JenkinsHttpSessionListener.sessionCreated(JenkinsHttpSessionListener.java:50)
      org.eclipse.jetty.server.session.SessionHandler.newHttpSession(SessionHandler.java:810)
      org.eclipse.jetty.server.Request.getSession(Request.java:1550)
      org.eclipse.jetty.server.Request.getSession(Request.java:1523)
      javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:279)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:498)
      org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258)
      org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
      org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
      org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
      org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
      org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
      hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
      org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
      org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
      org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112)
      org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:140)
      org.kohsuke.stapler.IndexViewDispatcher.dispatch(IndexViewDispatcher.java:30)
      org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      hudson.security.HudsonFilter.doFilter(HudsonFilter.java:169)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
      org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
      org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
      org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
      org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
      org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
      org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
      org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
      org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
      org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
      org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
      org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      org.eclipse.jetty.server.Server.handle(Server.java:564)
      org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
      org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
      org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
      org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
      org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
      org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
      org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
      org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:126)
      org.eclipse.jetty.io.ManagedSelector$$Lambda$16/549293029.run(Unknown Source)
      winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      java.lang.Thread.run(Thread.java:748)

            greybird Arnaud TAMAILLON
            flybeetlecricket Fly Cricket
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: