• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • None
    • Windows Server 2012 R2
      Jenkins 2.102
      Oracle Java HotSpot(TM) 64-Bit Server VM version 25.144-b01

      Recent changes on the affected server:
      Jenkins-Update to 2.102, including all installed plugins
      Update for WIndows 2012 R2

      Jenkins refuses to start due to a thread deadlock during the startup process. This deadlock also affects HTTP request workers; so the server does not respond to HTTP requests anymore.

      The stacks below show the deadlock between thread pool-6-thread-16 (a.k.a. GitSCM.onLoaded) and pool-6-thread-11. The last stack shows how RequestHandlerThread1 is blocked by the lock owned by GitSCM.onLoaded.

      ------------------------------------------------------------------------------
      pool-6-thread-16

      Name: GitSCM.onLoaded
      State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@29986f0 owned by: PreventRefreshFilter.initAutoRefreshFilter
      Total blocked: 91 Total waited: 39

      Stack trace:
      sun.misc.Unsafe.park(Native Method)
      java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
      java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
      java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
      com.google.inject.internal.CycleDetectingLock$CycleDetectingLockFactory$ReentrantCycleDetectingLock.lockOrDetectPotentialLocksCycle(CycleDetectingLock.java:160)
      com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:136)
      hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:424)
      com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
      com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
      com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
      com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
      hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:386)
      hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:377)
      hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:482)
      hudson.ExtensionList.load(ExtensionList.java:366)
      hudson.ExtensionList.ensureLoaded(ExtensionList.java:304)

      • locked hudson.ExtensionList$Lock@5cfdc450
        hudson.ExtensionList.iterator(ExtensionList.java:158)
        jenkins.model.Jenkins.getDescriptorByType(Jenkins.java:1519)
        hudson.plugins.git.GitSCM.onLoaded(GitSCM.java:1859)
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        java.lang.reflect.Method.invoke(Method.java:498)
        hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
        hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
        org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
        jenkins.model.Jenkins$5.runTask(Jenkins.java:1063)
        org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
        org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:748)

      ------------------------------------------------------------------------------
      pool-6-thread-11

      Name: PreventRefreshFilter.initAutoRefreshFilter
      State: BLOCKED on hudson.ExtensionList$Lock@5cfdc450 owned by: GitSCM.onLoaded
      Total blocked: 10 Total waited: 41

      Stack trace:
      hudson.ExtensionList.ensureLoaded(ExtensionList.java:303)
      hudson.ExtensionList.iterator(ExtensionList.java:158)
      hudson.ExtensionList.get(ExtensionList.java:149)
      org.jenkinsci.plugins.scriptsecurity.scripts.ScriptApproval.get(ScriptApproval.java:97)
      org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScript.configuring(SecureGroovyScript.java:126)
      org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScript.readResolve(SecureGroovyScript.java:102)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:498)
      com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callReadResolve(SerializationMethodInvoker.java:66)
      hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:271)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393)
      hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331)
      hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
      com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
      com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
      hudson.util.XStream2.unmarshal(XStream2.java:147)
      hudson.util.XStream2.unmarshal(XStream2.java:118)
      com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
      hudson.XmlFile.unmarshal(XmlFile.java:179)
      hudson.XmlFile.unmarshal(XmlFile.java:162)
      hudson.model.Descriptor.load(Descriptor.java:893)

      • locked hudson.plugins.claim.ClaimConfig@1d46c412
        hudson.plugins.claim.ClaimConfig.<init>(ClaimConfig.java:24)
        hudson.plugins.claim.ClaimConfig$$FastClassByGuice$$74f9760b.newInstance(<generated>)
        com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
        com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61)
        com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105)
        com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
        com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
        com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
        com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
        hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:424)
        com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
        com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
        com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1051)
        jenkins.ProxyInjector.getInstance(ProxyInjector.java:98)
        hudson.init.TaskMethodFinder.lookUp(TaskMethodFinder.java:124)
        hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:102)
        hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
        org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
        jenkins.model.Jenkins$5.runTask(Jenkins.java:1063)
        org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
        org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:748)

      ------------------------------------------------------------------------------
      RequestHandlerThread1

      Name: Handling GET / from 10.10.15.31 : RequestHandlerThread1 HudsonIsLoading/index.jelly
      State: BLOCKED on hudson.ExtensionList$Lock@5cfdc450 owned by: GitSCM.onLoaded
      Total blocked: 1 Total waited: 0

      Stack trace:
      hudson.ExtensionList.ensureLoaded(ExtensionList.java:303)
      hudson.ExtensionList.iterator(ExtensionList.java:158)
      jenkins.JenkinsHttpSessionListener.sessionCreated(JenkinsHttpSessionListener.java:50)
      org.eclipse.jetty.server.session.SessionHandler.newHttpSession(SessionHandler.java:810)
      org.eclipse.jetty.server.Request.getSession(Request.java:1550)
      org.eclipse.jetty.server.Request.getSession(Request.java:1523)
      javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:279)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:498)
      org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258)
      org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
      org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
      org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
      org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
      org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
      hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
      org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
      org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
      org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112)
      org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:140)
      org.kohsuke.stapler.IndexViewDispatcher.dispatch(IndexViewDispatcher.java:30)
      org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      hudson.security.HudsonFilter.doFilter(HudsonFilter.java:169)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
      org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
      org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
      org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
      org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
      org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
      org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
      org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
      org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
      org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
      org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
      org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      org.eclipse.jetty.server.Server.handle(Server.java:564)
      org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
      org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
      org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
      org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
      org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
      org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
      org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
      org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:126)
      org.eclipse.jetty.io.ManagedSelector$$Lambda$16/549293029.run(Unknown Source)
      winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      java.lang.Thread.run(Thread.java:748)

          [JENKINS-49038] Jenkins does not start due to a deadlock

          Fly Cricket added a comment -

          Further analysis shows that the deadlock can also be avoided by disabling the claim plugin

          Fly Cricket added a comment - Further analysis shows that the deadlock can also be avoided by disabling the  claim plugin

          Oleg Nenashev added a comment -

          It seems that calling "configuring()" in injector is generally risky. CC jglick who may point to the root cause or guidelines

          Oleg Nenashev added a comment - It seems that calling "configuring()" in injector is generally risky. CC jglick who may point to the root cause or guidelines

          Jesse Glick added a comment -

          Guice madness, no clue. ClaimConfig$$FastClassByGuice$$74f9760b.newInstance

          Jesse Glick added a comment - Guice madness, no clue. ClaimConfig$$FastClassByGuice$$74f9760b.newInstance

          Fly Cricket added a comment -

          Going back from Claim Plugin version 2.13.1 to 2.12 seems to solve the problem.

          Fly Cricket added a comment - Going back from Claim Plugin version 2.13.1 to 2.12 seems to solve the problem.

          Probably because in the 2.13, I introduced a new injection:

          @Initializer(after = InitMilestone.PLUGINS_STARTED)
          public static synchronized void initAutoRefreshFilter(ClaimConfig config) throws ServletException {

          I could not reproduce locally on my first attempts, can you tell me if the issue is occuring everytime, and if not, the frequence of occurence?

          Arnaud TAMAILLON added a comment - Probably because in the 2.13, I introduced a new injection: @Initializer(after = InitMilestone.PLUGINS_STARTED) public static synchronized void initAutoRefreshFilter(ClaimConfig config) throws ServletException { I could not reproduce locally on my first attempts, can you tell me if the issue is occuring everytime, and if not, the frequence of occurence?

          Fly Cricket added a comment -

          The issue is occuring everytime.

          Fly Cricket added a comment - The issue is occuring everytime.

          flybeetlecricket, can you test https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fclaim-plugin/detail/PR-53/1/artifacts version please, to tell me if situation is better ?

          I'm still not managing to reproduce, but am under the impression that it is probably linked to the actual list of plugins and maybe threads used by the task reactor lib, as on my dev pc, the deadlock does not occur.

          Arnaud TAMAILLON added a comment - flybeetlecricket , can you test https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fclaim-plugin/detail/PR-53/1/artifacts version please, to tell me if situation is better ? I'm still not managing to reproduce, but am under the impression that it is probably linked to the actual list of plugins and maybe threads used by the task reactor lib, as on my dev pc, the deadlock does not occur.

          Fly Cricket added a comment -

          I've uploaded the HPI via the Jenkins GUI. Jenkins could be restarted without a deadlock. A second restart also worked.

           

          Fly Cricket added a comment - I've uploaded the HPI via the Jenkins GUI. Jenkins could be restarted without a deadlock. A second restart also worked.  

          Code changed in jenkins
          User: Arnaud
          Path:
          src/main/java/hudson/plugins/claim/http/PreventRefreshFilter.java
          http://jenkins-ci.org/commit/claim-plugin/16e22544affcff18b8e4b4ae01036ad3482de698
          Log:
          JENKINS-49038 Jenkins does not start due to a deadlock

          the idea is to get the ClaimConfig retrieval out of guice logic.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Arnaud Path: src/main/java/hudson/plugins/claim/http/PreventRefreshFilter.java http://jenkins-ci.org/commit/claim-plugin/16e22544affcff18b8e4b4ae01036ad3482de698 Log: JENKINS-49038 Jenkins does not start due to a deadlock the idea is to get the ClaimConfig retrieval out of guice logic.

          Thanks for your help!

          Fixed in 2.14.1

          Arnaud TAMAILLON added a comment - Thanks for your help! Fixed in 2.14.1

            greybird Arnaud TAMAILLON
            flybeetlecricket Fly Cricket
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: