Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49038

Jenkins does not start due to a deadlock

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Major
    • Resolution: Fixed
    • claim-plugin, script-security-plugin
    • None
    • Windows Server 2012 R2
      Jenkins 2.102
      Oracle Java HotSpot(TM) 64-Bit Server VM version 25.144-b01

    Description

      Recent changes on the affected server:
      Jenkins-Update to 2.102, including all installed plugins
      Update for WIndows 2012 R2

      Jenkins refuses to start due to a thread deadlock during the startup process. This deadlock also affects HTTP request workers; so the server does not respond to HTTP requests anymore.

      The stacks below show the deadlock between thread pool-6-thread-16 (a.k.a. GitSCM.onLoaded) and pool-6-thread-11. The last stack shows how RequestHandlerThread1 is blocked by the lock owned by GitSCM.onLoaded.

      ------------------------------------------------------------------------------
      pool-6-thread-16

      Name: GitSCM.onLoaded
      State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@29986f0 owned by: PreventRefreshFilter.initAutoRefreshFilter
      Total blocked: 91 Total waited: 39

      Stack trace:
      sun.misc.Unsafe.park(Native Method)
      java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
      java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
      java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
      com.google.inject.internal.CycleDetectingLock$CycleDetectingLockFactory$ReentrantCycleDetectingLock.lockOrDetectPotentialLocksCycle(CycleDetectingLock.java:160)
      com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:136)
      hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:424)
      com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
      com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
      com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
      com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
      hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:386)
      hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:377)
      hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:482)
      hudson.ExtensionList.load(ExtensionList.java:366)
      hudson.ExtensionList.ensureLoaded(ExtensionList.java:304)

      • locked hudson.ExtensionList$Lock@5cfdc450
        hudson.ExtensionList.iterator(ExtensionList.java:158)
        jenkins.model.Jenkins.getDescriptorByType(Jenkins.java:1519)
        hudson.plugins.git.GitSCM.onLoaded(GitSCM.java:1859)
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        java.lang.reflect.Method.invoke(Method.java:498)
        hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
        hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
        org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
        jenkins.model.Jenkins$5.runTask(Jenkins.java:1063)
        org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
        org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:748)

      ------------------------------------------------------------------------------
      pool-6-thread-11

      Name: PreventRefreshFilter.initAutoRefreshFilter
      State: BLOCKED on hudson.ExtensionList$Lock@5cfdc450 owned by: GitSCM.onLoaded
      Total blocked: 10 Total waited: 41

      Stack trace:
      hudson.ExtensionList.ensureLoaded(ExtensionList.java:303)
      hudson.ExtensionList.iterator(ExtensionList.java:158)
      hudson.ExtensionList.get(ExtensionList.java:149)
      org.jenkinsci.plugins.scriptsecurity.scripts.ScriptApproval.get(ScriptApproval.java:97)
      org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScript.configuring(SecureGroovyScript.java:126)
      org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScript.readResolve(SecureGroovyScript.java:102)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:498)
      com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callReadResolve(SerializationMethodInvoker.java:66)
      hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:271)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393)
      hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331)
      hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
      com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
      com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
      hudson.util.XStream2.unmarshal(XStream2.java:147)
      hudson.util.XStream2.unmarshal(XStream2.java:118)
      com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
      hudson.XmlFile.unmarshal(XmlFile.java:179)
      hudson.XmlFile.unmarshal(XmlFile.java:162)
      hudson.model.Descriptor.load(Descriptor.java:893)

      • locked hudson.plugins.claim.ClaimConfig@1d46c412
        hudson.plugins.claim.ClaimConfig.<init>(ClaimConfig.java:24)
        hudson.plugins.claim.ClaimConfig$$FastClassByGuice$$74f9760b.newInstance(<generated>)
        com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
        com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61)
        com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105)
        com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
        com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
        com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
        com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
        hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:424)
        com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
        com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
        com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1051)
        jenkins.ProxyInjector.getInstance(ProxyInjector.java:98)
        hudson.init.TaskMethodFinder.lookUp(TaskMethodFinder.java:124)
        hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:102)
        hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
        org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
        jenkins.model.Jenkins$5.runTask(Jenkins.java:1063)
        org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
        org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:748)

      ------------------------------------------------------------------------------
      RequestHandlerThread1

      Name: Handling GET / from 10.10.15.31 : RequestHandlerThread1 HudsonIsLoading/index.jelly
      State: BLOCKED on hudson.ExtensionList$Lock@5cfdc450 owned by: GitSCM.onLoaded
      Total blocked: 1 Total waited: 0

      Stack trace:
      hudson.ExtensionList.ensureLoaded(ExtensionList.java:303)
      hudson.ExtensionList.iterator(ExtensionList.java:158)
      jenkins.JenkinsHttpSessionListener.sessionCreated(JenkinsHttpSessionListener.java:50)
      org.eclipse.jetty.server.session.SessionHandler.newHttpSession(SessionHandler.java:810)
      org.eclipse.jetty.server.Request.getSession(Request.java:1550)
      org.eclipse.jetty.server.Request.getSession(Request.java:1523)
      javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:279)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:498)
      org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258)
      org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
      org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
      org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
      org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
      org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
      hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
      org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
      org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
      org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112)
      org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:140)
      org.kohsuke.stapler.IndexViewDispatcher.dispatch(IndexViewDispatcher.java:30)
      org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      hudson.security.HudsonFilter.doFilter(HudsonFilter.java:169)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
      org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
      org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
      org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
      org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
      org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
      org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
      org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
      org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
      org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
      org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
      org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      org.eclipse.jetty.server.Server.handle(Server.java:564)
      org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
      org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
      org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
      org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
      org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
      org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
      org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
      org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:126)
      org.eclipse.jetty.io.ManagedSelector$$Lambda$16/549293029.run(Unknown Source)
      winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      java.lang.Thread.run(Thread.java:748)

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-55361 DescriptorExtensionList not locking correctly, leading to deadlocks

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-54974 Jenkins does not start due to a deadlock after upgrade from 2.121.2.2 to 2.138.2.2

          • Major - Major loss of function.
          • Resolved

          Activity

            Ascending order - Click to sort in descending order
            View 6 older comments
            flybeetlecricket Fly Cricket added a comment -

            The issue is occuring everytime.

            flybeetlecricket Fly Cricket added a comment - The issue is occuring everytime.
            greybird Arnaud TAMAILLON added a comment -

            flybeetlecricket, can you test https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fclaim-plugin/detail/PR-53/1/artifacts version please, to tell me if situation is better ?

            I'm still not managing to reproduce, but am under the impression that it is probably linked to the actual list of plugins and maybe threads used by the task reactor lib, as on my dev pc, the deadlock does not occur.

            greybird Arnaud TAMAILLON added a comment - flybeetlecricket , can you test https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fclaim-plugin/detail/PR-53/1/artifacts version please, to tell me if situation is better ? I'm still not managing to reproduce, but am under the impression that it is probably linked to the actual list of plugins and maybe threads used by the task reactor lib, as on my dev pc, the deadlock does not occur.
            flybeetlecricket Fly Cricket added a comment -

            I've uploaded the HPI via the Jenkins GUI. Jenkins could be restarted without a deadlock. A second restart also worked.

             

            flybeetlecricket Fly Cricket added a comment - I've uploaded the HPI via the Jenkins GUI. Jenkins could be restarted without a deadlock. A second restart also worked.  
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Arnaud
            Path:
            src/main/java/hudson/plugins/claim/http/PreventRefreshFilter.java
            http://jenkins-ci.org/commit/claim-plugin/16e22544affcff18b8e4b4ae01036ad3482de698
            Log:
            JENKINS-49038 Jenkins does not start due to a deadlock

            the idea is to get the ClaimConfig retrieval out of guice logic.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Arnaud Path: src/main/java/hudson/plugins/claim/http/PreventRefreshFilter.java http://jenkins-ci.org/commit/claim-plugin/16e22544affcff18b8e4b4ae01036ad3482de698 Log: JENKINS-49038 Jenkins does not start due to a deadlock the idea is to get the ClaimConfig retrieval out of guice logic.
            greybird Arnaud TAMAILLON added a comment -

            Thanks for your help!

            Fixed in 2.14.1

            greybird Arnaud TAMAILLON added a comment - Thanks for your help! Fixed in 2.14.1

            People

              greybird Arnaud TAMAILLON
              flybeetlecricket Fly Cricket
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: