Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-26249

Removing Docker node with OnceRetentionStrategy can lead to deadlock

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Manual removing of Docker build node with OnceRetentionStrategy causing deadlock with next stacktrace:

      Found one Java-level deadlock:
      =============================
      "Handling GET / from 192.168.10.18 : RequestHandlerThread[#255] View/index.jelly View/sidepanel.jelly":
        waiting to lock monitor 0x00007f4b9ae22bc8 (object 0x000000078713b8f0, a hudson.model.Queue),
        which is held by "jenkins.util.Timer [#3]"
      "jenkins.util.Timer [#3]":
        waiting to lock monitor 0x00007f4b9ae219e8 (object 0x0000000787158e00, a hudson.model.Hudson),
        which is held by "Executor #0 for bfe04df8701e@vm : executing project-release #125"
      "Executor #0 for bfe04df8701e@vm : executing project-release #125":
        waiting to lock monitor 0x00007f4b997e99b8 (object 0x0000000792523c40, a org.jenkinsci.plugins.durabletask.executors.OnceRetentionStrategy),
        which is held by "jenkins.util.Timer [#3]"
      
      Java stack information for the threads listed above:
      ===================================================
      "Handling GET / from 192.168.10.18 : RequestHandlerThread[#255] View/index.jelly View/sidepanel.jelly":
      	at hudson.model.Queue.getItems(Queue.java:693)
      	- waiting to lock <0x000000078713b8f0> (a hudson.model.Queue)
      	at hudson.model.Queue$CachedItemList.get(Queue.java:229)
      	at hudson.model.Queue.getApproximateItemsQuickly(Queue.java:723)
      	at hudson.model.View.getApproximateQueueItemsQuickly(View.java:484)
      	at sun.reflect.GeneratedMethodAccessor3580.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      	at groovy.lang.MetaClassImpl$GetBeanMethodMetaProperty.getProperty(MetaClassImpl.java:3500)
      	at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:61)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:227)
      	at jenkins.widgets.BuildQueueWidget.index.run(index.groovy:5)
      	at org.kohsuke.stapler.jelly.groovy.GroovierJellyScript.run(GroovierJellyScript.java:74)
      	at org.kohsuke.stapler.jelly.groovy.GroovierJellyScript.run(GroovierJellyScript.java:62)
      	at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)
      	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
      	at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
      	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
      	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
      	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      	at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
      	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      	at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
      	at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)
      	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
      	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
      	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
      	at org.kohsuke.stapler.jelly.CompressTag.doTag(CompressTag.java:44)
      	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      	at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
      	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
      	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
      	at org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112)
      	at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:127)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:735)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:813)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
      	at org.eclipse.jetty.server.Server.handle(Server.java:370)
      	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
      	at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
      	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
      	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
      	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
      	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
      	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
      	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
      	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      "jenkins.util.Timer [#3]":
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1703)
      	- waiting to lock <0x0000000787158e00> (a hudson.model.Hudson)
      	at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:65)
      	at hudson.slaves.CloudRetentionStrategy$1.run(CloudRetentionStrategy.java:67)
      	at hudson.model.Queue._withLock(Queue.java:1035)
      	- locked <0x000000078713b8f0> (a hudson.model.Queue)
      	at hudson.model.Queue.withLock(Queue.java:983)
      	at hudson.slaves.CloudRetentionStrategy.check(CloudRetentionStrategy.java:58)
      	- locked <0x0000000792523c40> (a org.jenkinsci.plugins.durabletask.executors.OnceRetentionStrategy)
      	at hudson.slaves.CloudRetentionStrategy.check(CloudRetentionStrategy.java:45)
      	at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:66)
      	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      "Executor #0 for bfe04df8701e@vm : executing project-release #125":
      	at hudson.slaves.CloudRetentionStrategy.check(CloudRetentionStrategy.java:54)
      	- waiting to lock <0x0000000792523c40> (a org.jenkinsci.plugins.durabletask.executors.OnceRetentionStrategy)
      	at hudson.slaves.CloudRetentionStrategy.check(CloudRetentionStrategy.java:45)
      	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:661)
      	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
      	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
      	- locked <0x0000000787158f18> (a java.lang.Object)
      	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1218)
      	at jenkins.model.Jenkins.setNodes(Jenkins.java:1714)
      	at jenkins.model.Jenkins.removeNode(Jenkins.java:1709)
      	- locked <0x0000000787158e00> (a hudson.model.Hudson)
      	at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:65)
      	at org.jenkinsci.plugins.durabletask.executors.OnceRetentionStrategy.done(OnceRetentionStrategy.java:81)
      	at org.jenkinsci.plugins.durabletask.executors.OnceRetentionStrategy.taskCompleted(OnceRetentionStrategy.java:63)
      	at hudson.slaves.SlaveComputer.taskCompleted(SlaveComputer.java:296)
      	at com.nirima.jenkins.plugins.docker.DockerComputer.taskCompleted(DockerComputer.java:34)
      	at hudson.model.queue.WorkUnitContext.synchronizeEnd(WorkUnitContext.java:140)
      	at hudson.model.Executor.run(Executor.java:253)
      
      Found 1 deadlock.
      

        Attachments

          Activity

          3cky Victor Antonovich created issue -
          integer Kanstantsin Shautsou made changes -
          Field Original Value New Value
          Component/s core [ 15593 ]
          integer Kanstantsin Shautsou made changes -
          Resolution Won't Fix [ 2 ]
          Status Open [ 1 ] Resolved [ 5 ]
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 160243 ] JNJira + In-Review [ 196377 ]

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            3cky Victor Antonovich
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: