Distributed deadlock with remote classloading

This issue is archived. You can view it, but you can't modify it. Learn more

XMLWordPrintable

      There is an apparent distributed deadlock in the Jenkins messaging system.

      Jenkins JVM:

      Thread 72214: (state = BLOCKED)
       - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
       - hudson.remoting.Request.call(hudson.remoting.Channel) @bci=128, line=146 (Interpreted frame)
       - hudson.remoting.Channel.call(hudson.remoting.Callable) @bci=14, line=751 (Interpreted frame)
       - hudson.slaves.SlaveComputer.getLogRecords() @bci=23, line=543 (Interpreted frame)
       - hudson.logging.LogRecorder.getSlaveLogRecords() @bci=66, line=384 (Compiled frame)
       - sun.reflect.GeneratedMethodAccessor328.invoke(java.lang.Object, java.lang.Object[]) @bci=36 (Interpreted frame)
       - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Compiled frame)
       - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=57, line=606 (Compiled frame)
       - org.apache.commons.jexl.util.PropertyExecutor.execute(java.lang.Object) @bci=15, line=125 (Compiled frame)
       - org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(java.lang.Object) @bci=5, line=314 (Compiled frame)
       - org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(java.lang.Object, java.lang.Object, org.apache.commons.jexl.util.introspection.Uberspect) @bci=129, line=185 (Compiled frame)
       - org.apache.commons.jexl.parser.ASTIdentifier.execute(java.lang.Object, org.apache.commons.jexl.JexlContext) @bci=9, line=75 (Compiled frame)
       - org.apache.commons.jexl.parser.ASTReference.execute(java.lang.Object, org.apache.commons.jexl.JexlContext) @bci=32, line=83 (Compiled frame)
       - org.apache.commons.jexl.parser.ASTReference.value(org.apache.commons.jexl.JexlContext) @bci=3, line=57 (Compiled frame)
       - org.apache.commons.jexl.parser.ASTReferenceExpression.value(org.apache.commons.jexl.JexlContext) @bci=9, line=51 (Compiled frame)
       - org.apache.commons.jexl.ExpressionImpl.evaluate(org.apache.commons.jexl.JexlContext) @bci=33, line=80 (Compiled frame)
       - hudson.ExpressionFactory2$JexlExpression.evaluate(org.apache.commons.jelly.JellyContext) @bci=21, line=74 (Compiled frame)
       - org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(org.apache.commons.jelly.JellyContext) @bci=2, line=61 (Compiled frame)
       - org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsIterator(org.apache.commons.jelly.JellyContext) @bci=2, line=94 (Interpreted frame)
       - org.apache.commons.jelly.tags.core.ForEachTag.doTag(org.apache.commons.jelly.XMLOutput) @bci=56, line=89 (Compiled frame)
       - org.apache.commons.jelly.impl.TagScript.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=443, line=269 (Compiled frame)
       - org.apache.commons.jelly.impl.ScriptBlock.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=34, line=95 (Compiled frame)
       - org.kohsuke.stapler.jelly.CallTagLibScript$1.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=33, line=99 (Interpreted frame)
       - org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(org.apache.commons.jelly.XMLOutput) @bci=57, line=91 (Interpreted frame)
       - org.apache.commons.jelly.impl.TagScript.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=443, line=269 (Compiled frame)
       - org.apache.commons.jelly.impl.ScriptBlock.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=34, line=95 (Compiled frame)
       - org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=21, line=98 (Interpreted frame)
       - org.apache.commons.jelly.impl.ScriptBlock.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=34, line=95 (Compiled frame)
       - org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=6, line=105 (Compiled frame)
       - org.kohsuke.stapler.jelly.CallTagLibScript.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=274, line=120 (Compiled frame)
       - org.apache.commons.jelly.impl.ScriptBlock.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=34, line=95 (Compiled frame)
       - org.kohsuke.stapler.jelly.CallTagLibScript$1.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=33, line=99 (Interpreted frame)
       - org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(org.apache.commons.jelly.XMLOutput) @bci=57, line=91 (Interpreted frame)
       - org.apache.commons.jelly.impl.TagScript.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=443, line=269 (Compiled frame)
       - org.apache.commons.jelly.impl.ScriptBlock.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=34, line=95 (Compiled frame)
       - org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=49, line=99 (Compiled frame)
       - org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=49, line=99 (Compiled frame)
       - org.apache.commons.jelly.impl.ScriptBlock.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=34, line=95 (Compiled frame)
       - org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=49, line=99 (Compiled frame)
       - org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=49, line=99 (Compiled frame)
       - org.apache.commons.jelly.impl.ScriptBlock.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=34, line=95 (Compiled frame)
       - org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=49, line=99 (Compiled frame)
       - org.apache.commons.jelly.impl.ScriptBlock.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=34, line=95 (Compiled frame)
       - org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=49, line=99 (Compiled frame)
       - org.apache.commons.jelly.impl.ScriptBlock.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=34, line=95 (Compiled frame)
       - org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=6, line=105 (Compiled frame)
       - org.kohsuke.stapler.jelly.CallTagLibScript.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=274, line=120 (Compiled frame)
       - org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=6, line=105 (Interpreted frame)
       - org.kohsuke.stapler.jelly.JellyViewScript.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=120, line=95 (Interpreted frame)
       - org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(org.kohsuke.stapler.StaplerRequest, org.kohsuke.stapler.StaplerResponse, org.apache.commons.jelly.Script, java.lang.Object, org.apache.commons.jelly.XMLOutput) @bci=27, line=63 (Interpreted frame)
       - org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(org.kohsuke.stapler.StaplerRequest, org.kohsuke.stapler.StaplerResponse, org.apache.commons.jelly.Script, java.lang.Object) @bci=19, line=53 (Interpreted frame)
       - org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(org.kohsuke.stapler.StaplerRequest, org.kohsuke.stapler.StaplerResponse, java.lang.Object) @bci=80, line=112 (Interpreted frame)
       - org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(org.kohsuke.stapler.RequestImpl, org.kohsuke.stapler.ResponseImpl, java.lang.Object, org.kohsuke.stapler.MetaClass) @bci=14, line=127 (Interpreted frame)
       - org.kohsuke.stapler.Stapler.tryInvoke(org.kohsuke.stapler.RequestImpl, org.kohsuke.stapler.ResponseImpl, java.lang.Object) @bci=452, line=734 (Compiled frame)
       - org.kohsuke.stapler.Stapler.invoke(org.kohsuke.stapler.RequestImpl, org.kohsuke.stapler.ResponseImpl, java.lang.Object) @bci=119, line=875 (Interpreted frame)
       - org.kohsuke.stapler.MetaClass$12.dispatch(org.kohsuke.stapler.RequestImpl, org.kohsuke.stapler.ResponseImpl, java.lang.Object) @bci=92, line=391 (Interpreted frame)
       - org.kohsuke.stapler.Stapler.tryInvoke(org.kohsuke.stapler.RequestImpl, org.kohsuke.stapler.ResponseImpl, java.lang.Object) @bci=530, line=745 (Compiled frame)
       - org.kohsuke.stapler.Stapler.invoke(org.kohsuke.stapler.RequestImpl, org.kohsuke.stapler.ResponseImpl, java.lang.Object) @bci=119, line=875 (Interpreted frame)
       - org.kohsuke.stapler.MetaClass$4.doDispatch(org.kohsuke.stapler.RequestImpl, org.kohsuke.stapler.ResponseImpl, java.lang.Object) @bci=57, line=211 (Interpreted frame)
       - org.kohsuke.stapler.NameBasedDispatcher.dispatch(org.kohsuke.stapler.RequestImpl, org.kohsuke.stapler.ResponseImpl, java.lang.Object) @bci=57, line=53 (Compiled frame)
       - org.kohsuke.stapler.Stapler.tryInvoke(org.kohsuke.stapler.RequestImpl, org.kohsuke.stapler.ResponseImpl, java.lang.Object) @bci=530, line=745 (Compiled frame)
       - org.kohsuke.stapler.Stapler.invoke(org.kohsuke.stapler.RequestImpl, org.kohsuke.stapler.ResponseImpl, java.lang.Object) @bci=119, line=875 (Interpreted frame)
       - org.kohsuke.stapler.Stapler.invoke(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, java.lang.Object, java.lang.String) @bci=82, line=648 (Interpreted frame)
       - org.kohsuke.stapler.Stapler.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=335, line=237 (Interpreted frame)
       - javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=30, line=848 (Interpreted frame)
       - org.eclipse.jetty.servlet.ServletHolder.handle(org.eclipse.jetty.server.Request, javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=217, line=686 (Interpreted frame)
       - org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=237, line=1494 (Interpreted frame)
       - hudson.util.PluginServletFilter$1.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=18, line=96 (Interpreted frame)
       - jenkins.metrics.impl.MetricsFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=33, line=117 (Interpreted frame)
       - hudson.util.PluginServletFilter$1.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=45, line=99 (Interpreted frame)
       - net.bull.javamelody.MonitoringFilter.doFilter(javax.servlet.FilterChain, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=96, line=198 (Interpreted frame)
       - net.bull.javamelody.MonitoringFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=113, line=176 (Interpreted frame)
       - net.bull.javamelody.PluginMonitoringFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=31, line=85 (Interpreted frame)
       - org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=199, line=97 (Interpreted frame)
       - hudson.util.PluginServletFilter$1.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=45, line=99 (Interpreted frame)
       - hudson.util.PluginServletFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=11, line=88 (Interpreted frame)
       - org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=166, line=1482 (Interpreted frame)
       - hudson.security.csrf.CrumbFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=21, line=48 (Interpreted frame)
       - org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=166, line=1482 (Interpreted frame)
       - hudson.security.ChainedServletFilter$1.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=18, line=84 (Interpreted frame)
       - hudson.security.ChainedServletFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=25, line=76 (Interpreted frame)
       - hudson.security.HudsonFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=41, line=164 (Interpreted frame)
       - org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=166, line=1482 (Interpreted frame)
       - org.kohsuke.stapler.compression.CompressionFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=33, line=46 (Interpreted frame)
       - org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=166, line=1482 (Interpreted frame)
       - hudson.util.CharacterEncodingFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=43, line=81 (Interpreted frame)
       - org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=132, line=1474 (Interpreted frame)
       - org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=197, line=499 (Interpreted frame)
       - org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Interpreted frame)
       - org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=420, line=533 (Interpreted frame)
       - org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Interpreted frame)
       - org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=206, line=1086 (Interpreted frame)
       - org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=277, line=428 (Interpreted frame)
       - org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Interpreted frame)
       - org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=437, line=1020 (Interpreted frame)
       - org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Interpreted frame)
       - org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled frame)
       - org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection) @bci=76, line=366 (Compiled frame)
       - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest() @bci=328, line=489 (Compiled frame)
       - org.eclipse.jetty.server.AbstractHttpConnection.headerComplete() @bci=472, line=949 (Interpreted frame)
       - org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete() @bci=4, line=1011 (Interpreted frame)
       - org.eclipse.jetty.http.HttpParser.parseNext() @bci=327, line=323 (Compiled frame)
       - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=1, line=235 (Compiled frame)
       - org.eclipse.jetty.server.AsyncHttpConnection.handle() @bci=80, line=82 (Compiled frame)
       - org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle() @bci=10, line=668 (Compiled frame)
       - org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run() @bci=4, line=52 (Interpreted frame)
       - winstone.BoundedExecutorService$1.run() @bci=4, line=77 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1145 (Compiled frame)
       - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
       - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
      

      Slave:

      "pool-1-thread-1 for channel / waiting for hudson.remoting.Channel@7977cdb1:channel" prio=10 tid=0x00007f3af4001800 nid=0xd5b in Object.wait() [0x00007f3b401a8000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              - waiting on <0x00000000eac6bbd8> (a hudson.remoting.RemoteInvocationHandler$RPCRequest)
              at hudson.remoting.Request.call(Request.java:146)
              - locked <0x00000000eac6bbd8> (a hudson.remoting.RemoteInvocationHandler$RPCRequest)
              at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:176)
              at com.sun.proxy.$Proxy2.fetch3(Unknown Source)
              at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:157)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
              - locked <0x00000000c7337918> (a hudson.remoting.RemoteClassLoader)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
              at hudson.maven.Messages.<clinit>(Messages.java:14)
              at hudson.maven.AbstractMavenBuilder.waitForAsynchronousExecutions(AbstractMavenBuilder.java:184)
              at hudson.maven.Maven3Builder.call(Maven3Builder.java:143)
              at hudson.maven.Maven3Builder.call(Maven3Builder.java:71)
              at hudson.remoting.UserRequest.perform(UserRequest.java:121)
              at hudson.remoting.UserRequest.perform(UserRequest.java:49)
              at hudson.remoting.Request$2.run(Request.java:324)
              at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
              at java.util.concurrent.FutureTask.run(FutureTask.java:262)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)
      

            Assignee:
            Unassigned
            Reporter:
            Arcadiy Ivanov
            Archiver:
            Jenkins Service Account

              Created:
              Updated:
              Archived: