Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-25672

Distributed deadlock with remote classloading

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Blocker Blocker
    • maven-plugin, remoting
    • Jenkins 1.580.1 on EC2 Amazon Linux 2014.09

      There is an apparent distributed deadlock in the Jenkins messaging system.

      Jenkins JVM:

      Thread 72214: (state = BLOCKED)
       - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
       - hudson.remoting.Request.call(hudson.remoting.Channel) @bci=128, line=146 (Interpreted frame)
       - hudson.remoting.Channel.call(hudson.remoting.Callable) @bci=14, line=751 (Interpreted frame)
       - hudson.slaves.SlaveComputer.getLogRecords() @bci=23, line=543 (Interpreted frame)
       - hudson.logging.LogRecorder.getSlaveLogRecords() @bci=66, line=384 (Compiled frame)
       - sun.reflect.GeneratedMethodAccessor328.invoke(java.lang.Object, java.lang.Object[]) @bci=36 (Interpreted frame)
       - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Compiled frame)
       - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=57, line=606 (Compiled frame)
       - org.apache.commons.jexl.util.PropertyExecutor.execute(java.lang.Object) @bci=15, line=125 (Compiled frame)
       - org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(java.lang.Object) @bci=5, line=314 (Compiled frame)
       - org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(java.lang.Object, java.lang.Object, org.apache.commons.jexl.util.introspection.Uberspect) @bci=129, line=185 (Compiled frame)
       - org.apache.commons.jexl.parser.ASTIdentifier.execute(java.lang.Object, org.apache.commons.jexl.JexlContext) @bci=9, line=75 (Compiled frame)
       - org.apache.commons.jexl.parser.ASTReference.execute(java.lang.Object, org.apache.commons.jexl.JexlContext) @bci=32, line=83 (Compiled frame)
       - org.apache.commons.jexl.parser.ASTReference.value(org.apache.commons.jexl.JexlContext) @bci=3, line=57 (Compiled frame)
       - org.apache.commons.jexl.parser.ASTReferenceExpression.value(org.apache.commons.jexl.JexlContext) @bci=9, line=51 (Compiled frame)
       - org.apache.commons.jexl.ExpressionImpl.evaluate(org.apache.commons.jexl.JexlContext) @bci=33, line=80 (Compiled frame)
       - hudson.ExpressionFactory2$JexlExpression.evaluate(org.apache.commons.jelly.JellyContext) @bci=21, line=74 (Compiled frame)
       - org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(org.apache.commons.jelly.JellyContext) @bci=2, line=61 (Compiled frame)
       - org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsIterator(org.apache.commons.jelly.JellyContext) @bci=2, line=94 (Interpreted frame)
       - org.apache.commons.jelly.tags.core.ForEachTag.doTag(org.apache.commons.jelly.XMLOutput) @bci=56, line=89 (Compiled frame)
       - org.apache.commons.jelly.impl.TagScript.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=443, line=269 (Compiled frame)
       - org.apache.commons.jelly.impl.ScriptBlock.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=34, line=95 (Compiled frame)
       - org.kohsuke.stapler.jelly.CallTagLibScript$1.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=33, line=99 (Interpreted frame)
       - org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(org.apache.commons.jelly.XMLOutput) @bci=57, line=91 (Interpreted frame)
       - org.apache.commons.jelly.impl.TagScript.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=443, line=269 (Compiled frame)
       - org.apache.commons.jelly.impl.ScriptBlock.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=34, line=95 (Compiled frame)
       - org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=21, line=98 (Interpreted frame)
       - org.apache.commons.jelly.impl.ScriptBlock.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=34, line=95 (Compiled frame)
       - org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=6, line=105 (Compiled frame)
       - org.kohsuke.stapler.jelly.CallTagLibScript.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=274, line=120 (Compiled frame)
       - org.apache.commons.jelly.impl.ScriptBlock.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=34, line=95 (Compiled frame)
       - org.kohsuke.stapler.jelly.CallTagLibScript$1.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=33, line=99 (Interpreted frame)
       - org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(org.apache.commons.jelly.XMLOutput) @bci=57, line=91 (Interpreted frame)
       - org.apache.commons.jelly.impl.TagScript.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=443, line=269 (Compiled frame)
       - org.apache.commons.jelly.impl.ScriptBlock.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=34, line=95 (Compiled frame)
       - org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=49, line=99 (Compiled frame)
       - org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=49, line=99 (Compiled frame)
       - org.apache.commons.jelly.impl.ScriptBlock.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=34, line=95 (Compiled frame)
       - org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=49, line=99 (Compiled frame)
       - org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=49, line=99 (Compiled frame)
       - org.apache.commons.jelly.impl.ScriptBlock.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=34, line=95 (Compiled frame)
       - org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=49, line=99 (Compiled frame)
       - org.apache.commons.jelly.impl.ScriptBlock.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=34, line=95 (Compiled frame)
       - org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=49, line=99 (Compiled frame)
       - org.apache.commons.jelly.impl.ScriptBlock.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=34, line=95 (Compiled frame)
       - org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=6, line=105 (Compiled frame)
       - org.kohsuke.stapler.jelly.CallTagLibScript.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=274, line=120 (Compiled frame)
       - org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=6, line=105 (Interpreted frame)
       - org.kohsuke.stapler.jelly.JellyViewScript.run(org.apache.commons.jelly.JellyContext, org.apache.commons.jelly.XMLOutput) @bci=120, line=95 (Interpreted frame)
       - org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(org.kohsuke.stapler.StaplerRequest, org.kohsuke.stapler.StaplerResponse, org.apache.commons.jelly.Script, java.lang.Object, org.apache.commons.jelly.XMLOutput) @bci=27, line=63 (Interpreted frame)
       - org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(org.kohsuke.stapler.StaplerRequest, org.kohsuke.stapler.StaplerResponse, org.apache.commons.jelly.Script, java.lang.Object) @bci=19, line=53 (Interpreted frame)
       - org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(org.kohsuke.stapler.StaplerRequest, org.kohsuke.stapler.StaplerResponse, java.lang.Object) @bci=80, line=112 (Interpreted frame)
       - org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(org.kohsuke.stapler.RequestImpl, org.kohsuke.stapler.ResponseImpl, java.lang.Object, org.kohsuke.stapler.MetaClass) @bci=14, line=127 (Interpreted frame)
       - org.kohsuke.stapler.Stapler.tryInvoke(org.kohsuke.stapler.RequestImpl, org.kohsuke.stapler.ResponseImpl, java.lang.Object) @bci=452, line=734 (Compiled frame)
       - org.kohsuke.stapler.Stapler.invoke(org.kohsuke.stapler.RequestImpl, org.kohsuke.stapler.ResponseImpl, java.lang.Object) @bci=119, line=875 (Interpreted frame)
       - org.kohsuke.stapler.MetaClass$12.dispatch(org.kohsuke.stapler.RequestImpl, org.kohsuke.stapler.ResponseImpl, java.lang.Object) @bci=92, line=391 (Interpreted frame)
       - org.kohsuke.stapler.Stapler.tryInvoke(org.kohsuke.stapler.RequestImpl, org.kohsuke.stapler.ResponseImpl, java.lang.Object) @bci=530, line=745 (Compiled frame)
       - org.kohsuke.stapler.Stapler.invoke(org.kohsuke.stapler.RequestImpl, org.kohsuke.stapler.ResponseImpl, java.lang.Object) @bci=119, line=875 (Interpreted frame)
       - org.kohsuke.stapler.MetaClass$4.doDispatch(org.kohsuke.stapler.RequestImpl, org.kohsuke.stapler.ResponseImpl, java.lang.Object) @bci=57, line=211 (Interpreted frame)
       - org.kohsuke.stapler.NameBasedDispatcher.dispatch(org.kohsuke.stapler.RequestImpl, org.kohsuke.stapler.ResponseImpl, java.lang.Object) @bci=57, line=53 (Compiled frame)
       - org.kohsuke.stapler.Stapler.tryInvoke(org.kohsuke.stapler.RequestImpl, org.kohsuke.stapler.ResponseImpl, java.lang.Object) @bci=530, line=745 (Compiled frame)
       - org.kohsuke.stapler.Stapler.invoke(org.kohsuke.stapler.RequestImpl, org.kohsuke.stapler.ResponseImpl, java.lang.Object) @bci=119, line=875 (Interpreted frame)
       - org.kohsuke.stapler.Stapler.invoke(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, java.lang.Object, java.lang.String) @bci=82, line=648 (Interpreted frame)
       - org.kohsuke.stapler.Stapler.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=335, line=237 (Interpreted frame)
       - javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=30, line=848 (Interpreted frame)
       - org.eclipse.jetty.servlet.ServletHolder.handle(org.eclipse.jetty.server.Request, javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=217, line=686 (Interpreted frame)
       - org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=237, line=1494 (Interpreted frame)
       - hudson.util.PluginServletFilter$1.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=18, line=96 (Interpreted frame)
       - jenkins.metrics.impl.MetricsFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=33, line=117 (Interpreted frame)
       - hudson.util.PluginServletFilter$1.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=45, line=99 (Interpreted frame)
       - net.bull.javamelody.MonitoringFilter.doFilter(javax.servlet.FilterChain, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=96, line=198 (Interpreted frame)
       - net.bull.javamelody.MonitoringFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=113, line=176 (Interpreted frame)
       - net.bull.javamelody.PluginMonitoringFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=31, line=85 (Interpreted frame)
       - org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=199, line=97 (Interpreted frame)
       - hudson.util.PluginServletFilter$1.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=45, line=99 (Interpreted frame)
       - hudson.util.PluginServletFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=11, line=88 (Interpreted frame)
       - org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=166, line=1482 (Interpreted frame)
       - hudson.security.csrf.CrumbFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=21, line=48 (Interpreted frame)
       - org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=166, line=1482 (Interpreted frame)
       - hudson.security.ChainedServletFilter$1.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=18, line=84 (Interpreted frame)
       - hudson.security.ChainedServletFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=25, line=76 (Interpreted frame)
       - hudson.security.HudsonFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=41, line=164 (Interpreted frame)
       - org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=166, line=1482 (Interpreted frame)
       - org.kohsuke.stapler.compression.CompressionFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=33, line=46 (Interpreted frame)
       - org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=166, line=1482 (Interpreted frame)
       - hudson.util.CharacterEncodingFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=43, line=81 (Interpreted frame)
       - org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=132, line=1474 (Interpreted frame)
       - org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=197, line=499 (Interpreted frame)
       - org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Interpreted frame)
       - org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=420, line=533 (Interpreted frame)
       - org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Interpreted frame)
       - org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=206, line=1086 (Interpreted frame)
       - org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=277, line=428 (Interpreted frame)
       - org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Interpreted frame)
       - org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=437, line=1020 (Interpreted frame)
       - org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Interpreted frame)
       - org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String, org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled frame)
       - org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection) @bci=76, line=366 (Compiled frame)
       - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest() @bci=328, line=489 (Compiled frame)
       - org.eclipse.jetty.server.AbstractHttpConnection.headerComplete() @bci=472, line=949 (Interpreted frame)
       - org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete() @bci=4, line=1011 (Interpreted frame)
       - org.eclipse.jetty.http.HttpParser.parseNext() @bci=327, line=323 (Compiled frame)
       - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=1, line=235 (Compiled frame)
       - org.eclipse.jetty.server.AsyncHttpConnection.handle() @bci=80, line=82 (Compiled frame)
       - org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle() @bci=10, line=668 (Compiled frame)
       - org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run() @bci=4, line=52 (Interpreted frame)
       - winstone.BoundedExecutorService$1.run() @bci=4, line=77 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1145 (Compiled frame)
       - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
       - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
      

      Slave:

      "pool-1-thread-1 for channel / waiting for hudson.remoting.Channel@7977cdb1:channel" prio=10 tid=0x00007f3af4001800 nid=0xd5b in Object.wait() [0x00007f3b401a8000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              - waiting on <0x00000000eac6bbd8> (a hudson.remoting.RemoteInvocationHandler$RPCRequest)
              at hudson.remoting.Request.call(Request.java:146)
              - locked <0x00000000eac6bbd8> (a hudson.remoting.RemoteInvocationHandler$RPCRequest)
              at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:176)
              at com.sun.proxy.$Proxy2.fetch3(Unknown Source)
              at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:157)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
              - locked <0x00000000c7337918> (a hudson.remoting.RemoteClassLoader)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
              at hudson.maven.Messages.<clinit>(Messages.java:14)
              at hudson.maven.AbstractMavenBuilder.waitForAsynchronousExecutions(AbstractMavenBuilder.java:184)
              at hudson.maven.Maven3Builder.call(Maven3Builder.java:143)
              at hudson.maven.Maven3Builder.call(Maven3Builder.java:71)
              at hudson.remoting.UserRequest.perform(UserRequest.java:121)
              at hudson.remoting.UserRequest.perform(UserRequest.java:49)
              at hudson.remoting.Request$2.run(Request.java:324)
              at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
              at java.util.concurrent.FutureTask.run(FutureTask.java:262)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)
      

            Unassigned Unassigned
            arcivanov Arcadiy Ivanov
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: