Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-24008

Jobs stuck while waiting for an executor (and executors are available)

XMLWordPrintable

      I have some similar trace as JENKINS-10567 (may or may not be related)

      there are about 10 items queued in /job/runtest/job/DevPrecheckin/:

      • they are all waiting for the same slave (job tied to the slave)
      • the slave has 8 free executors
      • all 10 items are stuck.

      after removing the first few queued item, was able to recover.
      I am using the LTS version 1.554.3, and can provide more details upon request (mostly because of sensitive info I would need to scrub).

      Jul 28, 2014 12:09:41 PM WARNING hudson.ExpressionFactory2$JexlExpression evaluate
      Caught exception evaluating: item.getCauseOfBlockage() in /job/runtest/job/DevPrecheckin/buildHistory/ajax. Reason: java.lang.NullPointerException
      java.lang.NullPointerException
      
      Jul 28, 2014 12:09:41 PM WARNING hudson.ExpressionFactory2$JexlExpression evaluate
      Caught exception evaluating: item.params in /job/runtest/job/DevPrecheckin/buildHistory/ajax. Reason: java.lang.reflect.InvocationTargetException
      java.lang.reflect.InvocationTargetException
      	at sun.reflect.GeneratedMethodAccessor88.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:622)
      	at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
      	at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
      	at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
      	at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
      	at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
      	at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
      	at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
      	at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
      	at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
      	at org.apache.commons.jelly.parser.EscapingExpression.evaluate(EscapingExpression.java:24)
      	at org.apache.commons.jelly.impl.ExpressionScript.run(ExpressionScript.java:66)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
      	at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
      	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      	at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      	at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
      	at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)
      	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      	at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
      	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
      	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
      	at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41)
      	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
      	at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
      	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
      	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      	at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
      	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
      	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
      	at org.kohsuke.stapler.jelly.JellyRequestDispatcher.forward(JellyRequestDispatcher.java:55)
      	at hudson.widgets.HistoryWidget.doAjax(HistoryWidget.java:195)
      	at sun.reflect.GeneratedMethodAccessor483.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:622)
      	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
      	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
      	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
      	at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:390)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
      	at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
      	at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:631)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:225)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
      	at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:79)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
      	at org.eclipse.jetty.server.Server.handle(Server.java:370)
      	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
      	at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
      	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
      	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
      	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
      	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
      	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
      	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
      	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:701)
      Caused by: java.lang.NullPointerException
      
      Jul 28, 2014 12:09:41 PM WARNING hudson.ExpressionFactory2$JexlExpression evaluate
      Caught exception evaluating: item.getCauseOfBlockage() in /job/runtest/job/DevPrecheckin/buildHistory/ajax. Reason: java.lang.NullPointerException
      java.lang.NullPointerException
      
      Jul 28, 2014 12:09:46 PM SEVERE hudson.triggers.SafeTimerTask run
      Timer task hudson.model.Queue$MaintainTask@34a07387 failed
      java.lang.NullPointerException
      
      Jul 28, 2014 12:09:50 PM SEVERE hudson.triggers.SafeTimerTask run
      Timer task hudson.slaves.NodeProvisioner$NodeProvisionerInvoker@1189caf7 failed
      java.lang.NullPointerException
      

            Unassigned Unassigned
            dnozay Damien Nozay
            Votes:
            3 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: