-
Improvement
-
Resolution: Unresolved
-
Minor
-
None
When init script fail to complete in time, slave is silently not provisioned. There is no indication in logs as well. Manual provisioning shows following:
javax.servlet.ServletException: java.lang.RuntimeException: org.jclouds.compute.RunNodesException: error running 1 node group(slave-name) location(LOCATION) image(ca365c26-dbda-40a1-9def-7deb0fb885b2) size(16) options({loginUser=root, loginPasswordPresent=true, loginPrivateKeyPresent=true, scriptPresent=true, userMetadata={Name=slave-name}, keyPairName=key-pair-name, userData=[B@2f6a014a, configDrive=false}) Execution failures: 0 error[s] Node failures: 1) RuntimeException on node LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b: java.lang.RuntimeException: org.jclouds.compute.callables.ScriptStillRunningException: time up waiting 600s for BlockUntilInitScriptStatusIsZeroThenReturnOutput{commandRunner=RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete{node=LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b, name=bootstrap, runAsRoot=true}} to complete. call get() on this exception to get access to the task in progress at shaded.com.google.common.base.Throwables.propagate(Throwables.java:160) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.doCall(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:63) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:77) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:125) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:146) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:53) at shaded.com.google.common.util.concurrent.Futures$1.apply(Futures.java:711) at shaded.com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:849) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.jclouds.compute.callables.ScriptStillRunningException: time up waiting 600s for BlockUntilInitScriptStatusIsZeroThenReturnOutput{commandRunner=RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete{node=LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b, name=bootstrap, runAsRoot=true}} to complete. call get() on this exception to get access to the task in progress at org.jclouds.compute.callables.BlockUntilInitScriptStatusIsZeroThenReturnOutput.get(BlockUntilInitScriptStatusIsZeroThenReturnOutput.java:196) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.doCall(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:60) ... 9 more Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task. at shaded.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:269) at shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96) at org.jclouds.compute.callables.BlockUntilInitScriptStatusIsZeroThenReturnOutput.get(BlockUntilInitScriptStatusIsZeroThenReturnOutput.java:194) ... 10 more 1 error[s] at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:778) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:631) at org.kohsuke.stapler.Stapler.service(Stapler.java:225) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:74) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: org.jclouds.compute.RunNodesException: error running 1 node group(slave-name) location(LOCATION) image(ca365c26-dbda-40a1-9def-7deb0fb885b2) size(16) options({loginUser=root, loginPasswordPresent=true, loginPrivateKeyPresent=true, scriptPresent=true, userMetadata={Name=slave-name}, keyPairName=key-pair-name, userData=[B@2f6a014a, configDrive=false}) Execution failures: 0 error[s] Node failures: 1) RuntimeException on node LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b: java.lang.RuntimeException: org.jclouds.compute.callables.ScriptStillRunningException: time up waiting 600s for BlockUntilInitScriptStatusIsZeroThenReturnOutput{commandRunner=RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete{node=LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b, name=bootstrap, runAsRoot=true}} to complete. call get() on this exception to get access to the task in progress at shaded.com.google.common.base.Throwables.propagate(Throwables.java:160) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.doCall(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:63) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:77) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:125) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:146) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:53) at shaded.com.google.common.util.concurrent.Futures$1.apply(Futures.java:711) at shaded.com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:849) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.jclouds.compute.callables.ScriptStillRunningException: time up waiting 600s for BlockUntilInitScriptStatusIsZeroThenReturnOutput{commandRunner=RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete{node=LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b, name=bootstrap, runAsRoot=true}} to complete. call get() on this exception to get access to the task in progress at org.jclouds.compute.callables.BlockUntilInitScriptStatusIsZeroThenReturnOutput.get(BlockUntilInitScriptStatusIsZeroThenReturnOutput.java:196) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.doCall(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:60) ... 9 more Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task. at shaded.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:269) at shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96) at org.jclouds.compute.callables.BlockUntilInitScriptStatusIsZeroThenReturnOutput.get(BlockUntilInitScriptStatusIsZeroThenReturnOutput.java:194) ... 10 more 1 error[s] at shaded.com.google.common.base.Throwables.propagate(Throwables.java:160) at jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate.destroyBadNodesAndPropagate(JCloudsSlaveTemplate.java:347) at jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate.get(JCloudsSlaveTemplate.java:336) at jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate.provisionSlave(JCloudsSlaveTemplate.java:191) at jenkins.plugins.jclouds.compute.JCloudsCloud.doProvision(JCloudsCloud.java:281) at sun.reflect.GeneratedMethodAccessor1963.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728) ... 66 more Caused by: org.jclouds.compute.RunNodesException: error running 1 node group(slave-name) location(LOCATION) image(ca365c26-dbda-40a1-9def-7deb0fb885b2) size(16) options({loginUser=root, loginPasswordPresent=true, loginPrivateKeyPresent=true, scriptPresent=true, userMetadata={Name=slave-name}, keyPairName=key-pair-name, userData=[B@2f6a014a, configDrive=false}) Execution failures: 0 error[s] Node failures: 1) RuntimeException on node LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b: java.lang.RuntimeException: org.jclouds.compute.callables.ScriptStillRunningException: time up waiting 600s for BlockUntilInitScriptStatusIsZeroThenReturnOutput{commandRunner=RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete{node=LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b, name=bootstrap, runAsRoot=true}} to complete. call get() on this exception to get access to the task in progress at shaded.com.google.common.base.Throwables.propagate(Throwables.java:160) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.doCall(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:63) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:77) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:125) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:146) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:53) at shaded.com.google.common.util.concurrent.Futures$1.apply(Futures.java:711) at shaded.com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:849) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.jclouds.compute.callables.ScriptStillRunningException: time up waiting 600s for BlockUntilInitScriptStatusIsZeroThenReturnOutput{commandRunner=RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete{node=LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b, name=bootstrap, runAsRoot=true}} to complete. call get() on this exception to get access to the task in progress at org.jclouds.compute.callables.BlockUntilInitScriptStatusIsZeroThenReturnOutput.get(BlockUntilInitScriptStatusIsZeroThenReturnOutput.java:196) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.doCall(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:60) ... 9 more Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task. at shaded.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:269) at shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96) at org.jclouds.compute.callables.BlockUntilInitScriptStatusIsZeroThenReturnOutput.get(BlockUntilInitScriptStatusIsZeroThenReturnOutput.java:194) ... 10 more 1 error[s] at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:227) at sun.reflect.GeneratedMethodAccessor210.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:40) at com.sun.proxy.$Proxy88.createNodesInGroup(Unknown Source) at jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate.get(JCloudsSlaveTemplate.java:334) ... 77 more
Running init script can be essential to log in and therefore to see logs in /tmp/bootstrap/.