[JENKINS-25834] ScriptStillRunningException does not provide stdout/stderr

Type: Improvement
Resolution: Unresolved
Priority: Minor
Component/s: jclouds-plugin
Labels:
None

Similar Issues:
Powered by SuggestiMate

Show

When init script fail to complete in time, slave is silently not provisioned. There is no indication in logs as well. Manual provisioning shows following:

javax.servlet.ServletException: java.lang.RuntimeException: org.jclouds.compute.RunNodesException: error running 1 node group(slave-name) location(LOCATION) image(ca365c26-dbda-40a1-9def-7deb0fb885b2) size(16) options({loginUser=root, loginPasswordPresent=true, loginPrivateKeyPresent=true, scriptPresent=true, userMetadata={Name=slave-name}, keyPairName=key-pair-name, userData=[B@2f6a014a, configDrive=false})
Execution failures:

0 error[s]
Node failures:

1) RuntimeException on node LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b:
java.lang.RuntimeException: org.jclouds.compute.callables.ScriptStillRunningException: time up waiting 600s for BlockUntilInitScriptStatusIsZeroThenReturnOutput{commandRunner=RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete{node=LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b, name=bootstrap, runAsRoot=true}} to complete. call get() on this exception to get access to the task in progress
  at shaded.com.google.common.base.Throwables.propagate(Throwables.java:160)
  at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.doCall(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:63)
  at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:77)
  at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:125)
  at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:146)
  at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:53)
  at shaded.com.google.common.util.concurrent.Futures$1.apply(Futures.java:711)
  at shaded.com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:849)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
Caused by: org.jclouds.compute.callables.ScriptStillRunningException: time up waiting 600s for BlockUntilInitScriptStatusIsZeroThenReturnOutput{commandRunner=RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete{node=LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b, name=bootstrap, runAsRoot=true}} to complete. call get() on this exception to get access to the task in progress
  at org.jclouds.compute.callables.BlockUntilInitScriptStatusIsZeroThenReturnOutput.get(BlockUntilInitScriptStatusIsZeroThenReturnOutput.java:196)
  at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.doCall(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:60)
  ... 9 more
Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task.
  at shaded.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:269)
  at shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96)
  at org.jclouds.compute.callables.BlockUntilInitScriptStatusIsZeroThenReturnOutput.get(BlockUntilInitScriptStatusIsZeroThenReturnOutput.java:194)
  ... 10 more


1 error[s]
  at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:778)
  at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
  at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
  at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
  at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
  at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
  at org.kohsuke.stapler.Stapler.invoke(Stapler.java:631)
  at org.kohsuke.stapler.Stapler.service(Stapler.java:225)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
  at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
  at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58)
  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
  at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
  at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
  at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:74)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
  at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
  at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
  at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
  at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
  at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
  at org.eclipse.jetty.server.Server.handle(Server.java:370)
  at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
  at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960)
  at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021)
  at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
  at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
  at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
  at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
  at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
  at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
  at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.jclouds.compute.RunNodesException: error running 1 node group(slave-name) location(LOCATION) image(ca365c26-dbda-40a1-9def-7deb0fb885b2) size(16) options({loginUser=root, loginPasswordPresent=true, loginPrivateKeyPresent=true, scriptPresent=true, userMetadata={Name=slave-name}, keyPairName=key-pair-name, userData=[B@2f6a014a, configDrive=false})
Execution failures:

0 error[s]
Node failures:

1) RuntimeException on node LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b:
java.lang.RuntimeException: org.jclouds.compute.callables.ScriptStillRunningException: time up waiting 600s for BlockUntilInitScriptStatusIsZeroThenReturnOutput{commandRunner=RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete{node=LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b, name=bootstrap, runAsRoot=true}} to complete. call get() on this exception to get access to the task in progress
  at shaded.com.google.common.base.Throwables.propagate(Throwables.java:160)
  at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.doCall(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:63)
  at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:77)
  at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:125)
  at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:146)
  at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:53)
  at shaded.com.google.common.util.concurrent.Futures$1.apply(Futures.java:711)
  at shaded.com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:849)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
Caused by: org.jclouds.compute.callables.ScriptStillRunningException: time up waiting 600s for BlockUntilInitScriptStatusIsZeroThenReturnOutput{commandRunner=RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete{node=LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b, name=bootstrap, runAsRoot=true}} to complete. call get() on this exception to get access to the task in progress
  at org.jclouds.compute.callables.BlockUntilInitScriptStatusIsZeroThenReturnOutput.get(BlockUntilInitScriptStatusIsZeroThenReturnOutput.java:196)
  at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.doCall(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:60)
  ... 9 more
Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task.
  at shaded.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:269)
  at shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96)
  at org.jclouds.compute.callables.BlockUntilInitScriptStatusIsZeroThenReturnOutput.get(BlockUntilInitScriptStatusIsZeroThenReturnOutput.java:194)
  ... 10 more


1 error[s]
  at shaded.com.google.common.base.Throwables.propagate(Throwables.java:160)
  at jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate.destroyBadNodesAndPropagate(JCloudsSlaveTemplate.java:347)
  at jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate.get(JCloudsSlaveTemplate.java:336)
  at jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate.provisionSlave(JCloudsSlaveTemplate.java:191)
  at jenkins.plugins.jclouds.compute.JCloudsCloud.doProvision(JCloudsCloud.java:281)
  at sun.reflect.GeneratedMethodAccessor1963.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
  at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
  at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
  at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
  at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
  at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
  ... 66 more
Caused by: org.jclouds.compute.RunNodesException: error running 1 node group(slave-name) location(LOCATION) image(ca365c26-dbda-40a1-9def-7deb0fb885b2) size(16) options({loginUser=root, loginPasswordPresent=true, loginPrivateKeyPresent=true, scriptPresent=true, userMetadata={Name=slave-name}, keyPairName=key-pair-name, userData=[B@2f6a014a, configDrive=false})
Execution failures:

0 error[s]
Node failures:

1) RuntimeException on node LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b:
java.lang.RuntimeException: org.jclouds.compute.callables.ScriptStillRunningException: time up waiting 600s for BlockUntilInitScriptStatusIsZeroThenReturnOutput{commandRunner=RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete{node=LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b, name=bootstrap, runAsRoot=true}} to complete. call get() on this exception to get access to the task in progress
  at shaded.com.google.common.base.Throwables.propagate(Throwables.java:160)
  at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.doCall(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:63)
  at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:77)
  at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:125)
  at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:146)
  at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:53)
  at shaded.com.google.common.util.concurrent.Futures$1.apply(Futures.java:711)
  at shaded.com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:849)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
Caused by: org.jclouds.compute.callables.ScriptStillRunningException: time up waiting 600s for BlockUntilInitScriptStatusIsZeroThenReturnOutput{commandRunner=RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete{node=LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b, name=bootstrap, runAsRoot=true}} to complete. call get() on this exception to get access to the task in progress
  at org.jclouds.compute.callables.BlockUntilInitScriptStatusIsZeroThenReturnOutput.get(BlockUntilInitScriptStatusIsZeroThenReturnOutput.java:196)
  at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.doCall(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:60)
  ... 9 more
Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task.
  at shaded.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:269)
  at shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96)
  at org.jclouds.compute.callables.BlockUntilInitScriptStatusIsZeroThenReturnOutput.get(BlockUntilInitScriptStatusIsZeroThenReturnOutput.java:194)
  ... 10 more


1 error[s]
  at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:227)
  at sun.reflect.GeneratedMethodAccessor210.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:40)
  at com.sun.proxy.$Proxy88.createNodesInGroup(Unknown Source)
  at jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate.get(JCloudsSlaveTemplate.java:334)
  ... 77 more

Running init script can be essential to log in and therefore to see logs in /tmp/bootstrap/.

Oliver Gondža created issue - 2014-11-30 12:06

Oliver Gondža made changes - 2015-02-17 10:33

Assignee

Original: Andrew Bayer [ abayer ]

New: Oliver Gondža [ olivergondza ]

Oliver Gondža made changes - 2015-06-16 18:15

Assignee

Original: Oliver Gondža [ olivergondza ]

New: Andrew Bayer [ abayer ]

R. Tyler Croy made changes - 2016-07-25 23:51

Workflow

Original: JNJira [ 159815 ]

New: JNJira + In-Review [ 180142 ]

Fritz Elfert made changes - 2016-12-05 09:12

Assignee

Original: Andrew Bayer [ abayer ]

New: Fritz Elfert [ felfert ]

Assignee:: Fritz Elfert

Reporter:: Oliver Gondža

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Created:: 2014-11-30 12:06

Updated:: 2016-12-05 09:12

Jenkins

Details

Description

Attachments

Activity

People

Dates