Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-25834

ScriptStillRunningException does not provide stdout/stderr

XMLWordPrintable

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • jclouds-plugin
    • None

      When init script fail to complete in time, slave is silently not provisioned. There is no indication in logs as well. Manual provisioning shows following:

      javax.servlet.ServletException: java.lang.RuntimeException: org.jclouds.compute.RunNodesException: error running 1 node group(slave-name) location(LOCATION) image(ca365c26-dbda-40a1-9def-7deb0fb885b2) size(16) options({loginUser=root, loginPasswordPresent=true, loginPrivateKeyPresent=true, scriptPresent=true, userMetadata={Name=slave-name}, keyPairName=key-pair-name, userData=[B@2f6a014a, configDrive=false})
      Execution failures:
      
      0 error[s]
      Node failures:
      
      1) RuntimeException on node LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b:
      java.lang.RuntimeException: org.jclouds.compute.callables.ScriptStillRunningException: time up waiting 600s for BlockUntilInitScriptStatusIsZeroThenReturnOutput{commandRunner=RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete{node=LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b, name=bootstrap, runAsRoot=true}} to complete. call get() on this exception to get access to the task in progress
        at shaded.com.google.common.base.Throwables.propagate(Throwables.java:160)
        at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.doCall(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:63)
        at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:77)
        at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:125)
        at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:146)
        at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:53)
        at shaded.com.google.common.util.concurrent.Futures$1.apply(Futures.java:711)
        at shaded.com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:849)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
      Caused by: org.jclouds.compute.callables.ScriptStillRunningException: time up waiting 600s for BlockUntilInitScriptStatusIsZeroThenReturnOutput{commandRunner=RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete{node=LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b, name=bootstrap, runAsRoot=true}} to complete. call get() on this exception to get access to the task in progress
        at org.jclouds.compute.callables.BlockUntilInitScriptStatusIsZeroThenReturnOutput.get(BlockUntilInitScriptStatusIsZeroThenReturnOutput.java:196)
        at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.doCall(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:60)
        ... 9 more
      Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task.
        at shaded.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:269)
        at shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96)
        at org.jclouds.compute.callables.BlockUntilInitScriptStatusIsZeroThenReturnOutput.get(BlockUntilInitScriptStatusIsZeroThenReturnOutput.java:194)
        ... 10 more
      
      
      1 error[s]
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:778)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
        at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
        at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:631)
        at org.kohsuke.stapler.Stapler.service(Stapler.java:225)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
        at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
        at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
        at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
        at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:74)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
        at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
        at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
        at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
        at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
        at org.eclipse.jetty.server.Server.handle(Server.java:370)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
        at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
        at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
        at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.RuntimeException: org.jclouds.compute.RunNodesException: error running 1 node group(slave-name) location(LOCATION) image(ca365c26-dbda-40a1-9def-7deb0fb885b2) size(16) options({loginUser=root, loginPasswordPresent=true, loginPrivateKeyPresent=true, scriptPresent=true, userMetadata={Name=slave-name}, keyPairName=key-pair-name, userData=[B@2f6a014a, configDrive=false})
      Execution failures:
      
      0 error[s]
      Node failures:
      
      1) RuntimeException on node LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b:
      java.lang.RuntimeException: org.jclouds.compute.callables.ScriptStillRunningException: time up waiting 600s for BlockUntilInitScriptStatusIsZeroThenReturnOutput{commandRunner=RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete{node=LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b, name=bootstrap, runAsRoot=true}} to complete. call get() on this exception to get access to the task in progress
        at shaded.com.google.common.base.Throwables.propagate(Throwables.java:160)
        at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.doCall(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:63)
        at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:77)
        at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:125)
        at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:146)
        at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:53)
        at shaded.com.google.common.util.concurrent.Futures$1.apply(Futures.java:711)
        at shaded.com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:849)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
      Caused by: org.jclouds.compute.callables.ScriptStillRunningException: time up waiting 600s for BlockUntilInitScriptStatusIsZeroThenReturnOutput{commandRunner=RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete{node=LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b, name=bootstrap, runAsRoot=true}} to complete. call get() on this exception to get access to the task in progress
        at org.jclouds.compute.callables.BlockUntilInitScriptStatusIsZeroThenReturnOutput.get(BlockUntilInitScriptStatusIsZeroThenReturnOutput.java:196)
        at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.doCall(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:60)
        ... 9 more
      Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task.
        at shaded.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:269)
        at shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96)
        at org.jclouds.compute.callables.BlockUntilInitScriptStatusIsZeroThenReturnOutput.get(BlockUntilInitScriptStatusIsZeroThenReturnOutput.java:194)
        ... 10 more
      
      
      1 error[s]
        at shaded.com.google.common.base.Throwables.propagate(Throwables.java:160)
        at jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate.destroyBadNodesAndPropagate(JCloudsSlaveTemplate.java:347)
        at jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate.get(JCloudsSlaveTemplate.java:336)
        at jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate.provisionSlave(JCloudsSlaveTemplate.java:191)
        at jenkins.plugins.jclouds.compute.JCloudsCloud.doProvision(JCloudsCloud.java:281)
        at sun.reflect.GeneratedMethodAccessor1963.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
        at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
        at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
        at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
        at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
        ... 66 more
      Caused by: org.jclouds.compute.RunNodesException: error running 1 node group(slave-name) location(LOCATION) image(ca365c26-dbda-40a1-9def-7deb0fb885b2) size(16) options({loginUser=root, loginPasswordPresent=true, loginPrivateKeyPresent=true, scriptPresent=true, userMetadata={Name=slave-name}, keyPairName=key-pair-name, userData=[B@2f6a014a, configDrive=false})
      Execution failures:
      
      0 error[s]
      Node failures:
      
      1) RuntimeException on node LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b:
      java.lang.RuntimeException: org.jclouds.compute.callables.ScriptStillRunningException: time up waiting 600s for BlockUntilInitScriptStatusIsZeroThenReturnOutput{commandRunner=RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete{node=LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b, name=bootstrap, runAsRoot=true}} to complete. call get() on this exception to get access to the task in progress
        at shaded.com.google.common.base.Throwables.propagate(Throwables.java:160)
        at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.doCall(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:63)
        at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:77)
        at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:125)
        at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:146)
        at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.apply(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:53)
        at shaded.com.google.common.util.concurrent.Futures$1.apply(Futures.java:711)
        at shaded.com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:849)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
      Caused by: org.jclouds.compute.callables.ScriptStillRunningException: time up waiting 600s for BlockUntilInitScriptStatusIsZeroThenReturnOutput{commandRunner=RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete{node=LOCATION/976594b1-5dbf-45ad-87c5-5a288d0a8f3b, name=bootstrap, runAsRoot=true}} to complete. call get() on this exception to get access to the task in progress
        at org.jclouds.compute.callables.BlockUntilInitScriptStatusIsZeroThenReturnOutput.get(BlockUntilInitScriptStatusIsZeroThenReturnOutput.java:196)
        at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.doCall(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:60)
        ... 9 more
      Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task.
        at shaded.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:269)
        at shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96)
        at org.jclouds.compute.callables.BlockUntilInitScriptStatusIsZeroThenReturnOutput.get(BlockUntilInitScriptStatusIsZeroThenReturnOutput.java:194)
        ... 10 more
      
      
      1 error[s]
        at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:227)
        at sun.reflect.GeneratedMethodAccessor210.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:40)
        at com.sun.proxy.$Proxy88.createNodesInGroup(Unknown Source)
        at jenkins.plugins.jclouds.compute.JCloudsSlaveTemplate.get(JCloudsSlaveTemplate.java:334)
        ... 77 more
      
      

      Running init script can be essential to log in and therefore to see logs in /tmp/bootstrap/.

            felfert Fritz Elfert
            olivergondza Oliver Gondža
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: