Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38115

spot instance slaves are not removed, blocks instance termination

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Blocker
    • Resolution: Fixed
    • Component/s: ec2-fleet-plugin
    • Labels:
    • Environment:
      Jenkins ver. 2.7.1
      ec2-fleet-plugin ver. 1.0
      Amazon Linux AMI 2015.03
      java version "1.7.0_85"
      OpenJDK Runtime Environment (amzn-2.6.1.3.61.amzn1-x86_64 u85-b01)
      OpenJDK 64-Bit Server VM (build 24.85-b03, mixed mode)
    • Similar Issues:

      Description

      Scenario:
      I've created a fleet under clouds in jenkins config, and selected my spot request. test returns an error, which may or may not be related. the ids i've redacted all match the ids i see in the ec2 management console.

      The SpotFleetRequestId(s) sfr-<stuff> do not exist. (Service: AmazonEC2; Status Code: 400; Error Code: InvalidSpotFleetRequestId.NotFound; Request ID: <other stuff> )

      to test the plugin, i set it up with a max cluster size of 3, scaledown timeout of 15 minutes. the spot request capacity defaults to 1.

      expected:
      when the test job runs, two additional slaves are created, backed by new spot instances. everything runs fine until the job is done and the scaledown timeout hits, at which point the slaves are not removed. one or more may show disconnected, and the log begins to show errors:

      SEVERE: I/O error in channel i-cdcef210
      java.io.IOException: Unexpected termination of the channel
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
      Caused by: java.io.EOFException
      at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2332)
      at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2801)
      at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801)
      at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
      at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
      at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

      SEVERE: Timer task hudson.slaves.ComputerRetentionWork@5654d212 failed
      java.lang.IllegalStateException: Unknown instance terminated: i-cdcef210
      at com.amazon.jenkins.ec2fleet.EC2FleetCloud.terminateInstance(EC2FleetCloud.java:266)
      at com.amazon.jenkins.ec2fleet.IdleRetentionStrategy.check(IdleRetentionStrategy.java:28)
      at com.amazon.jenkins.ec2fleet.IdleRetentionStrategy.check(IdleRetentionStrategy.java:12)
      at hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:72)
      at hudson.model.Queue._withLock(Queue.java:1315)
      at hudson.model.Queue.withLock(Queue.java:1192)
      at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:63)
      at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)

      the second of those errors repeats until the slave is manually removed (which sometimes results in a jenkins stack trace in the UI with the same error), at which point the rest of the process is unblocked. the spot instances are terminated and the slaves are removed.

      the plugin is essentially nonfunctional in this state

        Attachments

          Activity

          Hide
          dimitrigilbert Dimitri Gilbert added a comment -

          Same here, everything works fine but that, it is extremelly frustrating.
          Here is the output of the crash :
          javax.servlet.ServletException: java.lang.IllegalStateException: Unknown instance terminated: i-4c8a415b
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:796)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:380)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          at org.kohsuke.stapler.MetaClass$3.doDispatch(MetaClass.java:196)
          at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
          at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
          at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
          at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:80)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
          at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
          at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
          at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
          at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
          at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
          at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
          at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
          at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
          at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
          at org.eclipse.jetty.server.Server.handle(Server.java:499)
          at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
          at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
          at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
          at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
          Caused by: java.lang.IllegalStateException: Unknown instance terminated: i-4c8a415b
          at com.amazon.jenkins.ec2fleet.EC2FleetCloud.terminateInstance(EC2FleetCloud.java:266)
          at com.amazon.jenkins.ec2fleet.IdleRetentionStrategy.check(IdleRetentionStrategy.java:28)
          at com.amazon.jenkins.ec2fleet.IdleRetentionStrategy.check(IdleRetentionStrategy.java:12)
          at hudson.slaves.SlaveComputer$4.run(SlaveComputer.java:726)
          at hudson.model.Queue._withLock(Queue.java:1303)
          at hudson.model.Queue.withLock(Queue.java:1180)
          at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:723)
          at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:118)
          at hudson.model.AbstractCIBase.access$000(AbstractCIBase.java:44)
          at hudson.model.AbstractCIBase$2.run(AbstractCIBase.java:186)
          at hudson.model.Queue._withLock(Queue.java:1303)
          at hudson.model.Queue.withLock(Queue.java:1180)
          at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:169)
          at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1564)
          at jenkins.model.Nodes$5.run(Nodes.java:246)
          at hudson.model.Queue._withLock(Queue.java:1303)
          at hudson.model.Queue.withLock(Queue.java:1180)
          at jenkins.model.Nodes.removeNode(Nodes.java:237)
          at jenkins.model.Jenkins.removeNode(Jenkins.java:2086)
          at hudson.model.Computer.doDoDelete(Computer.java:1498)
          at sun.reflect.GeneratedMethodAccessor877.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:324)
          at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:52)
          at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
          at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:167)
          at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:100)
          at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
          at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)

          Show
          dimitrigilbert Dimitri Gilbert added a comment - Same here, everything works fine but that, it is extremelly frustrating. Here is the output of the crash : javax.servlet.ServletException: java.lang.IllegalStateException: Unknown instance terminated: i-4c8a415b at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:796) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:380) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$3.doDispatch(MetaClass.java:196) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:80) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException: Unknown instance terminated: i-4c8a415b at com.amazon.jenkins.ec2fleet.EC2FleetCloud.terminateInstance(EC2FleetCloud.java:266) at com.amazon.jenkins.ec2fleet.IdleRetentionStrategy.check(IdleRetentionStrategy.java:28) at com.amazon.jenkins.ec2fleet.IdleRetentionStrategy.check(IdleRetentionStrategy.java:12) at hudson.slaves.SlaveComputer$4.run(SlaveComputer.java:726) at hudson.model.Queue._withLock(Queue.java:1303) at hudson.model.Queue.withLock(Queue.java:1180) at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:723) at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:118) at hudson.model.AbstractCIBase.access$000(AbstractCIBase.java:44) at hudson.model.AbstractCIBase$2.run(AbstractCIBase.java:186) at hudson.model.Queue._withLock(Queue.java:1303) at hudson.model.Queue.withLock(Queue.java:1180) at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:169) at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1564) at jenkins.model.Nodes$5.run(Nodes.java:246) at hudson.model.Queue._withLock(Queue.java:1303) at hudson.model.Queue.withLock(Queue.java:1180) at jenkins.model.Nodes.removeNode(Nodes.java:237) at jenkins.model.Jenkins.removeNode(Jenkins.java:2086) at hudson.model.Computer.doDoDelete(Computer.java:1498) at sun.reflect.GeneratedMethodAccessor877.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:324) at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:52) at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:167) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:100) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          Hide
          weargoggles Pete Wildsmith added a comment -

          This issue, along with a number of others, is claimed to be resolved by the PR https://github.com/jenkinsci/ec2-fleet-plugin/pull/3

          Please advise whether further action is required.

          Show
          weargoggles Pete Wildsmith added a comment - This issue, along with a number of others, is claimed to be resolved by the PR https://github.com/jenkinsci/ec2-fleet-plugin/pull/3 Please advise whether further action is required.
          Hide
          markl_lagendijk Mark Lagendijk added a comment -

          I think this indeed is fixed in the current version of the plugin. This issue should be closed.

          Show
          markl_lagendijk Mark Lagendijk added a comment - I think this indeed is fixed in the current version of the plugin. This issue should be closed.
          Hide
          markl_lagendijk Mark Lagendijk added a comment -

          Closing this old issue which was already fixed. If a similar issue still exists, reopen this one or create a new one.

          Show
          markl_lagendijk Mark Lagendijk added a comment - Closing this old issue which was already fixed. If a similar issue still exists, reopen this one or create a new one.

            People

            Assignee:
            cyberax Aleksei Besogonov
            Reporter:
            jvais_soasta jude vais
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: