Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38481

AWS method change causes too many slaves to be launched

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • aws-java-sdk-plugin, ec2-plugin
    • None
    • Jenkins 2.7.1 ec2-plugin v1.35

      Got the following error while trying to launch new EC2 build slave.
      I noticed that a new EC2 instance has been launched successfully, but the error still pop up.

      Stack trace
      javax.servlet.ServletException: java.lang.NoSuchMethodError: com.amazonaws.services.ec2.AmazonEC2.createTags(Lcom/amazonaws/services/ec2/model/CreateTagsRequest;)V
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:796)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:233)
      at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
      at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      at org.eclipse.jetty.server.Server.handle(Server.java:499)
      at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
      at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
      at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
      at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.NoSuchMethodError: com.amazonaws.services.ec2.AmazonEC2.createTags(Lcom/amazonaws/services/ec2/model/CreateTagsRequest;)V
      at hudson.plugins.ec2.SlaveTemplate.updateRemoteTags(SlaveTemplate.java:882)
      at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:585)
      at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:377)
      at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:515)
      at hudson.plugins.ec2.EC2Cloud.doProvision(EC2Cloud.java:340)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:324)
      at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:167)
      at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:100)
      at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
      at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      ... 58 more

          [JENKINS-38481] AWS method change causes too many slaves to be launched

          Tim Feng created issue -

          Sebastian Hillig added a comment -

          Bug is caused by upgrading Amazon Web Services SDK to 1.11.37

          Sebastian Hillig added a comment - Bug is caused by upgrading Amazon Web Services SDK to 1.11.37

          James Huston added a comment -

          I can confirm that downgrading the Amazon Web Services SDK plugin to 1.10.50 resolves this issue for me.

          James Huston added a comment - I can confirm that downgrading the Amazon Web Services SDK plugin to 1.10.50 resolves this issue for me.
          Mika Karjalainen made changes -
          Component/s New: aws-java-sdk-plugin [ 21126 ]

          Stéphan Kochen added a comment -

          As a side note, this has bitten us quite a bit, because Jenkins proceeded to fill our AWS account up to the limit with EC2 instances. Doesn't look like there's a way to limit instances per IAM user on the AWS side either.

          Perhaps, as a defensive measure against future bugs, the code could also try to terminate the instance if provisioning fails?

          Stéphan Kochen added a comment - As a side note, this has bitten us quite a bit, because Jenkins proceeded to fill our AWS account up to the limit with EC2 instances. Doesn't look like there's a way to limit instances per IAM user on the AWS side either. Perhaps, as a defensive measure against future bugs, the code could also try to terminate the instance if provisioning fails?

          Stéphan Kochen added a comment -

          Temporary workaround: remove tags from your slave configuration. (If your Jenkins is offline, remove the <tags> element from config.xml.)

          This way, tags are not included in the filter when Jenkins looks for existing instances, and will match instances for which tagging failed.

          (The filter is otherwise sufficiently narrow to not match unrelated instances in most cases. It includes, e.g., instance type, AMI and SSH key name, security group, etc.)

          Stéphan Kochen added a comment - Temporary workaround: remove tags from your slave configuration. (If your Jenkins is offline, remove the <tags> element from config.xml.) This way, tags are not included in the filter when Jenkins looks for existing instances, and will match instances for which tagging failed. (The filter is otherwise sufficiently narrow to not match unrelated instances in most cases. It includes, e.g., instance type, AMI and SSH key name, security group, etc.)

          Chris Ernenwein added a comment -

          can we increase the priority on this? we also "has bitten us quite a bit, because Jenkins proceeded to fill our AWS account up to the limit with EC2 instances"

          Chris Ernenwein added a comment - can we increase the priority on this? we also "has bitten us quite a bit, because Jenkins proceeded to fill our AWS account up to the limit with EC2 instances"

          Paul Becotte added a comment - - edited

          Same here. I also cannot figure out how to proceed. downgrading this plugin to 1.34 and the sdk to 1.10.50 results in slaves being terminated immediately after launch...with them both at the latest version, I get the above error message (even with tags removed from all my AMI configs).

          Any idea on how to get back up and running?

          EDIT: The second time I tried downgrading them both, the system worked properly.

          Paul Becotte added a comment - - edited Same here. I also cannot figure out how to proceed. downgrading this plugin to 1.34 and the sdk to 1.10.50 results in slaves being terminated immediately after launch...with them both at the latest version, I get the above error message (even with tags removed from all my AMI configs). Any idea on how to get back up and running? EDIT: The second time I tried downgrading them both, the system worked properly.

          Michal Cichra added a comment - - edited

          We just paid 250$ in AWS because the plugin created 20 pretty big machines.
          I'm stunned that no one was able to issue some hotfix for days now. People can be paying a lot and not know. Or why it has priority `Minor`.

          I can confirm that downgrading the Amazon Web Services SDK plugin fixes the issue.

          Michal Cichra added a comment - - edited We just paid 250$ in AWS because the plugin created 20 pretty big machines. I'm stunned that no one was able to issue some hotfix for days now. People can be paying a lot and not know. Or why it has priority `Minor`. I can confirm that downgrading the Amazon Web Services SDK plugin fixes the issue.

          Jared Biel added a comment -

          Another confirmation that downgrading EC2 to 1.34 and the AWS SDK to 1.10.50 works for now.

          I too am surprised that this is a minor. Luckily due to the instance size that we use and account restrictions this bug only cost us ~$4 + time.

          Jared Biel added a comment - Another confirmation that downgrading EC2 to 1.34 and the AWS SDK to 1.10.50 works for now. I too am surprised that this is a minor. Luckily due to the instance size that we use and account restrictions this bug only cost us ~$4 + time.

            francisu Francis Upton
            timff Tim Feng
            Votes:
            24 Vote for this issue
            Watchers:
            32 Start watching this issue

              Created:
              Updated:
              Resolved: