Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38481

AWS method change causes too many slaves to be launched

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • aws-java-sdk-plugin, ec2-plugin
    • None
    • Jenkins 2.7.1 ec2-plugin v1.35

      Got the following error while trying to launch new EC2 build slave.
      I noticed that a new EC2 instance has been launched successfully, but the error still pop up.

      Stack trace
      javax.servlet.ServletException: java.lang.NoSuchMethodError: com.amazonaws.services.ec2.AmazonEC2.createTags(Lcom/amazonaws/services/ec2/model/CreateTagsRequest;)V
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:796)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:233)
      at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
      at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      at org.eclipse.jetty.server.Server.handle(Server.java:499)
      at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
      at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
      at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
      at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.NoSuchMethodError: com.amazonaws.services.ec2.AmazonEC2.createTags(Lcom/amazonaws/services/ec2/model/CreateTagsRequest;)V
      at hudson.plugins.ec2.SlaveTemplate.updateRemoteTags(SlaveTemplate.java:882)
      at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:585)
      at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:377)
      at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:515)
      at hudson.plugins.ec2.EC2Cloud.doProvision(EC2Cloud.java:340)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:324)
      at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:167)
      at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:100)
      at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
      at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      ... 58 more

          [JENKINS-38481] AWS method change causes too many slaves to be launched

          Sebastian Hillig added a comment -

          Bug is caused by upgrading Amazon Web Services SDK to 1.11.37

          Sebastian Hillig added a comment - Bug is caused by upgrading Amazon Web Services SDK to 1.11.37

          James Huston added a comment -

          I can confirm that downgrading the Amazon Web Services SDK plugin to 1.10.50 resolves this issue for me.

          James Huston added a comment - I can confirm that downgrading the Amazon Web Services SDK plugin to 1.10.50 resolves this issue for me.

          Stéphan Kochen added a comment -

          As a side note, this has bitten us quite a bit, because Jenkins proceeded to fill our AWS account up to the limit with EC2 instances. Doesn't look like there's a way to limit instances per IAM user on the AWS side either.

          Perhaps, as a defensive measure against future bugs, the code could also try to terminate the instance if provisioning fails?

          Stéphan Kochen added a comment - As a side note, this has bitten us quite a bit, because Jenkins proceeded to fill our AWS account up to the limit with EC2 instances. Doesn't look like there's a way to limit instances per IAM user on the AWS side either. Perhaps, as a defensive measure against future bugs, the code could also try to terminate the instance if provisioning fails?

          Stéphan Kochen added a comment -

          Temporary workaround: remove tags from your slave configuration. (If your Jenkins is offline, remove the <tags> element from config.xml.)

          This way, tags are not included in the filter when Jenkins looks for existing instances, and will match instances for which tagging failed.

          (The filter is otherwise sufficiently narrow to not match unrelated instances in most cases. It includes, e.g., instance type, AMI and SSH key name, security group, etc.)

          Stéphan Kochen added a comment - Temporary workaround: remove tags from your slave configuration. (If your Jenkins is offline, remove the <tags> element from config.xml.) This way, tags are not included in the filter when Jenkins looks for existing instances, and will match instances for which tagging failed. (The filter is otherwise sufficiently narrow to not match unrelated instances in most cases. It includes, e.g., instance type, AMI and SSH key name, security group, etc.)

          Chris Ernenwein added a comment -

          can we increase the priority on this? we also "has bitten us quite a bit, because Jenkins proceeded to fill our AWS account up to the limit with EC2 instances"

          Chris Ernenwein added a comment - can we increase the priority on this? we also "has bitten us quite a bit, because Jenkins proceeded to fill our AWS account up to the limit with EC2 instances"

          Paul Becotte added a comment - - edited

          Same here. I also cannot figure out how to proceed. downgrading this plugin to 1.34 and the sdk to 1.10.50 results in slaves being terminated immediately after launch...with them both at the latest version, I get the above error message (even with tags removed from all my AMI configs).

          Any idea on how to get back up and running?

          EDIT: The second time I tried downgrading them both, the system worked properly.

          Paul Becotte added a comment - - edited Same here. I also cannot figure out how to proceed. downgrading this plugin to 1.34 and the sdk to 1.10.50 results in slaves being terminated immediately after launch...with them both at the latest version, I get the above error message (even with tags removed from all my AMI configs). Any idea on how to get back up and running? EDIT: The second time I tried downgrading them both, the system worked properly.

          Michal Cichra added a comment - - edited

          We just paid 250$ in AWS because the plugin created 20 pretty big machines.
          I'm stunned that no one was able to issue some hotfix for days now. People can be paying a lot and not know. Or why it has priority `Minor`.

          I can confirm that downgrading the Amazon Web Services SDK plugin fixes the issue.

          Michal Cichra added a comment - - edited We just paid 250$ in AWS because the plugin created 20 pretty big machines. I'm stunned that no one was able to issue some hotfix for days now. People can be paying a lot and not know. Or why it has priority `Minor`. I can confirm that downgrading the Amazon Web Services SDK plugin fixes the issue.

          Jared Biel added a comment -

          Another confirmation that downgrading EC2 to 1.34 and the AWS SDK to 1.10.50 works for now.

          I too am surprised that this is a minor. Luckily due to the instance size that we use and account restrictions this bug only cost us ~$4 + time.

          Jared Biel added a comment - Another confirmation that downgrading EC2 to 1.34 and the AWS SDK to 1.10.50 works for now. I too am surprised that this is a minor. Luckily due to the instance size that we use and account restrictions this bug only cost us ~$4 + time.

          Nicolas De Loof added a comment - - edited

          Method seems unchanged in SDK 1.11.37, but actually a return type was added
          https://github.com/aws/aws-sdk-java/blob/1.11.37/aws-java-sdk-ec2/src/main/java/com/amazonaws/services/ec2/AmazonEC2.java#L1199
          which introduce binary compatibility issue.

          Proposed https://github.com/jenkinsci/ec2-plugin/pull/221

          Nicolas De Loof added a comment - - edited Method seems unchanged in SDK 1.11.37, but actually a return type was added https://github.com/aws/aws-sdk-java/blob/1.11.37/aws-java-sdk-ec2/src/main/java/com/amazonaws/services/ec2/AmazonEC2.java#L1199 which introduce binary compatibility issue. Proposed https://github.com/jenkinsci/ec2-plugin/pull/221

          Joshua Davis added a comment -

          This bug is bad news... I just had to terminate over 40 m4.large instances.

          Joshua Davis added a comment - This bug is bad news... I just had to terminate over 40 m4.large instances.

          Peter Reid added a comment -

          I am also experiencing this issue

          Peter Reid added a comment - I am also experiencing this issue

          Chris Kulinski added a comment -

          We also had this issue. Spun up slaves until we hit our EC2 instance cap. Not fun.

          Tested the PR fix linked above and it solved the problem. Release this fix soon!

          Chris Kulinski added a comment - We also had this issue. Spun up slaves until we hit our EC2 instance cap. Not fun. Tested the PR fix linked above and it solved the problem. Release this fix soon!

          SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Nicolas De Loof
          Path:
          pom.xml
          src/test/java/hudson/plugins/ec2/SlaveTemplateUnitTest.java
          http://jenkins-ci.org/commit/ec2-plugin/c8983e4a8c3b784ae5a9a58127c88f3013db490d
          Log:
          JENKINS-38481 1.11.37 changed com.amazonaws.services.ec2.AmazonEC2#createTags return type, so incompatible binary change

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Nicolas De Loof Path: pom.xml src/test/java/hudson/plugins/ec2/SlaveTemplateUnitTest.java http://jenkins-ci.org/commit/ec2-plugin/c8983e4a8c3b784ae5a9a58127c88f3013db490d Log: JENKINS-38481 1.11.37 changed com.amazonaws.services.ec2.AmazonEC2#createTags return type, so incompatible binary change

          Francis Upton added a comment -

          Fixed in release 1.36.

          Francis Upton added a comment - Fixed in release 1.36.

          Peter Reid added a comment -

          The release for this issue has not yet been released - can you please re-open this issue until the fix is available in the plugin repository.

          Peter Reid added a comment - The release for this issue has not yet been released - can you please re-open this issue until the fix is available in the plugin repository.

          Francis Upton added a comment -

          Generally we close the issue when the release is made. The release has been made and will be available in the next 24 hours in the repo, this action is automatic. You can see the release in the source repository: https://github.com/jenkinsci/ec2-plugin/releases

          Francis Upton added a comment - Generally we close the issue when the release is made. The release has been made and will be available in the next 24 hours in the repo, this action is automatic. You can see the release in the source repository: https://github.com/jenkinsci/ec2-plugin/releases

          Peter Reid added a comment -

          OK, cool. Just wanted clarification on that.

          Peter Reid added a comment - OK, cool. Just wanted clarification on that.

          Baptiste Mathus added a comment -

          timff just stumbled into this after seeing that log, there's something I don't understand though:

          you say

          I noticed that a new EC2 instance has been launched successfully, but the error still pop up.

          So I understand you're saying basically this stack is log spam, but you still file it as Critical? Is this impacting any visible feature of the plugin in the end?

          Thanks

          Baptiste Mathus added a comment - timff just stumbled into this after seeing that log, there's something I don't understand though: you say I noticed that a new EC2 instance has been launched successfully , but the error still pop up. So I understand you're saying basically this stack is log spam, but you still file it as Critical? Is this impacting any visible feature of the plugin in the end? Thanks

          Baptiste Mathus added a comment -

          OK, forget it. I think I understand re-reading the title.

          Baptiste Mathus added a comment - OK, forget it. I think I understand re-reading the title.

          Carlos Tadeu Panato added a comment -

          looks like now with this fix none are being launch

          Carlos Tadeu Panato added a comment - looks like now with this fix none are being launch

          Francis Upton added a comment -

          @Carlos, which release are you testing?

          Francis Upton added a comment - @Carlos, which release are you testing?

          Carlos Tadeu Panato added a comment -

          francisu 1.36, I did a downgrade to 1.35 and start working again

          Carlos Tadeu Panato added a comment - francisu 1.36, I did a downgrade to 1.35 and start working again

            francisu Francis Upton
            timff Tim Feng
            Votes:
            24 Vote for this issue
            Watchers:
            32 Start watching this issue

              Created:
              Updated:
              Resolved: