Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38481

AWS method change causes too many slaves to be launched

    XMLWordPrintable

Details

    Description

      Got the following error while trying to launch new EC2 build slave.
      I noticed that a new EC2 instance has been launched successfully, but the error still pop up.

      Stack trace
      javax.servlet.ServletException: java.lang.NoSuchMethodError: com.amazonaws.services.ec2.AmazonEC2.createTags(Lcom/amazonaws/services/ec2/model/CreateTagsRequest;)V
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:796)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:233)
      at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
      at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      at org.eclipse.jetty.server.Server.handle(Server.java:499)
      at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
      at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
      at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
      at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.NoSuchMethodError: com.amazonaws.services.ec2.AmazonEC2.createTags(Lcom/amazonaws/services/ec2/model/CreateTagsRequest;)V
      at hudson.plugins.ec2.SlaveTemplate.updateRemoteTags(SlaveTemplate.java:882)
      at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:585)
      at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:377)
      at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:515)
      at hudson.plugins.ec2.EC2Cloud.doProvision(EC2Cloud.java:340)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:324)
      at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:167)
      at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:100)
      at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
      at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      ... 58 more

      Attachments

        Issue Links

          Activity

            timff Tim Feng created issue -

            Bug is caused by upgrading Amazon Web Services SDK to 1.11.37

            bastih Sebastian Hillig added a comment - Bug is caused by upgrading Amazon Web Services SDK to 1.11.37
            hustonjs James Huston added a comment -

            I can confirm that downgrading the Amazon Web Services SDK plugin to 1.10.50 resolves this issue for me.

            hustonjs James Huston added a comment - I can confirm that downgrading the Amazon Web Services SDK plugin to 1.10.50 resolves this issue for me.
            xmiklis Mika Karjalainen made changes -
            Field Original Value New Value
            Component/s aws-java-sdk-plugin [ 21126 ]

            As a side note, this has bitten us quite a bit, because Jenkins proceeded to fill our AWS account up to the limit with EC2 instances. Doesn't look like there's a way to limit instances per IAM user on the AWS side either.

            Perhaps, as a defensive measure against future bugs, the code could also try to terminate the instance if provisioning fails?

            stephank Stéphan Kochen added a comment - As a side note, this has bitten us quite a bit, because Jenkins proceeded to fill our AWS account up to the limit with EC2 instances. Doesn't look like there's a way to limit instances per IAM user on the AWS side either. Perhaps, as a defensive measure against future bugs, the code could also try to terminate the instance if provisioning fails?

            Temporary workaround: remove tags from your slave configuration. (If your Jenkins is offline, remove the <tags> element from config.xml.)

            This way, tags are not included in the filter when Jenkins looks for existing instances, and will match instances for which tagging failed.

            (The filter is otherwise sufficiently narrow to not match unrelated instances in most cases. It includes, e.g., instance type, AMI and SSH key name, security group, etc.)

            stephank Stéphan Kochen added a comment - Temporary workaround: remove tags from your slave configuration. (If your Jenkins is offline, remove the <tags> element from config.xml.) This way, tags are not included in the filter when Jenkins looks for existing instances, and will match instances for which tagging failed. (The filter is otherwise sufficiently narrow to not match unrelated instances in most cases. It includes, e.g., instance type, AMI and SSH key name, security group, etc.)

            can we increase the priority on this? we also "has bitten us quite a bit, because Jenkins proceeded to fill our AWS account up to the limit with EC2 instances"

            cernenwein Chris Ernenwein added a comment - can we increase the priority on this? we also "has bitten us quite a bit, because Jenkins proceeded to fill our AWS account up to the limit with EC2 instances"
            pbecot01 Paul Becotte added a comment - - edited

            Same here. I also cannot figure out how to proceed. downgrading this plugin to 1.34 and the sdk to 1.10.50 results in slaves being terminated immediately after launch...with them both at the latest version, I get the above error message (even with tags removed from all my AMI configs).

            Any idea on how to get back up and running?

            EDIT: The second time I tried downgrading them both, the system worked properly.

            pbecot01 Paul Becotte added a comment - - edited Same here. I also cannot figure out how to proceed. downgrading this plugin to 1.34 and the sdk to 1.10.50 results in slaves being terminated immediately after launch...with them both at the latest version, I get the above error message (even with tags removed from all my AMI configs). Any idea on how to get back up and running? EDIT: The second time I tried downgrading them both, the system worked properly.
            mikz Michal Cichra added a comment - - edited

            We just paid 250$ in AWS because the plugin created 20 pretty big machines.
            I'm stunned that no one was able to issue some hotfix for days now. People can be paying a lot and not know. Or why it has priority `Minor`.

            I can confirm that downgrading the Amazon Web Services SDK plugin fixes the issue.

            mikz Michal Cichra added a comment - - edited We just paid 250$ in AWS because the plugin created 20 pretty big machines. I'm stunned that no one was able to issue some hotfix for days now. People can be paying a lot and not know. Or why it has priority `Minor`. I can confirm that downgrading the Amazon Web Services SDK plugin fixes the issue.
            jared_biel_bt Jared Biel added a comment -

            Another confirmation that downgrading EC2 to 1.34 and the AWS SDK to 1.10.50 works for now.

            I too am surprised that this is a minor. Luckily due to the instance size that we use and account restrictions this bug only cost us ~$4 + time.

            jared_biel_bt Jared Biel added a comment - Another confirmation that downgrading EC2 to 1.34 and the AWS SDK to 1.10.50 works for now. I too am surprised that this is a minor. Luckily due to the instance size that we use and account restrictions this bug only cost us ~$4 + time.
            chouaibi Hamza Chouaibi made changes -
            Priority Minor [ 4 ] Critical [ 2 ]
            ndeloof Nicolas De Loof added a comment - - edited

            Method seems unchanged in SDK 1.11.37, but actually a return type was added
            https://github.com/aws/aws-sdk-java/blob/1.11.37/aws-java-sdk-ec2/src/main/java/com/amazonaws/services/ec2/AmazonEC2.java#L1199
            which introduce binary compatibility issue.

            Proposed https://github.com/jenkinsci/ec2-plugin/pull/221

            ndeloof Nicolas De Loof added a comment - - edited Method seems unchanged in SDK 1.11.37, but actually a return type was added https://github.com/aws/aws-sdk-java/blob/1.11.37/aws-java-sdk-ec2/src/main/java/com/amazonaws/services/ec2/AmazonEC2.java#L1199 which introduce binary compatibility issue. Proposed https://github.com/jenkinsci/ec2-plugin/pull/221
            pgmjsd Joshua Davis added a comment -

            This bug is bad news... I just had to terminate over 40 m4.large instances.

            pgmjsd Joshua Davis added a comment - This bug is bad news... I just had to terminate over 40 m4.large instances.
            reid Peter Reid added a comment -

            I am also experiencing this issue

            reid Peter Reid added a comment - I am also experiencing this issue

            We also had this issue. Spun up slaves until we hit our EC2 instance cap. Not fun.

            Tested the PR fix linked above and it solved the problem. Release this fix soon!

            kulinski Chris Kulinski added a comment - We also had this issue. Spun up slaves until we hit our EC2 instance cap. Not fun. Tested the PR fix linked above and it solved the problem. Release this fix soon!

            Code changed in jenkins
            User: Nicolas De Loof
            Path:
            pom.xml
            src/test/java/hudson/plugins/ec2/SlaveTemplateUnitTest.java
            http://jenkins-ci.org/commit/ec2-plugin/c8983e4a8c3b784ae5a9a58127c88f3013db490d
            Log:
            JENKINS-38481 1.11.37 changed com.amazonaws.services.ec2.AmazonEC2#createTags return type, so incompatible binary change

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Nicolas De Loof Path: pom.xml src/test/java/hudson/plugins/ec2/SlaveTemplateUnitTest.java http://jenkins-ci.org/commit/ec2-plugin/c8983e4a8c3b784ae5a9a58127c88f3013db490d Log: JENKINS-38481 1.11.37 changed com.amazonaws.services.ec2.AmazonEC2#createTags return type, so incompatible binary change
            francisu Francis Upton made changes -
            Summary Error while launching new ec2 slave AWS method change causes too many slaves to be launched
            francisu Francis Upton added a comment -

            Fixed in release 1.36.

            francisu Francis Upton added a comment - Fixed in release 1.36.
            francisu Francis Upton made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Closed [ 6 ]
            reid Peter Reid added a comment -

            The release for this issue has not yet been released - can you please re-open this issue until the fix is available in the plugin repository.

            reid Peter Reid added a comment - The release for this issue has not yet been released - can you please re-open this issue until the fix is available in the plugin repository.
            francisu Francis Upton added a comment -

            Generally we close the issue when the release is made. The release has been made and will be available in the next 24 hours in the repo, this action is automatic. You can see the release in the source repository: https://github.com/jenkinsci/ec2-plugin/releases

            francisu Francis Upton added a comment - Generally we close the issue when the release is made. The release has been made and will be available in the next 24 hours in the repo, this action is automatic. You can see the release in the source repository: https://github.com/jenkinsci/ec2-plugin/releases
            reid Peter Reid added a comment -

            OK, cool. Just wanted clarification on that.

            reid Peter Reid added a comment - OK, cool. Just wanted clarification on that.
            danielbeck Daniel Beck made changes -
            Link This issue is duplicated by JENKINS-38556 [ JENKINS-38556 ]

            timff just stumbled into this after seeing that log, there's something I don't understand though:

            you say

            I noticed that a new EC2 instance has been launched successfully, but the error still pop up.

            So I understand you're saying basically this stack is log spam, but you still file it as Critical? Is this impacting any visible feature of the plugin in the end?

            Thanks

            batmat Baptiste Mathus added a comment - timff just stumbled into this after seeing that log, there's something I don't understand though: you say I noticed that a new EC2 instance has been launched successfully , but the error still pop up. So I understand you're saying basically this stack is log spam, but you still file it as Critical? Is this impacting any visible feature of the plugin in the end? Thanks

            OK, forget it. I think I understand re-reading the title.

            batmat Baptiste Mathus added a comment - OK, forget it. I think I understand re-reading the title.

            looks like now with this fix none are being launch

            ctadeu Carlos Tadeu Panato added a comment - looks like now with this fix none are being launch
            francisu Francis Upton added a comment -

            @Carlos, which release are you testing?

            francisu Francis Upton added a comment - @Carlos, which release are you testing?

            francisu 1.36, I did a downgrade to 1.35 and start working again

            ctadeu Carlos Tadeu Panato added a comment - francisu 1.36, I did a downgrade to 1.35 and start working again

            People

              francisu Francis Upton
              timff Tim Feng
              Votes:
              24 Vote for this issue
              Watchers:
              32 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: