Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-26854

EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Blocker
    • Resolution: Fixed
    • ec2-plugin
    • None
    • * EC2 plugin version 1.26.
      * Jenkins 1.580.2 running inside the official Jenkins Docker LTS image.
      * Host O/S: Ubuntu 14.04 LTS 64-bit on an EC2 master.
      * EC2 rights are conferred via an EC2 InstanceProfile.

    Description

      After Jenkins first starts it is able to launch EC2 slaves, both manually and when jobs indicate they need to use the slave label.

      A few hours later (not sure how long, maybe 24 hours?) slaves no longer start, manually or automatically. In "Manage Jenkins -> System Log -> All Jenkins Logs" the following error occurs repeatedly. Restarting Jenkins solves the problem.

      Started EC2 alive slaves monitor
      Feb 09, 2015 5:14:47 AM INFO hudson.model.AsyncPeriodicWork$1 run
      Finished EC2 alive slaves monitor. 0 ms
      Feb 09, 2015 5:15:51 AM INFO hudson.plugins.ec2.EC2Cloud provision
      Excess workload after pending Spot instances: 1
      Feb 09, 2015 5:15:53 AM WARNING hudson.plugins.ec2.EC2Cloud provision
      Failed to count the # of live instances on EC2
      com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 59f7935f-15f0-455c-a6f1-f6057f5ffc77)
      	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:886)
      	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:484)
      	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:256)
      	at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:8798)
      	at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4137)
      	at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:8087)
      	at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:228)
      	at hudson.plugins.ec2.EC2Cloud.addProvisionedSlave(EC2Cloud.java:299)
      	at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:389)
      	at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:281)
      	at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51)
      	at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:368)
      	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)

      I also don't understand the log statement Excess workload after pending Spot instances: 1 as I have not ticked the "Use Spot instance" tick box.

      In my cloud settings I have ticked the "Use EC2 instance profile to obtain credentials" and have set both the access key and secret key values to "THIS VALUE IS NOT USED - THE INSTANCE PROFILE IS USED INSTEAD".

      Attachments

        Activity

          I've been looking at the underlying AWS SDK code and it looks like it already has built in support for refreshing the credentials before they expire, and that this support is already being used by the Jenkins EC2 plugin. The only things I can think of are either that NTP isn't working properly and clock drift causes the problem, or that the synchronous mode of credential refresh being used by the Jenkins EC2 plugin doesn't work for some reason and perhaps the asynchronous background thread mode needs to be used? I will investigate NTP on my side.

          ximon18 Ximon Eighteen added a comment - I've been looking at the underlying AWS SDK code and it looks like it already has built in support for refreshing the credentials before they expire, and that this support is already being used by the Jenkins EC2 plugin. The only things I can think of are either that NTP isn't working properly and clock drift causes the problem, or that the synchronous mode of credential refresh being used by the Jenkins EC2 plugin doesn't work for some reason and perhaps the asynchronous background thread mode needs to be used? I will investigate NTP on my side.
          ximon18 Ximon Eighteen added a comment - - edited

          My investigation into NTP hasn't found any problems:

          ubuntu@ip-172-30-0-149:~$ ntpq -pn
               remote           refid      st t when poll reach   delay   offset  jitter
          ==============================================================================
          -87.232.1.40     62.231.32.35     4 u 1037 1024  377    4.120   -1.394   2.634
          +78.143.174.10   193.1.219.116    2 u  382 1024  377   71.766   -7.342  48.358
          -86.43.77.42     193.120.10.3     2 u  624 1024  337   40.723    8.783   0.787
          +85.91.1.180     195.66.241.2     2 u   94 1024  377    1.812   -3.254   0.835
          *91.189.89.199   192.93.2.20      2 u  741 1024  377   10.945   -1.322   0.847
          

          The * shows the NTP server being used, and the reach 377 value shows that the NTP daemon was repeatedly able to contact the remote time server, the stratum number is low which is good, and the jitter and offset values are low which is good. Executing 'date' in both the Ubuntu host and the Docker container yields the same date and time.

          Unrelated to the NTP investigation, but related to my point about restarting Jenkins in the initial post, if I invoke http://<jenkins>/safeRestart then the slave is started correctly after Jenkins restarts, without my fixing any clocks.

          ximon18 Ximon Eighteen added a comment - - edited My investigation into NTP hasn't found any problems: ubuntu@ip-172-30-0-149:~$ ntpq -pn remote refid st t when poll reach delay offset jitter ============================================================================== -87.232.1.40 62.231.32.35 4 u 1037 1024 377 4.120 -1.394 2.634 +78.143.174.10 193.1.219.116 2 u 382 1024 377 71.766 -7.342 48.358 -86.43.77.42 193.120.10.3 2 u 624 1024 337 40.723 8.783 0.787 +85.91.1.180 195.66.241.2 2 u 94 1024 377 1.812 -3.254 0.835 *91.189.89.199 192.93.2.20 2 u 741 1024 377 10.945 -1.322 0.847 The * shows the NTP server being used, and the reach 377 value shows that the NTP daemon was repeatedly able to contact the remote time server, the stratum number is low which is good, and the jitter and offset values are low which is good. Executing 'date' in both the Ubuntu host and the Docker container yields the same date and time. Unrelated to the NTP investigation, but related to my point about restarting Jenkins in the initial post, if I invoke http://<jenkins>/safeRestart then the slave is started correctly after Jenkins restarts, without my fixing any clocks.
          ximon18 Ximon Eighteen added a comment - - edited

          I have no reason to think this is the cause of my problem but I just noticed that the plugin is built using v1.8.3 of the Java SDK while the latest version is 1.9.17, in theory at least there could be a bug fix in the newer versions. I looked through the release notes of the interim versions but didn't find an obvious bug fix that could be related to this issue.

          Update: Actually the release 1.8.10 of the Java SDK added the InstanceProfilerCredentialsProvider(true) behaviour I refer to above, but this was not directly mentioned in the release notes. This release was not a happy release, two hot fixes 1.8.10.1 and 1.8.10.2 were released in the following week.

          Update: If I build the EC2 plugin with the latest 1.9.17 SDK version it fails with an HTTP 401 Auth error which I haven't tracked down yet. I suspect this is because my AWS IAM InstanceProfile role does not include a permission which newer versions of the SDK require, but I haven't determined which permission is missing yet.

          ximon18 Ximon Eighteen added a comment - - edited I have no reason to think this is the cause of my problem but I just noticed that the plugin is built using v1.8.3 of the Java SDK while the latest version is 1.9.17, in theory at least there could be a bug fix in the newer versions. I looked through the release notes of the interim versions but didn't find an obvious bug fix that could be related to this issue. Update: Actually the release 1.8.10 of the Java SDK added the InstanceProfilerCredentialsProvider(true) behaviour I refer to above, but this was not directly mentioned in the release notes . This release was not a happy release, two hot fixes 1.8.10.1 and 1.8.10.2 were released in the following week. Update: If I build the EC2 plugin with the latest 1.9.17 SDK version it fails with an HTTP 401 Auth error which I haven't tracked down yet. I suspect this is because my AWS IAM InstanceProfile role does not include a permission which newer versions of the SDK require, but I haven't determined which permission is missing yet.
          ximon18 Ximon Eighteen added a comment - - edited

          I'm going to put some logging into a subclass of InstanceProfilerCredentialsProvider() because I suspect that for some reason the credentials are not being refreshed by this class... will let you know what I find out.

          Update: Indeed the EC2 credential refresh functionality of the AWS Java SDK is not invoked. See attached jenkins.log (look for lines containing "Ximon:") and gitdiff.txt. Rough highlights from the log (might not be entirely accurate, trying to remember what I did last night):

          1. 11:33:00 UTC: Jenkins finished responding to a /safeRestart request that I performed.
          2. 11:34:36 UTC: I instructed Jenkins to launch a new EC2 slave. The EC2 plugin fetched the EC2 credentials as part of launching the instance.
          3. 11:45:46 UTC: I instructed Jenkins to terminate the EC2 slave. The EC2 plugin did not refetch the credentials.
          4. 11:46:09 UTC: I instructed Jenkins to launch a new EC2 slave. The EC2 plugin did not refetch the credentials.
          5. 12:22:46 UTC: The EC2 plugin correctly stopped the EC2 slave instance after the idle timeout expired. The EC2 plugin did not refetch the credentials.
          6. 04:32:58 UTC: UNRELATED BUG: Jenkins logged "Making <NODE NAME> (i-a2a41545) offline because it’s not responding". Why is this logged over 4 hours after the EC2 plugin stopped the slave?
          7. 05:30:22 UTC: I refreshed the Jenkins log web page which seems to have caused the EC2 plugin to attempt to update its knowledge about the state of the slave. By this point the EC2 credentials have expired, but the EC2 plugin did not refetch the credentials. The call to the EC2 API failed with HTTP 400 AmazonServiceException "Request has expired".
          ximon18 Ximon Eighteen added a comment - - edited I'm going to put some logging into a subclass of InstanceProfilerCredentialsProvider() because I suspect that for some reason the credentials are not being refreshed by this class... will let you know what I find out. Update: Indeed the EC2 credential refresh functionality of the AWS Java SDK is not invoked. See attached jenkins.log (look for lines containing "Ximon:") and gitdiff.txt . Rough highlights from the log (might not be entirely accurate, trying to remember what I did last night): 11:33:00 UTC: Jenkins finished responding to a /safeRestart request that I performed. 11:34:36 UTC: I instructed Jenkins to launch a new EC2 slave. The EC2 plugin fetched the EC2 credentials as part of launching the instance. 11:45:46 UTC: I instructed Jenkins to terminate the EC2 slave. The EC2 plugin did not refetch the credentials. 11:46:09 UTC: I instructed Jenkins to launch a new EC2 slave. The EC2 plugin did not refetch the credentials. 12:22:46 UTC: The EC2 plugin correctly stopped the EC2 slave instance after the idle timeout expired. The EC2 plugin did not refetch the credentials. 04:32:58 UTC: UNRELATED BUG: Jenkins logged "Making <NODE NAME> (i-a2a41545) offline because it’s not responding" . Why is this logged over 4 hours after the EC2 plugin stopped the slave? 05:30:22 UTC: I refreshed the Jenkins log web page which seems to have caused the EC2 plugin to attempt to update its knowledge about the state of the slave. By this point the EC2 credentials have expired, but the EC2 plugin did not refetch the credentials. The call to the EC2 API failed with HTTP 400 AmazonServiceException "Request has expired".
          ximon18 Ximon Eighteen added a comment - - edited

          I tried building the EC2 plugin with AWS Java SDK 1.8.11, the first stable release after 1.8.10 that introduced the new InstanceProfileCredentialsProvider(true) functionality. This solves the missing refetch of credentials, causing the SDK to check them once a minute. However, it fails with the Auth error I referred to above:

          com.amazonaws.AmazonServiceException: AWS was not able to validate the provided access credentials (Service: AmazonEC2; Status Code: 401; Error Code: AuthFailure; Request ID: 0ad2caa1-4f05-41e7-b168-3dc37940265b)
          	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032)
          	at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687)
          	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441)
          	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292)
          	at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225)
          	at com.amazonaws.services.ec2.AmazonEC2Client.describeKeyPairs(AmazonEC2Client.java:6321)
          	at com.amazonaws.services.ec2.AmazonEC2Client.describeKeyPairs(AmazonEC2Client.java:8879)
          	at hudson.plugins.ec2.EC2PrivateKey.find(EC2PrivateKey.java:135)
          	at hudson.plugins.ec2.SlaveTemplate.getKeyPair(SlaveTemplate.java:719)
          	at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:303)
          	at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:287)
          	at hudson.plugins.ec2.EC2Cloud.doProvision(EC2Cloud.java:283)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:606)
          	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
          	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
          	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
          	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
          	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:745)
          	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:875)
          	at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
          	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:745)
          	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:875)
          	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:648)
          	at org.kohsuke.stapler.Stapler.service(Stapler.java:237)
          	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
          	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
          	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
          	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
          	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
          	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
          	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
          	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:86)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
          	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
          	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
          	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
          	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
          	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
          	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
          	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
          	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
          	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
          	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
          	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
          	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
          	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
          	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
          	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
          	at org.eclipse.jetty.server.Server.handle(Server.java:370)
          	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
          	at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960)
          	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021)
          	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
          	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
          	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
          	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
          	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
          	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          	at java.lang.Thread.run(Thread.java:745)
          

          I suspect this is something to do with my IAM configuration, and not with the EC2 plugin or the Java SDK. So I suspect that upgrading the SDK to 1.8.11 and invoking the InstanceProfileCredentialProvider(true) constructor will solve this issue for other people.

          ximon18 Ximon Eighteen added a comment - - edited I tried building the EC2 plugin with AWS Java SDK 1.8.11, the first stable release after 1.8.10 that introduced the new InstanceProfileCredentialsProvider(true) functionality. This solves the missing refetch of credentials, causing the SDK to check them once a minute. However, it fails with the Auth error I referred to above: com.amazonaws.AmazonServiceException: AWS was not able to validate the provided access credentials (Service: AmazonEC2; Status Code: 401; Error Code: AuthFailure; Request ID: 0ad2caa1-4f05-41e7-b168-3dc37940265b) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225) at com.amazonaws.services.ec2.AmazonEC2Client.describeKeyPairs(AmazonEC2Client.java:6321) at com.amazonaws.services.ec2.AmazonEC2Client.describeKeyPairs(AmazonEC2Client.java:8879) at hudson.plugins.ec2.EC2PrivateKey.find(EC2PrivateKey.java:135) at hudson.plugins.ec2.SlaveTemplate.getKeyPair(SlaveTemplate.java:719) at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:303) at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:287) at hudson.plugins.ec2.EC2Cloud.doProvision(EC2Cloud.java:283) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:745) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:875) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:745) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:875) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:648) at org.kohsuke.stapler.Stapler.service(Stapler.java:237) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:86) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) I suspect this is something to do with my IAM configuration, and not with the EC2 plugin or the Java SDK. So I suspect that upgrading the SDK to 1.8.11 and invoking the InstanceProfileCredentialProvider(true) constructor will solve this issue for other people.

          I have created a minimal pull request. See: https://github.com/jenkinsci/ec2-plugin/pull/131

          ximon18 Ximon Eighteen added a comment - I have created a minimal pull request. See: https://github.com/jenkinsci/ec2-plugin/pull/131
          ximon18 Ximon Eighteen added a comment - - edited

          Ah, the AuthFailure may affect others too. See: https://forums.aws.amazon.com/thread.jspa?messageID=574914&tstart=0. I'm seeing this issue in the eu-west-1 region.

          Update: Applying the force previous signer configuration solved this issue for me, that is this part of the forum article that I referred to:

          clientConfiguration.setSignerOverride(“QueryStringSignerType”);
          AmazonEC2 ec2 = new AmazonEC2Client(configuration);
          
          ximon18 Ximon Eighteen added a comment - - edited Ah, the AuthFailure may affect others too. See: https://forums.aws.amazon.com/thread.jspa?messageID=574914&tstart=0 . I'm seeing this issue in the eu-west-1 region. Update: Applying the force previous signer configuration solved this issue for me, that is this part of the forum article that I referred to: clientConfiguration.setSignerOverride(“QueryStringSignerType”); AmazonEC2 ec2 = new AmazonEC2Client(configuration);
          martinfr62 martinfr62 added a comment - - edited

          Seeing same issue, restarting jenkins from the web browser cleared the issue, so clearly some sort of issue in the ec2 plugin itself, not an issue with the date on the instance hosting jenkins.

          martinfr62 martinfr62 added a comment - - edited Seeing same issue, restarting jenkins from the web browser cleared the issue, so clearly some sort of issue in the ec2 plugin itself, not an issue with the date on the instance hosting jenkins.

          To be clear, I have a working (well seems to work, I must admit it's not thoroughly tested yet) solution, but I'm not sure if it's wise for others to use. My solution is the pull request here, with the following tiny modification:

          └─(.../scratch/ximon-ec2-plugin)─> git diff
          diff --git a/src/main/java/hudson/plugins/ec2/EC2Cloud.java b/src/main/java/hudson/plugins/ec2/EC2Cloud.java
          index 8891999..7feed81 100644
          --- a/src/main/java/hudson/plugins/ec2/EC2Cloud.java
          +++ b/src/main/java/hudson/plugins/ec2/EC2Cloud.java
          @@ -476,6 +476,7 @@ public abstract class EC2Cloud extends Cloud {
               public synchronized static AmazonEC2 connect(AWSCredentialsProvider credentialsProvider, URL endpoint) {
                   awsCredentialsProvider = credentialsProvider;
                   ClientConfiguration config = new ClientConfiguration();
          +        config.setSignerOverride("QueryStringSignerType");
                   ProxyConfiguration proxyConfig = Jenkins.getInstance().proxy;
                   Proxy proxy = proxyConfig == null ? Proxy.NO_PROXY : proxyConfig.createProxy(endpoint.getHost());
                   if (! proxy.equals(Proxy.NO_PROXY) && proxy.address() instanceof InetSocketAddress) {
          
          ximon18 Ximon Eighteen added a comment - To be clear, I have a working (well seems to work, I must admit it's not thoroughly tested yet) solution, but I'm not sure if it's wise for others to use. My solution is the pull request here , with the following tiny modification: └─(.../scratch/ximon-ec2-plugin)─> git diff diff --git a/src/main/java/hudson/plugins/ec2/EC2Cloud.java b/src/main/java/hudson/plugins/ec2/EC2Cloud.java index 8891999..7feed81 100644 --- a/src/main/java/hudson/plugins/ec2/EC2Cloud.java +++ b/src/main/java/hudson/plugins/ec2/EC2Cloud.java @@ -476,6 +476,7 @@ public abstract class EC2Cloud extends Cloud { public synchronized static AmazonEC2 connect(AWSCredentialsProvider credentialsProvider, URL endpoint) { awsCredentialsProvider = credentialsProvider; ClientConfiguration config = new ClientConfiguration(); + config.setSignerOverride( "QueryStringSignerType" ); ProxyConfiguration proxyConfig = Jenkins.getInstance().proxy; Proxy proxy = proxyConfig == null ? Proxy.NO_PROXY : proxyConfig.createProxy(endpoint.getHost()); if (! proxy.equals(Proxy.NO_PROXY) && proxy.address() instanceof InetSocketAddress) {
          martinfr62 martinfr62 added a comment -

          Pulled copy of 131 and built locally - just realized I had to apply above change as well. Will rebuild, deploy and test myself.

          martinfr62 martinfr62 added a comment - Pulled copy of 131 and built locally - just realized I had to apply above change as well. Will rebuild, deploy and test myself.
          martinfr62 martinfr62 added a comment -

          So still seeing the issue - this time though when the instance is terminated after it times out - instance then is left running in AWS, but dropped from jenkins, and cannot start any more instances until the aws instances have been terminated.

          Failed to fetch EC2 instance: i-2107a6db
          com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 1b435e23-f37f-4433-9526-f7226801cab9)
          at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032)
          at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687)
          at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441)
          at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292)
          at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225)
          at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541)
          at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:208)
          at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:354)
          at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:326)
          at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:42)
          at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:53)
          at java.lang.Thread.run(Thread.java:745)

          Feb 12, 2015 6:31:26 AM INFO hudson.plugins.ec2.EC2SlaveMonitor execute
          EC2 instance is dead: i-2107a6db
          Feb 12, 2015 6:31:28 AM WARNING hudson.plugins.ec2.EC2AbstractSlave getInstance
          Failed to fetch EC2 instance: i-2107a6db
          com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 8e53415f-6d34-4669-a76e-536d9cedcf37)
          at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032)
          at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687)
          at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441)
          at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292)
          at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225)
          at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541)
          at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:208)
          at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:354)
          at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:326)
          at hudson.plugins.ec2.EC2OndemandSlave.terminate(EC2OndemandSlave.java:63)
          at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:44)
          at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:53)
          at java.lang.Thread.run(Thread.java:745)

          Feb 12, 2015 6:31:28 AM INFO hudson.plugins.ec2.EC2OndemandSlave terminate
          EC2 instance already terminated: i-2107a6db
          Feb 12, 2015 6:31:28 AM INFO hudson.plugins.ec2.EC2OndemandSlave terminate
          Removed EC2 instance from jenkins master: i-2107a6db
          Feb 12, 2015 6:31:30 AM WARNING hudson.plugins.ec2.EC2AbstractSlave getInstance
          Failed to fetch EC2 instance: i-1a04a5e0
          com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 4a7feb85-c0ff-4dda-9f88-f605513408a8)
          at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032)
          at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687)
          at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441)
          at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292)
          at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225)
          at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541)
          at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:208)
          at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:354)
          at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:326)
          at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:42)
          at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:53)
          at java.lang.Thread.run(Thread.java:745)

          Feb 12, 2015 6:31:30 AM INFO hudson.plugins.ec2.EC2SlaveMonitor execute
          EC2 instance is dead: i-1a04a5e0
          Feb 12, 2015 6:31:32 AM WARNING hudson.plugins.ec2.EC2AbstractSlave getInstance
          Failed to fetch EC2 instance: i-1a04a5e0
          com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: be105033-7ff8-4ef4-9dbc-860e8cd6a73a)
          at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032)
          at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687)
          at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441)
          at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292)
          at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225)
          at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541)
          at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:208)
          at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:354)
          at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:326)
          at hudson.plugins.ec2.EC2OndemandSlave.terminate(EC2OndemandSlave.java:63)
          at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:44)
          at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:53)
          at java.lang.Thread.run(Thread.java:745)

          Feb 12, 2015 6:31:32 AM INFO hudson.plugins.ec2.EC2OndemandSlave terminate

          martinfr62 martinfr62 added a comment - So still seeing the issue - this time though when the instance is terminated after it times out - instance then is left running in AWS, but dropped from jenkins, and cannot start any more instances until the aws instances have been terminated. Failed to fetch EC2 instance: i-2107a6db com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 1b435e23-f37f-4433-9526-f7226801cab9) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541) at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:208) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:354) at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:326) at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:42) at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:53) at java.lang.Thread.run(Thread.java:745) Feb 12, 2015 6:31:26 AM INFO hudson.plugins.ec2.EC2SlaveMonitor execute EC2 instance is dead: i-2107a6db Feb 12, 2015 6:31:28 AM WARNING hudson.plugins.ec2.EC2AbstractSlave getInstance Failed to fetch EC2 instance: i-2107a6db com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 8e53415f-6d34-4669-a76e-536d9cedcf37) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541) at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:208) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:354) at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:326) at hudson.plugins.ec2.EC2OndemandSlave.terminate(EC2OndemandSlave.java:63) at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:44) at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:53) at java.lang.Thread.run(Thread.java:745) Feb 12, 2015 6:31:28 AM INFO hudson.plugins.ec2.EC2OndemandSlave terminate EC2 instance already terminated: i-2107a6db Feb 12, 2015 6:31:28 AM INFO hudson.plugins.ec2.EC2OndemandSlave terminate Removed EC2 instance from jenkins master: i-2107a6db Feb 12, 2015 6:31:30 AM WARNING hudson.plugins.ec2.EC2AbstractSlave getInstance Failed to fetch EC2 instance: i-1a04a5e0 com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 4a7feb85-c0ff-4dda-9f88-f605513408a8) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541) at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:208) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:354) at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:326) at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:42) at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:53) at java.lang.Thread.run(Thread.java:745) Feb 12, 2015 6:31:30 AM INFO hudson.plugins.ec2.EC2SlaveMonitor execute EC2 instance is dead: i-1a04a5e0 Feb 12, 2015 6:31:32 AM WARNING hudson.plugins.ec2.EC2AbstractSlave getInstance Failed to fetch EC2 instance: i-1a04a5e0 com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: be105033-7ff8-4ef4-9dbc-860e8cd6a73a) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541) at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:208) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:354) at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:326) at hudson.plugins.ec2.EC2OndemandSlave.terminate(EC2OndemandSlave.java:63) at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:44) at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:53) at java.lang.Thread.run(Thread.java:745) Feb 12, 2015 6:31:32 AM INFO hudson.plugins.ec2.EC2OndemandSlave terminate
          martinfr62 martinfr62 added a comment -

          Looks like EC2Cloud.connect() caches the AmazonEC2 connection - testing patch which obtains the connection every time. Will taker a while, have to let the terminate event fire for the instances.

          martinfr62 martinfr62 added a comment - Looks like EC2Cloud.connect() caches the AmazonEC2 connection - testing patch which obtains the connection every time. Will taker a while, have to let the terminate event fire for the instances.
          martinfr62 martinfr62 added a comment -

          Okay - here's what i did in patch form against the pull 131 (so includes your Ximon).

          Removed the transient connection object and created a new one every time in connect() and then updated/added some more logging so i could tell when the terminate was about to fire. I also found a typo in the log by doing this in EC2RetentionStrategy - was treating freeSecondsLeft as milliseconds - not seconds - so fixed that also.


          src/main/java/hudson/plugins/ec2/EC2Cloud.java | 9 +++++----
          src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java | 6 +++++-
          2 files changed, 10 insertions, 5 deletions

          diff --git a/src/main/java/hudson/plugins/ec2/EC2Cloud.java b/src/main/java/hudson/plugins/ec2/EC2Cloud.java
          index 8891999..ca102e6 100644
          — a/src/main/java/hudson/plugins/ec2/EC2Cloud.java
          +++ b/src/main/java/hudson/plugins/ec2/EC2Cloud.java
          @@ -113,7 +113,7 @@ public abstract class EC2Cloud extends Cloud {
          private final List<? extends SlaveTemplate> templates;
          private transient KeyPair usableKeyPair;

          • protected transient AmazonEC2 connection;
            + // protected transient AmazonEC2 connection;

          private static AWSCredentialsProvider awsCredentialsProvider;

          @@ -460,9 +460,9 @@ public abstract class EC2Cloud extends Cloud {
          */
          public synchronized AmazonEC2 connect() throws AmazonClientException {
          try {

          • if (connection == null) { - connection = connect(createCredentialsProvider(), getEc2EndpointUrl()); - }

            + // if (connection == null)

            { + AmazonEC2 connection = connect(createCredentialsProvider(), getEc2EndpointUrl()); + // }

            return connection;
            } catch (IOException e) {
            throw new AmazonClientException("Failed to retrieve the endpoint",e);
            @@ -476,6 +476,7 @@ public abstract class EC2Cloud extends Cloud {
            public synchronized static AmazonEC2 connect(AWSCredentialsProvider credentialsProvider, URL endpoint) {
            awsCredentialsProvider = credentialsProvider;
            ClientConfiguration config = new ClientConfiguration();
            + config.setSignerOverride("QueryStringSignerType");
            ProxyConfiguration proxyConfig = Jenkins.getInstance().proxy;
            Proxy proxy = proxyConfig == null ? Proxy.NO_PROXY : proxyConfig.createProxy(endpoint.getHost());
            if (! proxy.equals(Proxy.NO_PROXY) && proxy.address() instanceof InetSocketAddress) {
            diff --git a/src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java b/src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java
            index 22e185a..52d8f49 100644

              • a/src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java
                +++ b/src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java
                @@ -94,6 +94,8 @@ public class EC2RetentionStrategy extends RetentionStrategy<EC2Computer> {
                final long idleMilliseconds = System.currentTimeMillis() - c.getIdleStartMilliseconds();
                if (idleTerminationMinutes > 0) {
                // TODO: really think about the right strategy here, see JENKINS-23792
                + LOGGER.fine("Test Idle timeout of "+c.getName() + " after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes");
                +
                if (idleMilliseconds > TimeUnit2.MINUTES.toMillis(idleTerminationMinutes)) {
                LOGGER.info("Idle timeout of "+c.getName() + " after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes");
                c.getNode().idleTimeout();
                @@ -110,8 +112,10 @@ public class EC2RetentionStrategy extends RetentionStrategy<EC2Computer>
                Unknown macro: { final int freeSecondsLeft = (60*60) - (int)(TimeUnit2.SECONDS.convert(uptime, TimeUnit2.MILLISECONDS) % (60*60)); // if we have less "free" (aka already paid for) time left than our idle time, stop/terminate the instance // See JENKINS-23821+ LOGGER.fine("Test Idle timeout of "+c.getName()+" after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes, with " + TimeUnit2.SECONDS.toMinutes(freeSecondsLeft) + " minutes remaining in billing period");+ if (freeSecondsLeft <= (Math.abs(idleTerminationMinutes*60))) { - LOGGER.info("Idle timeout of "+c.getName()+" after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes, with " + TimeUnit2.MILLISECONDS.toMinutes(freeSecondsLeft) + " minutes remaining in billing period"); + LOGGER.info("Idle timeout of "+c.getName()+" after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes, with " + TimeUnit2.SECONDS.toMinutes(freeSecondsLeft) + " minutes remaining in billing period"); c.getNode().idleTimeout(); } }

            • 1.9.4.msysgit.2
          martinfr62 martinfr62 added a comment - Okay - here's what i did in patch form against the pull 131 (so includes your Ximon). Removed the transient connection object and created a new one every time in connect() and then updated/added some more logging so i could tell when the terminate was about to fire. I also found a typo in the log by doing this in EC2RetentionStrategy - was treating freeSecondsLeft as milliseconds - not seconds - so fixed that also. — src/main/java/hudson/plugins/ec2/EC2Cloud.java | 9 +++++---- src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java | 6 +++++- 2 files changed, 10 insertions , 5 deletions diff --git a/src/main/java/hudson/plugins/ec2/EC2Cloud.java b/src/main/java/hudson/plugins/ec2/EC2Cloud.java index 8891999..ca102e6 100644 — a/src/main/java/hudson/plugins/ec2/EC2Cloud.java +++ b/src/main/java/hudson/plugins/ec2/EC2Cloud.java @@ -113,7 +113,7 @@ public abstract class EC2Cloud extends Cloud { private final List<? extends SlaveTemplate> templates; private transient KeyPair usableKeyPair; protected transient AmazonEC2 connection; + // protected transient AmazonEC2 connection; private static AWSCredentialsProvider awsCredentialsProvider; @@ -460,9 +460,9 @@ public abstract class EC2Cloud extends Cloud { */ public synchronized AmazonEC2 connect() throws AmazonClientException { try { if (connection == null) { - connection = connect(createCredentialsProvider(), getEc2EndpointUrl()); - } + // if (connection == null) { + AmazonEC2 connection = connect(createCredentialsProvider(), getEc2EndpointUrl()); + // } return connection; } catch (IOException e) { throw new AmazonClientException("Failed to retrieve the endpoint",e); @@ -476,6 +476,7 @@ public abstract class EC2Cloud extends Cloud { public synchronized static AmazonEC2 connect(AWSCredentialsProvider credentialsProvider, URL endpoint) { awsCredentialsProvider = credentialsProvider; ClientConfiguration config = new ClientConfiguration(); + config.setSignerOverride("QueryStringSignerType"); ProxyConfiguration proxyConfig = Jenkins.getInstance().proxy; Proxy proxy = proxyConfig == null ? Proxy.NO_PROXY : proxyConfig.createProxy(endpoint.getHost()); if (! proxy.equals(Proxy.NO_PROXY) && proxy.address() instanceof InetSocketAddress) { diff --git a/src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java b/src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java index 22e185a..52d8f49 100644 a/src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java +++ b/src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java @@ -94,6 +94,8 @@ public class EC2RetentionStrategy extends RetentionStrategy<EC2Computer> { final long idleMilliseconds = System.currentTimeMillis() - c.getIdleStartMilliseconds(); if (idleTerminationMinutes > 0) { // TODO: really think about the right strategy here, see JENKINS-23792 + LOGGER.fine("Test Idle timeout of "+c.getName() + " after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes"); + if (idleMilliseconds > TimeUnit2.MINUTES.toMillis(idleTerminationMinutes)) { LOGGER.info("Idle timeout of "+c.getName() + " after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes"); c.getNode().idleTimeout(); @@ -110,8 +112,10 @@ public class EC2RetentionStrategy extends RetentionStrategy<EC2Computer> Unknown macro: { final int freeSecondsLeft = (60*60) - (int)(TimeUnit2.SECONDS.convert(uptime, TimeUnit2.MILLISECONDS) % (60*60)); // if we have less "free" (aka already paid for) time left than our idle time, stop/terminate the instance // See JENKINS-23821+ LOGGER.fine("Test Idle timeout of "+c.getName()+" after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes, with " + TimeUnit2.SECONDS.toMinutes(freeSecondsLeft) + " minutes remaining in billing period");+ if (freeSecondsLeft <= (Math.abs(idleTerminationMinutes*60))) { - LOGGER.info("Idle timeout of "+c.getName()+" after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes, with " + TimeUnit2.MILLISECONDS.toMinutes(freeSecondsLeft) + " minutes remaining in billing period"); + LOGGER.info("Idle timeout of "+c.getName()+" after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes, with " + TimeUnit2.SECONDS.toMinutes(freeSecondsLeft) + " minutes remaining in billing period"); c.getNode().idleTimeout(); } } 1.9.4.msysgit.2

          FYI: I'm running into the same issue on Jenkins ver. 1.580.3 with Amazon EC2 Plugin v1.26 on Amazon Linux with an IAM Instance Profile assigned to the instance.

          Would be great to get this fixed so we don't have to restart our Jenkins master every 6 hours.

          vrivellino Vincent Rivellino added a comment - FYI: I'm running into the same issue on Jenkins ver. 1.580.3 with Amazon EC2 Plugin v1.26 on Amazon Linux with an IAM Instance Profile assigned to the instance. Would be great to get this fixed so we don't have to restart our Jenkins master every 6 hours.
          cfair chris fair added a comment -

          I built Ximon's branch above, along with Martin's changes, and haven't seen this issue occur again. I did have to add 'config.setSignerOverride("QueryStringSignerType");' and I'm in region us-east-1.

          cfair chris fair added a comment - I built Ximon's branch above, along with Martin's changes, and haven't seen this issue occur again. I did have to add 'config.setSignerOverride("QueryStringSignerType");' and I'm in region us-east-1.

          I've updated my pull request with a new boolean EC2 Cloud parameter to enable using the query string signer, and the fix from Martin (though I just realized I only mentioned the query string signer fix in the commit message, wasn't intentional not to mention that...). Just thought it would be handy to put the changes in one place for anyone else reading this issue until the fix makes it into a proper release build, and to make sure that if the query string signer type is needed that users can enable it.

          ximon18 Ximon Eighteen added a comment - I've updated my pull request with a new boolean EC2 Cloud parameter to enable using the query string signer, and the fix from Martin (though I just realized I only mentioned the query string signer fix in the commit message, wasn't intentional not to mention that...). Just thought it would be handy to put the changes in one place for anyone else reading this issue until the fix makes it into a proper release build, and to make sure that if the query string signer type is needed that users can enable it.
          martinfr62 martinfr62 added a comment - - edited

          Tried pulling 131 and patching current master - it fails to connect to aws.

          Pulled current master - built and depoyed - connects okay (but has normal bug in it)
          Pulled 131 and split the patch in two - first half (ie prior 131) works - can connect to aws
          Apply second patch as a straight merge - and it builds - but cannot connect

          Next tried pull master, apply first half - and then apply my patch alone - this works.

          Looks like something in the second patch causes aws to fail.

          Let me know if you need a stack trace or anything if you cannot reproduce - but happens every time for me.

          Try executing 'Test Connection' from jenkins config page for ec2 - fails immediately for both profile and real keys.

          martinfr62 martinfr62 added a comment - - edited Tried pulling 131 and patching current master - it fails to connect to aws. Pulled current master - built and depoyed - connects okay (but has normal bug in it) Pulled 131 and split the patch in two - first half (ie prior 131) works - can connect to aws Apply second patch as a straight merge - and it builds - but cannot connect Next tried pull master, apply first half - and then apply my patch alone - this works. Looks like something in the second patch causes aws to fail. Let me know if you need a stack trace or anything if you cannot reproduce - but happens every time for me. Try executing 'Test Connection' from jenkins config page for ec2 - fails immediately for both profile and real keys.
          ximon18 Ximon Eighteen added a comment - - edited

          Hi martinfr62,

          I have verified that I have not left any files uncommitted, and that the pull request latest commit ID and commit message match what I have locally. I am now using that pull request on several Jenkins instances in AWS and they are all now working. The pull request introduced a "Use old signing mechanism for authenticating request" tick box in the Amazon EC2 Cloud configuration, above the secret key and access key, did you tick that box? Without ticking that box I get authentication errors.

          It was implied by the original AWS staff member who responded to the forum request about the v4 vs query text signer algorithm issue that the default (v4) signer algorithm should work in most cases and thus I have not made the box ticked by default. However, thinking about it, the previous behaviour of the EC2 plugin was to use the old signer algorithm, so perhaps the old behaviour should be the default?

          There is currently no description or help for the tick box, I should add that.

          'Test Connection' and 'Check AMI' both work for me when the box is ticked.

          My Jenkins instances are in the EU (Ireland) AWS region.

          If you could give me the error message and stack trace that would be very appreciated, and please verify that you have tested both with and without ticking the tick box.

          ximon18 Ximon Eighteen added a comment - - edited Hi martinfr62, I have verified that I have not left any files uncommitted, and that the pull request latest commit ID and commit message match what I have locally. I am now using that pull request on several Jenkins instances in AWS and they are all now working. The pull request introduced a "Use old signing mechanism for authenticating request" tick box in the Amazon EC2 Cloud configuration, above the secret key and access key, did you tick that box? Without ticking that box I get authentication errors. It was implied by the original AWS staff member who responded to the forum request about the v4 vs query text signer algorithm issue that the default (v4) signer algorithm should work in most cases and thus I have not made the box ticked by default. However, thinking about it, the previous behaviour of the EC2 plugin was to use the old signer algorithm, so perhaps the old behaviour should be the default? There is currently no description or help for the tick box, I should add that. 'Test Connection' and 'Check AMI' both work for me when the box is ticked. My Jenkins instances are in the EU (Ireland) AWS region. If you could give me the error message and stack trace that would be very appreciated, and please verify that you have tested both with and without ticking the tick box.
          martinfr62 martinfr62 added a comment -

          Here's the stack trace I get

          using us-east-1 with instance profile enabled - hit test connection and get this in the log.

          Process I followed - clone master repo, sync to head, pull down 131 patch and apply using tortoisegit (using windows).Build plugin and deploy to working jenkins.

          Hit test connection and it fails as below.

          Failed to check EC2 credential
          com.amazonaws.AmazonServiceException: AWS was not able to validate the provided access credentials (Service: AmazonEC2; Status Code: 401; Error Code: AuthFailure; Request ID: 43877fc7-b5e0-43ac-8dda-b9f802645343)
          at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032)
          at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687)
          at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441)
          at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292)
          at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225)
          at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541)
          at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:8665)
          at hudson.plugins.ec2.EC2Cloud$DescriptorImpl.doTestConnection(EC2Cloud.java:598)
          at hudson.plugins.ec2.AmazonEC2Cloud$DescriptorImpl.doTestConnection(AmazonEC2Cloud.java:159)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:606)
          at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
          at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
          at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
          at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
          at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
          at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
          at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
          at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:123)
          at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:46)
          at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:103)
          at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:42)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:120)
          at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:114)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
          at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
          at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
          at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
          at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
          at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
          at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
          at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
          at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
          at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
          at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
          at org.eclipse.jetty.server.Server.handle(Server.java:370)
          at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
          at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960)
          at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021)
          at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
          at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
          at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
          at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
          at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
          at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          martinfr62 martinfr62 added a comment - Here's the stack trace I get using us-east-1 with instance profile enabled - hit test connection and get this in the log. Process I followed - clone master repo, sync to head, pull down 131 patch and apply using tortoisegit (using windows).Build plugin and deploy to working jenkins. Hit test connection and it fails as below. Failed to check EC2 credential com.amazonaws.AmazonServiceException: AWS was not able to validate the provided access credentials (Service: AmazonEC2; Status Code: 401; Error Code: AuthFailure; Request ID: 43877fc7-b5e0-43ac-8dda-b9f802645343) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:8665) at hudson.plugins.ec2.EC2Cloud$DescriptorImpl.doTestConnection(EC2Cloud.java:598) at hudson.plugins.ec2.AmazonEC2Cloud$DescriptorImpl.doTestConnection(AmazonEC2Cloud.java:159) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:123) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:46) at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:103) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:42) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:120) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:114) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)

          martinfr62 contacted me privately, we'll work out what the problem is.

          Separately to that, it occurred to me that end users won't want to know or care about request signing mechanism, or even having to specify whether or not instance profile credentials should be used. Instead both could be determined by the plugin by testing the connection, checking if it works with instance profile credentials, checking if it works with old or new signing algorithm, and just use what works.

          ximon18 Ximon Eighteen added a comment - martinfr62 contacted me privately, we'll work out what the problem is. Separately to that, it occurred to me that end users won't want to know or care about request signing mechanism, or even having to specify whether or not instance profile credentials should be used. Instead both could be determined by the plugin by testing the connection, checking if it works with instance profile credentials, checking if it works with old or new signing algorithm, and just use what works.
          martinfr62 martinfr62 added a comment -

          So having worked with Ximon offline, the signing step was the issue. I didnt notice it so didnt have it set.

          Would prefer that test connection generate an error and indicate to try changing the signing to see if that works - or test both if flag isnt set and indicate that it requires setting.

          So user 'has control'.

          martinfr62 martinfr62 added a comment - So having worked with Ximon offline, the signing step was the issue. I didnt notice it so didnt have it set. Would prefer that test connection generate an error and indicate to try changing the signing to see if that works - or test both if flag isnt set and indicate that it requires setting. So user 'has control'.
          ximon18 Ximon Eighteen added a comment - - edited

          So having worked with Ximon offline, the signing step was the issue. I didnt notice it so didnt have it set.

          I should have made it the default as that was effectively the behaviour prior to this new version of the plugin.

          However, I think that automated self configuration as suggested by martinfr62 is a better idea. Sadly I haven't had any time for this issue for the last few weeks as the urgency on my side decreased the moment we managed to find a work around, i.e. a custom build of the plugin containing the fix documented in this issue. However, this isn't a sustainable long term solution, this PR is already one version behind the official version as 1.27 was just released and the PR is based on 1.26, so I hope to get back to this issue and increase the quality of my/our pull request.

          It bothers me however that there has been no obvious interest from the original plugin authors/maintainers in this issue... if I do improve the pull request I hope that it will be accepted and merged into a new release of the plugin. The plugin does seem to be actively maintained as there was a new 1.27 release a few days ago so there is hope I suppose.

          ximon18 Ximon Eighteen added a comment - - edited So having worked with Ximon offline, the signing step was the issue. I didnt notice it so didnt have it set. I should have made it the default as that was effectively the behaviour prior to this new version of the plugin. However, I think that automated self configuration as suggested by martinfr62 is a better idea. Sadly I haven't had any time for this issue for the last few weeks as the urgency on my side decreased the moment we managed to find a work around, i.e. a custom build of the plugin containing the fix documented in this issue. However, this isn't a sustainable long term solution, this PR is already one version behind the official version as 1.27 was just released and the PR is based on 1.26, so I hope to get back to this issue and increase the quality of my/our pull request. It bothers me however that there has been no obvious interest from the original plugin authors/maintainers in this issue... if I do improve the pull request I hope that it will be accepted and merged into a new release of the plugin. The plugin does seem to be actively maintained as there was a new 1.27 release a few days ago so there is hope I suppose.
          ximon18 Ximon Eighteen added a comment - - edited

          Interesting. It seems that JENKINS-26414 has updated to the latest Java AWS SDK and added the signer override. Is this issue now obsolete?

          ximon18 Ximon Eighteen added a comment - - edited Interesting. It seems that JENKINS-26414 has updated to the latest Java AWS SDK and added the signer override . Is this issue now obsolete?
          djopfer Dave Opfer added a comment -

          Appears that this is still an issue with the latest version.

          djopfer Dave Opfer added a comment - Appears that this is still an issue with the latest version.

          Unfortunately https://github.com/jenkinsci/ec2-plugin/pull/131 has now been closed without merging because the closer thought that ec2-1.27 fixed the issue.

          ximon18 Ximon Eighteen added a comment - Unfortunately https://github.com/jenkinsci/ec2-plugin/pull/131 has now been closed without merging because the closer thought that ec2-1.27 fixed the issue.

          Ximon - not sure if you noticed, but your pull request (#131) has merge conflicts now.

          I'd love to see this issue resolved!

          vrivellino Vincent Rivellino added a comment - Ximon - not sure if you noticed, but your pull request ( #131 ) has merge conflicts now. I'd love to see this issue resolved!
          jjudd James Judd added a comment - - edited

          I resolved the conflicts on (#131) and opened a new pull request (#147). Ran into issues using this fix. I think removing the cached connection in EC2Cloud might cause requests to be made too frequently to the metadata service. Our log file is full of these errors and jenkins stops working. The line

           INFO: Excess workload after pending Spot instances: 23 

          is strange, because we are not using spot instances.

          Going to try applying only the first commit from (#131) and seeing if that works, sans the fix that martin made.

          .... com.amazonaws.auth.InstanceProfileCredentialsProvider handleError
          SEVERE: Unable to load credentials from Amazon EC2 metadata service
          com.amazonaws.AmazonClientException: Unable to load credentials from Amazon EC2 metadata service
                  at com.amazonaws.auth.InstanceProfileCredentialsProvider.handleError(InstanceProfileCredentialsProvider.java:244)
                  at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:225)
                  at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:124)
                  at com.amazonaws.auth.InstanceProfileCredentialsProvider$1.run(InstanceProfileCredentialsProvider.java:104)
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
                  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                  at java.lang.Thread.run(Thread.java:745)
          Caused by: java.io.IOException: Server returned HTTP response code: 429 for URL: http://169.254.169.254/latest/meta-data/iam/security-credentials/
                  at sun.reflect.GeneratedConstructorAccessor460.newInstance(Unknown Source)
                  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                  at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
                  at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1676)
                  at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1674)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1672)
                  at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1245)
                  at com.amazonaws.internal.EC2MetadataClient.readResponse(EC2MetadataClient.java:113)
                  at com.amazonaws.internal.EC2MetadataClient.readResource(EC2MetadataClient.java:92)
                  at com.amazonaws.internal.EC2MetadataClient.getDefaultCredentials(EC2MetadataClient.java:55)
                  at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:186)
                  ... 9 more
          Caused by: java.io.IOException: Server returned HTTP response code: 429 for URL: http://169.254.169.254/latest/meta-data/iam/security-credentials/
                  at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1627)
                  at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
                  at com.amazonaws.internal.EC2MetadataClient.readResponse(EC2MetadataClient.java:110)
                  ... 12 more
          
          May 21, 2015 3:15:37 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Excess workload after pending Spot instances: 23
          May 21, 2015 3:15:37 AM hudson.triggers.SafeTimerTask run
          SEVERE: Timer task hudson.slaves.NodeProvisioner$NodeProvisionerInvoker@39ed6b39 failed
          java.lang.OutOfMemoryError: unable to create new native thread
                  at java.lang.Thread.start0(Native Method)
                  at java.lang.Thread.start(Thread.java:714)
                  at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
                  at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
                  at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
                  at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
                  at com.amazonaws.auth.InstanceProfileCredentialsProvider.<init>(InstanceProfileCredentialsProvider.java:100)
                  at hudson.plugins.ec2.EC2Cloud.createCredentialsProvider(EC2Cloud.java:456)
                  at hudson.plugins.ec2.EC2Cloud.createCredentialsProvider(EC2Cloud.java:440)
                  at hudson.plugins.ec2.EC2Cloud.connect(EC2Cloud.java:468)
                  at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:233)
                  at hudson.plugins.ec2.EC2Cloud.addProvisionedSlave(EC2Cloud.java:304)
                  at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:394)
                  at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:683)
                  at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:284)
                  at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:57)
                  at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:778)
                  at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
                  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                  at java.lang.Thread.run(Thread.java:745)
          
          jjudd James Judd added a comment - - edited I resolved the conflicts on ( #131 ) and opened a new pull request ( #147 ). Ran into issues using this fix. I think removing the cached connection in EC2Cloud might cause requests to be made too frequently to the metadata service. Our log file is full of these errors and jenkins stops working. The line INFO: Excess workload after pending Spot instances: 23 is strange, because we are not using spot instances. Going to try applying only the first commit from ( #131 ) and seeing if that works, sans the fix that martin made. .... com.amazonaws.auth.InstanceProfileCredentialsProvider handleError SEVERE: Unable to load credentials from Amazon EC2 metadata service com.amazonaws.AmazonClientException: Unable to load credentials from Amazon EC2 metadata service at com.amazonaws.auth.InstanceProfileCredentialsProvider.handleError(InstanceProfileCredentialsProvider.java:244) at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:225) at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:124) at com.amazonaws.auth.InstanceProfileCredentialsProvider$1.run(InstanceProfileCredentialsProvider.java:104) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Server returned HTTP response code: 429 for URL: http://169.254.169.254/latest/meta-data/iam/security-credentials/ at sun.reflect.GeneratedConstructorAccessor460.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1676) at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1674) at java.security.AccessController.doPrivileged(Native Method) at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1672) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1245) at com.amazonaws.internal.EC2MetadataClient.readResponse(EC2MetadataClient.java:113) at com.amazonaws.internal.EC2MetadataClient.readResource(EC2MetadataClient.java:92) at com.amazonaws.internal.EC2MetadataClient.getDefaultCredentials(EC2MetadataClient.java:55) at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:186) ... 9 more Caused by: java.io.IOException: Server returned HTTP response code: 429 for URL: http://169.254.169.254/latest/meta-data/iam/security-credentials/ at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1627) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) at com.amazonaws.internal.EC2MetadataClient.readResponse(EC2MetadataClient.java:110) ... 12 more May 21, 2015 3:15:37 AM hudson.plugins.ec2.EC2Cloud provision INFO: Excess workload after pending Spot instances: 23 May 21, 2015 3:15:37 AM hudson.triggers.SafeTimerTask run SEVERE: Timer task hudson.slaves.NodeProvisioner$NodeProvisionerInvoker@39ed6b39 failed java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590) at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333) at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594) at com.amazonaws.auth.InstanceProfileCredentialsProvider.<init>(InstanceProfileCredentialsProvider.java:100) at hudson.plugins.ec2.EC2Cloud.createCredentialsProvider(EC2Cloud.java:456) at hudson.plugins.ec2.EC2Cloud.createCredentialsProvider(EC2Cloud.java:440) at hudson.plugins.ec2.EC2Cloud.connect(EC2Cloud.java:468) at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:233) at hudson.plugins.ec2.EC2Cloud.addProvisionedSlave(EC2Cloud.java:304) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:394) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:683) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:284) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:57) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:778) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
          martinfr62 martinfr62 added a comment -

          James - just an fyi the message about pending spot instances is actually n othing to do with spot instances. Weird I know - it just means that you have a pending queue size of 23, after all spot instances have been launched - as you dont have any spot instances - you just have a queue size of 23. That's all the message means. Nothing to do really with spot instances.

          martinfr62 martinfr62 added a comment - James - just an fyi the message about pending spot instances is actually n othing to do with spot instances. Weird I know - it just means that you have a pending queue size of 23, after all spot instances have been launched - as you dont have any spot instances - you just have a queue size of 23. That's all the message means. Nothing to do really with spot instances.
          jjudd James Judd added a comment - - edited

          Thanks for the info Martin.

          I spent some more time looking into this tonight and I think I found the cause. Even better, I think the fix is quite simple. At the moment, in EC2Cloud: we create an AmazonEC2Client like so

          AmazonEC2 client = new AmazonEC2Client(credentialsProvider.getCredentials(), config);
          

          According to the Amazon SDK source this creates a StaticCredentialsProvider using the given credentials. From what I can tell, StaticCredentialsProvider never refreshes its credentials, leading to expiration.

          Instead, you can create an AmazonEC2Client with a credentials provider directly. This should, as far as I can tell, refresh the credentials as needed.

          AmazonEC2 client = new AmazonEC2Client(credentialsProvider, config);
          

          This is further supported by this amazon documentation, which states

          Important

          The automatic credentials refresh happens only when you use the default client constructor, which creates its own InstanceProfileCredentialsProvider as part of the default provider chain, or when you pass an InstanceProfileCredentialsProvider instance directly to the client constructor. If you use another method to obtain or pass instance profile credentials, you are responsible for checking for and refreshing expired credentials.

          [emphasis mine]

          I just uploaded a version of the plugin with this change to our Jenkins server. I'll let it run and report back tomorrow if I see any errors. If it works, I'll create a pull request.

          jjudd James Judd added a comment - - edited Thanks for the info Martin. I spent some more time looking into this tonight and I think I found the cause. Even better, I think the fix is quite simple. At the moment, in EC2Cloud: we create an AmazonEC2Client like so AmazonEC2 client = new AmazonEC2Client(credentialsProvider.getCredentials(), config); According to the Amazon SDK source this creates a StaticCredentialsProvider using the given credentials. From what I can tell, StaticCredentialsProvider never refreshes its credentials, leading to expiration. Instead, you can create an AmazonEC2Client with a credentials provider directly . This should, as far as I can tell, refresh the credentials as needed. AmazonEC2 client = new AmazonEC2Client(credentialsProvider, config); This is further supported by this amazon documentation , which states Important The automatic credentials refresh happens only when you use the default client constructor, which creates its own InstanceProfileCredentialsProvider as part of the default provider chain, or when you pass an InstanceProfileCredentialsProvider instance directly to the client constructor . If you use another method to obtain or pass instance profile credentials, you are responsible for checking for and refreshing expired credentials. [emphasis mine] I just uploaded a version of the plugin with this change to our Jenkins server. I'll let it run and report back tomorrow if I see any errors. If it works, I'll create a pull request.
          jjudd James Judd added a comment -

          It's been almost 24 hours and I have not had any expirations. Created PR #148

          jjudd James Judd added a comment - It's been almost 24 hours and I have not had any expirations. Created PR #148

          Code changed in jenkins
          User: James Judd
          Path:
          src/main/java/hudson/plugins/ec2/EC2Cloud.java
          http://jenkins-ci.org/commit/ec2-plugin/2855d1d925dcfe92043ec6ce0b58111c116eb330
          Log:
          Creates an AmazonEC2Client with an AWSCredentialsProvider instead of the
          AWSCredentials directly. This is done so the credentials will refresh
          instead of expire. Resolves JENKINS-26854.

          At the moment, in EC2Cloud we create an AmazonEC2Client by passing in
          the credentials directly. This creates a StaticCredentialsProvider using
          the given credentials. StaticCredentialsProvider never refreshes its
          credentials, leading to expiration. Instead, you can create an
          AmazonEC2Client with a credentials provider directly. This refreshes the
          credentials as needed.

          See http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-roles.html
          for more information.AWSCredentials directly. This is done so the
          credentials will refresh instead of expire.

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: James Judd Path: src/main/java/hudson/plugins/ec2/EC2Cloud.java http://jenkins-ci.org/commit/ec2-plugin/2855d1d925dcfe92043ec6ce0b58111c116eb330 Log: Creates an AmazonEC2Client with an AWSCredentialsProvider instead of the AWSCredentials directly. This is done so the credentials will refresh instead of expire. Resolves JENKINS-26854 . At the moment, in EC2Cloud we create an AmazonEC2Client by passing in the credentials directly. This creates a StaticCredentialsProvider using the given credentials. StaticCredentialsProvider never refreshes its credentials, leading to expiration. Instead, you can create an AmazonEC2Client with a credentials provider directly. This refreshes the credentials as needed. See http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-roles.html for more information.AWSCredentials directly. This is done so the credentials will refresh instead of expire.

          Code changed in jenkins
          User: Francis Upton
          Path:
          src/main/java/hudson/plugins/ec2/EC2Cloud.java
          http://jenkins-ci.org/commit/ec2-plugin/1fa2ee1126e007d874cb40d5dee25a031746a635
          Log:
          Merge pull request #148 from jjudd/request-expired

          JENKINS-26854: Fixing 'RequestExpired'

          Compare: https://github.com/jenkinsci/ec2-plugin/compare/b576bb3163db...1fa2ee1126e0

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Francis Upton Path: src/main/java/hudson/plugins/ec2/EC2Cloud.java http://jenkins-ci.org/commit/ec2-plugin/1fa2ee1126e007d874cb40d5dee25a031746a635 Log: Merge pull request #148 from jjudd/request-expired JENKINS-26854 : Fixing 'RequestExpired' Compare: https://github.com/jenkinsci/ec2-plugin/compare/b576bb3163db...1fa2ee1126e0
          ximon18 Ximon Eighteen added a comment - - edited

          I see that there is a 1.28 release tag which includes fixes for this issue. However I don't see a 1.28 release at the Jenkins updates site or on the ec2 plugin home page. Also this issue is still marked as 'Open' - has the 1.28 build been tested by anyone following this issue? When will this 1.28 version be released?

          Additional: Can I test using this HPI? https://buildhive.cloudbees.com/job/jenkinsci/job/ec2-plugin/115/org.jenkins-ci.plugins$ec2/artifact/org.jenkins-ci.plugins/ec2/1.28-SNAPSHOT/ec2-1.28-SNAPSHOT.hpi

          ximon18 Ximon Eighteen added a comment - - edited I see that there is a 1.28 release tag which includes fixes for this issue. However I don't see a 1.28 release at the Jenkins updates site or on the ec2 plugin home page . Also this issue is still marked as 'Open' - has the 1.28 build been tested by anyone following this issue? When will this 1.28 version be released? Additional: Can I test using this HPI? https://buildhive.cloudbees.com/job/jenkinsci/job/ec2-plugin/115/org.jenkins-ci.plugins$ec2/artifact/org.jenkins-ci.plugins/ec2/1.28-SNAPSHOT/ec2-1.28-SNAPSHOT.hpi
          francisu Francis Upton added a comment -

          @Ximon, I had intended to do the release and I think something went wrong. I will try to fix this today.

          francisu Francis Upton added a comment - @Ximon, I had intended to do the release and I think something went wrong. I will try to fix this today.
          jjudd James Judd added a comment -

          francisu Just curious when 1.28 will be released.

          jjudd James Judd added a comment - francisu Just curious when 1.28 will be released.
          francisu Francis Upton added a comment -

          I think the release actually worked this time, so by tomorrow it should be present on the wiki page and available.

          francisu Francis Upton added a comment - I think the release actually worked this time, so by tomorrow it should be present on the wiki page and available.

          I updated to 1.28 yesterday and have disabled my auto-restart job.

          After 19 hours of uptime, no expired credentials. I think things are copacetic.

          vrivellino Vincent Rivellino added a comment - I updated to 1.28 yesterday and have disabled my auto-restart job. After 19 hours of uptime, no expired credentials. I think things are copacetic.
          francisu Francis Upton added a comment -

          Fixed in 1.28.

          francisu Francis Upton added a comment - Fixed in 1.28.

          People

            francisu Francis Upton
            ximon18 Ximon Eighteen
            Votes:
            5 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: