[JENKINS-26854] EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"

Ximon Eighteen added a comment - 2015-02-09 20:34

I've been looking at the underlying AWS SDK code and it looks like it already has built in support for refreshing the credentials before they expire, and that this support is already being used by the Jenkins EC2 plugin. The only things I can think of are either that NTP isn't working properly and clock drift causes the problem, or that the synchronous mode of credential refresh being used by the Jenkins EC2 plugin doesn't work for some reason and perhaps the asynchronous background thread mode needs to be used? I will investigate NTP on my side.

Ximon Eighteen added a comment - 2015-02-09 20:34 I've been looking at the underlying AWS SDK code and it looks like it already has built in support for refreshing the credentials before they expire, and that this support is already being used by the Jenkins EC2 plugin. The only things I can think of are either that NTP isn't working properly and clock drift causes the problem, or that the synchronous mode of credential refresh being used by the Jenkins EC2 plugin doesn't work for some reason and perhaps the asynchronous background thread mode needs to be used? I will investigate NTP on my side.

Ximon Eighteen added a comment - 2015-02-09 22:03 - edited

My investigation into NTP hasn't found any problems:

ubuntu@ip-172-30-0-149:~$ ntpq -pn
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
-87.232.1.40     62.231.32.35     4 u 1037 1024  377    4.120   -1.394   2.634
+78.143.174.10   193.1.219.116    2 u  382 1024  377   71.766   -7.342  48.358
-86.43.77.42     193.120.10.3     2 u  624 1024  337   40.723    8.783   0.787
+85.91.1.180     195.66.241.2     2 u   94 1024  377    1.812   -3.254   0.835
*91.189.89.199   192.93.2.20      2 u  741 1024  377   10.945   -1.322   0.847

The * shows the NTP server being used, and the reach 377 value shows that the NTP daemon was repeatedly able to contact the remote time server, the stratum number is low which is good, and the jitter and offset values are low which is good. Executing 'date' in both the Ubuntu host and the Docker container yields the same date and time.

Unrelated to the NTP investigation, but related to my point about restarting Jenkins in the initial post, if I invoke http://<jenkins>/safeRestart then the slave is started correctly after Jenkins restarts, without my fixing any clocks.

Ximon Eighteen added a comment - 2015-02-09 22:03 - edited My investigation into NTP hasn't found any problems: ubuntu@ip-172-30-0-149:~$ ntpq -pn remote refid st t when poll reach delay offset jitter ============================================================================== -87.232.1.40 62.231.32.35 4 u 1037 1024 377 4.120 -1.394 2.634 +78.143.174.10 193.1.219.116 2 u 382 1024 377 71.766 -7.342 48.358 -86.43.77.42 193.120.10.3 2 u 624 1024 337 40.723 8.783 0.787 +85.91.1.180 195.66.241.2 2 u 94 1024 377 1.812 -3.254 0.835 *91.189.89.199 192.93.2.20 2 u 741 1024 377 10.945 -1.322 0.847 The * shows the NTP server being used, and the reach 377 value shows that the NTP daemon was repeatedly able to contact the remote time server, the stratum number is low which is good, and the jitter and offset values are low which is good. Executing 'date' in both the Ubuntu host and the Docker container yields the same date and time. Unrelated to the NTP investigation, but related to my point about restarting Jenkins in the initial post, if I invoke http://<jenkins>/safeRestart then the slave is started correctly after Jenkins restarts, without my fixing any clocks.

Ximon Eighteen added a comment - 2015-02-09 22:24 - edited

I have no reason to think this is the cause of my problem but I just noticed that the plugin is built using v1.8.3 of the Java SDK while the latest version is 1.9.17, in theory at least there could be a bug fix in the newer versions. I looked through the release notes of the interim versions but didn't find an obvious bug fix that could be related to this issue.

Update: Actually the release 1.8.10 of the Java SDK added the InstanceProfilerCredentialsProvider(true) behaviour I refer to above, but this was not directly mentioned in the release notes. This release was not a happy release, two hot fixes 1.8.10.1 and 1.8.10.2 were released in the following week.

Update: If I build the EC2 plugin with the latest 1.9.17 SDK version it fails with an HTTP 401 Auth error which I haven't tracked down yet. I suspect this is because my AWS IAM InstanceProfile role does not include a permission which newer versions of the SDK require, but I haven't determined which permission is missing yet.

Ximon Eighteen added a comment - 2015-02-09 22:24 - edited I have no reason to think this is the cause of my problem but I just noticed that the plugin is built using v1.8.3 of the Java SDK while the latest version is 1.9.17, in theory at least there could be a bug fix in the newer versions. I looked through the release notes of the interim versions but didn't find an obvious bug fix that could be related to this issue. Update: Actually the release 1.8.10 of the Java SDK added the InstanceProfilerCredentialsProvider(true) behaviour I refer to above, but this was not directly mentioned in the release notes . This release was not a happy release, two hot fixes 1.8.10.1 and 1.8.10.2 were released in the following week. Update: If I build the EC2 plugin with the latest 1.9.17 SDK version it fails with an HTTP 401 Auth error which I haven't tracked down yet. I suspect this is because my AWS IAM InstanceProfile role does not include a permission which newer versions of the SDK require, but I haven't determined which permission is missing yet.

Ximon Eighteen added a comment - 2015-02-09 23:05 - edited

I'm going to put some logging into a subclass of InstanceProfilerCredentialsProvider() because I suspect that for some reason the credentials are not being refreshed by this class... will let you know what I find out.

Update: Indeed the EC2 credential refresh functionality of the AWS Java SDK is not invoked. See attached jenkins.log (look for lines containing "Ximon:") and gitdiff.txt. Rough highlights from the log (might not be entirely accurate, trying to remember what I did last night):

11:33:00 UTC: Jenkins finished responding to a /safeRestart request that I performed.
11:34:36 UTC: I instructed Jenkins to launch a new EC2 slave. The EC2 plugin fetched the EC2 credentials as part of launching the instance.
11:45:46 UTC: I instructed Jenkins to terminate the EC2 slave. The EC2 plugin did not refetch the credentials.
11:46:09 UTC: I instructed Jenkins to launch a new EC2 slave. The EC2 plugin did not refetch the credentials.
12:22:46 UTC: The EC2 plugin correctly stopped the EC2 slave instance after the idle timeout expired. The EC2 plugin did not refetch the credentials.
04:32:58 UTC: UNRELATED BUG: Jenkins logged "Making <NODE NAME> (i-a2a41545) offline because it’s not responding". Why is this logged over 4 hours after the EC2 plugin stopped the slave?
05:30:22 UTC: I refreshed the Jenkins log web page which seems to have caused the EC2 plugin to attempt to update its knowledge about the state of the slave. By this point the EC2 credentials have expired, but the EC2 plugin did not refetch the credentials. The call to the EC2 API failed with HTTP 400 AmazonServiceException "Request has expired".

Ximon Eighteen added a comment - 2015-02-09 23:05 - edited I'm going to put some logging into a subclass of InstanceProfilerCredentialsProvider() because I suspect that for some reason the credentials are not being refreshed by this class... will let you know what I find out. Update: Indeed the EC2 credential refresh functionality of the AWS Java SDK is not invoked. See attached jenkins.log (look for lines containing "Ximon:") and gitdiff.txt . Rough highlights from the log (might not be entirely accurate, trying to remember what I did last night): 11:33:00 UTC: Jenkins finished responding to a /safeRestart request that I performed. 11:34:36 UTC: I instructed Jenkins to launch a new EC2 slave. The EC2 plugin fetched the EC2 credentials as part of launching the instance. 11:45:46 UTC: I instructed Jenkins to terminate the EC2 slave. The EC2 plugin did not refetch the credentials. 11:46:09 UTC: I instructed Jenkins to launch a new EC2 slave. The EC2 plugin did not refetch the credentials. 12:22:46 UTC: The EC2 plugin correctly stopped the EC2 slave instance after the idle timeout expired. The EC2 plugin did not refetch the credentials. 04:32:58 UTC: UNRELATED BUG: Jenkins logged "Making <NODE NAME> (i-a2a41545) offline because it’s not responding" . Why is this logged over 4 hours after the EC2 plugin stopped the slave? 05:30:22 UTC: I refreshed the Jenkins log web page which seems to have caused the EC2 plugin to attempt to update its knowledge about the state of the slave. By this point the EC2 credentials have expired, but the EC2 plugin did not refetch the credentials. The call to the EC2 API failed with HTTP 400 AmazonServiceException "Request has expired".

Ximon Eighteen added a comment - 2015-02-10 06:50 - edited

I tried building the EC2 plugin with AWS Java SDK 1.8.11, the first stable release after 1.8.10 that introduced the new InstanceProfileCredentialsProvider(true) functionality. This solves the missing refetch of credentials, causing the SDK to check them once a minute. However, it fails with the Auth error I referred to above:

com.amazonaws.AmazonServiceException: AWS was not able to validate the provided access credentials (Service: AmazonEC2; Status Code: 401; Error Code: AuthFailure; Request ID: 0ad2caa1-4f05-41e7-b168-3dc37940265b)
	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032)
	at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687)
	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292)
	at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225)
	at com.amazonaws.services.ec2.AmazonEC2Client.describeKeyPairs(AmazonEC2Client.java:6321)
	at com.amazonaws.services.ec2.AmazonEC2Client.describeKeyPairs(AmazonEC2Client.java:8879)
	at hudson.plugins.ec2.EC2PrivateKey.find(EC2PrivateKey.java:135)
	at hudson.plugins.ec2.SlaveTemplate.getKeyPair(SlaveTemplate.java:719)
	at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:303)
	at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:287)
	at hudson.plugins.ec2.EC2Cloud.doProvision(EC2Cloud.java:283)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:745)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:875)
	at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:745)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:875)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:648)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:237)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:86)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:370)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
	at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

I suspect this is something to do with my IAM configuration, and not with the EC2 plugin or the Java SDK. So I suspect that upgrading the SDK to 1.8.11 and invoking the InstanceProfileCredentialProvider(true) constructor will solve this issue for other people.

Ximon Eighteen added a comment - 2015-02-10 06:50 - edited I tried building the EC2 plugin with AWS Java SDK 1.8.11, the first stable release after 1.8.10 that introduced the new InstanceProfileCredentialsProvider(true) functionality. This solves the missing refetch of credentials, causing the SDK to check them once a minute. However, it fails with the Auth error I referred to above: com.amazonaws.AmazonServiceException: AWS was not able to validate the provided access credentials (Service: AmazonEC2; Status Code: 401; Error Code: AuthFailure; Request ID: 0ad2caa1-4f05-41e7-b168-3dc37940265b) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225) at com.amazonaws.services.ec2.AmazonEC2Client.describeKeyPairs(AmazonEC2Client.java:6321) at com.amazonaws.services.ec2.AmazonEC2Client.describeKeyPairs(AmazonEC2Client.java:8879) at hudson.plugins.ec2.EC2PrivateKey.find(EC2PrivateKey.java:135) at hudson.plugins.ec2.SlaveTemplate.getKeyPair(SlaveTemplate.java:719) at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:303) at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:287) at hudson.plugins.ec2.EC2Cloud.doProvision(EC2Cloud.java:283) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:745) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:875) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:745) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:875) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:648) at org.kohsuke.stapler.Stapler.service(Stapler.java:237) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:86) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) I suspect this is something to do with my IAM configuration, and not with the EC2 plugin or the Java SDK. So I suspect that upgrading the SDK to 1.8.11 and invoking the InstanceProfileCredentialProvider(true) constructor will solve this issue for other people.

Ximon Eighteen added a comment - 2015-02-10 07:09

I have created a minimal pull request. See: https://github.com/jenkinsci/ec2-plugin/pull/131

Ximon Eighteen added a comment - 2015-02-10 07:09 I have created a minimal pull request. See: https://github.com/jenkinsci/ec2-plugin/pull/131

Ximon Eighteen added a comment - 2015-02-10 07:39 - edited

Ah, the AuthFailure may affect others too. See: https://forums.aws.amazon.com/thread.jspa?messageID=574914&tstart=0. I'm seeing this issue in the eu-west-1 region.

Update: Applying the force previous signer configuration solved this issue for me, that is this part of the forum article that I referred to:

clientConfiguration.setSignerOverride(“QueryStringSignerType”);
AmazonEC2 ec2 = new AmazonEC2Client(configuration);

Ximon Eighteen added a comment - 2015-02-10 07:39 - edited Ah, the AuthFailure may affect others too. See: https://forums.aws.amazon.com/thread.jspa?messageID=574914&tstart=0 . I'm seeing this issue in the eu-west-1 region. Update: Applying the force previous signer configuration solved this issue for me, that is this part of the forum article that I referred to: clientConfiguration.setSignerOverride(“QueryStringSignerType”); AmazonEC2 ec2 = new AmazonEC2Client(configuration);

martinfr62 added a comment - 2015-02-10 16:52 - edited

Seeing same issue, restarting jenkins from the web browser cleared the issue, so clearly some sort of issue in the ec2 plugin itself, not an issue with the date on the instance hosting jenkins.

martinfr62 added a comment - 2015-02-10 16:52 - edited Seeing same issue, restarting jenkins from the web browser cleared the issue, so clearly some sort of issue in the ec2 plugin itself, not an issue with the date on the instance hosting jenkins.

Ximon Eighteen added a comment - 2015-02-10 17:27

To be clear, I have a working (well seems to work, I must admit it's not thoroughly tested yet) solution, but I'm not sure if it's wise for others to use. My solution is the pull request here, with the following tiny modification:

└─(.../scratch/ximon-ec2-plugin)─> git diff
diff --git a/src/main/java/hudson/plugins/ec2/EC2Cloud.java b/src/main/java/hudson/plugins/ec2/EC2Cloud.java
index 8891999..7feed81 100644
--- a/src/main/java/hudson/plugins/ec2/EC2Cloud.java
+++ b/src/main/java/hudson/plugins/ec2/EC2Cloud.java
@@ -476,6 +476,7 @@ public abstract class EC2Cloud extends Cloud {
     public synchronized static AmazonEC2 connect(AWSCredentialsProvider credentialsProvider, URL endpoint) {
         awsCredentialsProvider = credentialsProvider;
         ClientConfiguration config = new ClientConfiguration();
+        config.setSignerOverride("QueryStringSignerType");
         ProxyConfiguration proxyConfig = Jenkins.getInstance().proxy;
         Proxy proxy = proxyConfig == null ? Proxy.NO_PROXY : proxyConfig.createProxy(endpoint.getHost());
         if (! proxy.equals(Proxy.NO_PROXY) && proxy.address() instanceof InetSocketAddress) {

Ximon Eighteen added a comment - 2015-02-10 17:27 To be clear, I have a working (well seems to work, I must admit it's not thoroughly tested yet) solution, but I'm not sure if it's wise for others to use. My solution is the pull request here , with the following tiny modification: └─(.../scratch/ximon-ec2-plugin)─> git diff diff --git a/src/main/java/hudson/plugins/ec2/EC2Cloud.java b/src/main/java/hudson/plugins/ec2/EC2Cloud.java index 8891999..7feed81 100644 --- a/src/main/java/hudson/plugins/ec2/EC2Cloud.java +++ b/src/main/java/hudson/plugins/ec2/EC2Cloud.java @@ -476,6 +476,7 @@ public abstract class EC2Cloud extends Cloud { public synchronized static AmazonEC2 connect(AWSCredentialsProvider credentialsProvider, URL endpoint) { awsCredentialsProvider = credentialsProvider; ClientConfiguration config = new ClientConfiguration(); + config.setSignerOverride( "QueryStringSignerType" ); ProxyConfiguration proxyConfig = Jenkins.getInstance().proxy; Proxy proxy = proxyConfig == null ? Proxy.NO_PROXY : proxyConfig.createProxy(endpoint.getHost()); if (! proxy.equals(Proxy.NO_PROXY) && proxy.address() instanceof InetSocketAddress) {

martinfr62 added a comment - 2015-02-10 23:23

Pulled copy of 131 and built locally - just realized I had to apply above change as well. Will rebuild, deploy and test myself.

martinfr62 added a comment - 2015-02-10 23:23 Pulled copy of 131 and built locally - just realized I had to apply above change as well. Will rebuild, deploy and test myself.

martinfr62 added a comment - 2015-02-12 16:46

So still seeing the issue - this time though when the instance is terminated after it times out - instance then is left running in AWS, but dropped from jenkins, and cannot start any more instances until the aws instances have been terminated.

Failed to fetch EC2 instance: i-2107a6db
com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 1b435e23-f37f-4433-9526-f7226801cab9)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225)
at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541)
at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:208)
at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:354)
at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:326)
at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:42)
at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:53)
at java.lang.Thread.run(Thread.java:745)

Feb 12, 2015 6:31:26 AM INFO hudson.plugins.ec2.EC2SlaveMonitor execute
EC2 instance is dead: i-2107a6db
Feb 12, 2015 6:31:28 AM WARNING hudson.plugins.ec2.EC2AbstractSlave getInstance
Failed to fetch EC2 instance: i-2107a6db
com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 8e53415f-6d34-4669-a76e-536d9cedcf37)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225)
at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541)
at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:208)
at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:354)
at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:326)
at hudson.plugins.ec2.EC2OndemandSlave.terminate(EC2OndemandSlave.java:63)
at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:44)
at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:53)
at java.lang.Thread.run(Thread.java:745)

Feb 12, 2015 6:31:28 AM INFO hudson.plugins.ec2.EC2OndemandSlave terminate
EC2 instance already terminated: i-2107a6db
Feb 12, 2015 6:31:28 AM INFO hudson.plugins.ec2.EC2OndemandSlave terminate
Removed EC2 instance from jenkins master: i-2107a6db
Feb 12, 2015 6:31:30 AM WARNING hudson.plugins.ec2.EC2AbstractSlave getInstance
Failed to fetch EC2 instance: i-1a04a5e0
com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 4a7feb85-c0ff-4dda-9f88-f605513408a8)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225)
at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541)
at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:208)
at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:354)
at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:326)
at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:42)
at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:53)
at java.lang.Thread.run(Thread.java:745)

Feb 12, 2015 6:31:30 AM INFO hudson.plugins.ec2.EC2SlaveMonitor execute
EC2 instance is dead: i-1a04a5e0
Feb 12, 2015 6:31:32 AM WARNING hudson.plugins.ec2.EC2AbstractSlave getInstance
Failed to fetch EC2 instance: i-1a04a5e0
com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: be105033-7ff8-4ef4-9dbc-860e8cd6a73a)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225)
at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541)
at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:208)
at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:354)
at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:326)
at hudson.plugins.ec2.EC2OndemandSlave.terminate(EC2OndemandSlave.java:63)
at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:44)
at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:53)
at java.lang.Thread.run(Thread.java:745)

Feb 12, 2015 6:31:32 AM INFO hudson.plugins.ec2.EC2OndemandSlave terminate

martinfr62 added a comment - 2015-02-12 16:46 So still seeing the issue - this time though when the instance is terminated after it times out - instance then is left running in AWS, but dropped from jenkins, and cannot start any more instances until the aws instances have been terminated. Failed to fetch EC2 instance: i-2107a6db com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 1b435e23-f37f-4433-9526-f7226801cab9) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541) at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:208) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:354) at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:326) at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:42) at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:53) at java.lang.Thread.run(Thread.java:745) Feb 12, 2015 6:31:26 AM INFO hudson.plugins.ec2.EC2SlaveMonitor execute EC2 instance is dead: i-2107a6db Feb 12, 2015 6:31:28 AM WARNING hudson.plugins.ec2.EC2AbstractSlave getInstance Failed to fetch EC2 instance: i-2107a6db com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 8e53415f-6d34-4669-a76e-536d9cedcf37) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541) at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:208) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:354) at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:326) at hudson.plugins.ec2.EC2OndemandSlave.terminate(EC2OndemandSlave.java:63) at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:44) at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:53) at java.lang.Thread.run(Thread.java:745) Feb 12, 2015 6:31:28 AM INFO hudson.plugins.ec2.EC2OndemandSlave terminate EC2 instance already terminated: i-2107a6db Feb 12, 2015 6:31:28 AM INFO hudson.plugins.ec2.EC2OndemandSlave terminate Removed EC2 instance from jenkins master: i-2107a6db Feb 12, 2015 6:31:30 AM WARNING hudson.plugins.ec2.EC2AbstractSlave getInstance Failed to fetch EC2 instance: i-1a04a5e0 com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 4a7feb85-c0ff-4dda-9f88-f605513408a8) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541) at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:208) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:354) at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:326) at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:42) at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:53) at java.lang.Thread.run(Thread.java:745) Feb 12, 2015 6:31:30 AM INFO hudson.plugins.ec2.EC2SlaveMonitor execute EC2 instance is dead: i-1a04a5e0 Feb 12, 2015 6:31:32 AM WARNING hudson.plugins.ec2.EC2AbstractSlave getInstance Failed to fetch EC2 instance: i-1a04a5e0 com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: be105033-7ff8-4ef4-9dbc-860e8cd6a73a) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541) at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:208) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:354) at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:326) at hudson.plugins.ec2.EC2OndemandSlave.terminate(EC2OndemandSlave.java:63) at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:44) at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:53) at java.lang.Thread.run(Thread.java:745) Feb 12, 2015 6:31:32 AM INFO hudson.plugins.ec2.EC2OndemandSlave terminate

martinfr62 added a comment - 2015-02-12 17:09

Looks like EC2Cloud.connect() caches the AmazonEC2 connection - testing patch which obtains the connection every time. Will taker a while, have to let the terminate event fire for the instances.

martinfr62 added a comment - 2015-02-12 17:09 Looks like EC2Cloud.connect() caches the AmazonEC2 connection - testing patch which obtains the connection every time. Will taker a while, have to let the terminate event fire for the instances.

martinfr62 added a comment - 2015-02-12 22:53

Okay - here's what i did in patch form against the pull 131 (so includes your Ximon).

Removed the transient connection object and created a new one every time in connect() and then updated/added some more logging so i could tell when the terminate was about to fire. I also found a typo in the log by doing this in EC2RetentionStrategy - was treating freeSecondsLeft as milliseconds - not seconds - so fixed that also.

—
src/main/java/hudson/plugins/ec2/EC2Cloud.java | 9 +++++----
src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java | 6 +++++-
2 files changed, 10 insertions, 5 deletions

diff --git a/src/main/java/hudson/plugins/ec2/EC2Cloud.java b/src/main/java/hudson/plugins/ec2/EC2Cloud.java
index 8891999..ca102e6 100644
— a/src/main/java/hudson/plugins/ec2/EC2Cloud.java
+++ b/src/main/java/hudson/plugins/ec2/EC2Cloud.java
@@ -113,7 +113,7 @@ public abstract class EC2Cloud extends Cloud {
private final List<? extends SlaveTemplate> templates;
private transient KeyPair usableKeyPair;

protected transient AmazonEC2 connection;
+ // protected transient AmazonEC2 connection;

private static AWSCredentialsProvider awsCredentialsProvider;

@@ -460,9 +460,9 @@ public abstract class EC2Cloud extends Cloud {
*/
public synchronized AmazonEC2 connect() throws AmazonClientException {
try {

if (connection == null) { - connection = connect(createCredentialsProvider(), getEc2EndpointUrl()); - }
+ // if (connection == null)
{ + AmazonEC2 connection = connect(createCredentialsProvider(), getEc2EndpointUrl()); + // }
return connection;
} catch (IOException e) {
throw new AmazonClientException("Failed to retrieve the endpoint",e);
@@ -476,6 +476,7 @@ public abstract class EC2Cloud extends Cloud {
public synchronized static AmazonEC2 connect(AWSCredentialsProvider credentialsProvider, URL endpoint) {
awsCredentialsProvider = credentialsProvider;
ClientConfiguration config = new ClientConfiguration();
+ config.setSignerOverride("QueryStringSignerType");
ProxyConfiguration proxyConfig = Jenkins.getInstance().proxy;
Proxy proxy = proxyConfig == null ? Proxy.NO_PROXY : proxyConfig.createProxy(endpoint.getHost());
if (! proxy.equals(Proxy.NO_PROXY) && proxy.address() instanceof InetSocketAddress) {
diff --git a/src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java b/src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java
index 22e185a..52d8f49 100644
- - a/src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java
    +++ b/src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java
    @@ -94,6 +94,8 @@ public class EC2RetentionStrategy extends RetentionStrategy<EC2Computer> {
    final long idleMilliseconds = System.currentTimeMillis() - c.getIdleStartMilliseconds();
    if (idleTerminationMinutes > 0) {
    // TODO: really think about the right strategy here, see ~~JENKINS-23792~~
    + LOGGER.fine("Test Idle timeout of "+c.getName() + " after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes");
    +
    if (idleMilliseconds > TimeUnit2.MINUTES.toMillis(idleTerminationMinutes)) {
    LOGGER.info("Idle timeout of "+c.getName() + " after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes");
    c.getNode().idleTimeout();
    @@ -110,8 +112,10 @@ public class EC2RetentionStrategy extends RetentionStrategy<EC2Computer>
    Unknown macro: { final int freeSecondsLeft = (60*60) - (int)(TimeUnit2.SECONDS.convert(uptime, TimeUnit2.MILLISECONDS) % (60*60)); // if we have less "free" (aka already paid for) time left than our idle time, stop/terminate the instance // See JENKINS-23821+ LOGGER.fine("Test Idle timeout of "+c.getName()+" after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes, with " + TimeUnit2.SECONDS.toMinutes(freeSecondsLeft) + " minutes remaining in billing period");+ if (freeSecondsLeft <= (Math.abs(idleTerminationMinutes*60))) { - LOGGER.info("Idle timeout of "+c.getName()+" after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes, with " + TimeUnit2.MILLISECONDS.toMinutes(freeSecondsLeft) + " minutes remaining in billing period"); + LOGGER.info("Idle timeout of "+c.getName()+" after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes, with " + TimeUnit2.SECONDS.toMinutes(freeSecondsLeft) + " minutes remaining in billing period"); c.getNode().idleTimeout(); } }
- 1.9.4.msysgit.2

martinfr62 added a comment - 2015-02-12 22:53 Okay - here's what i did in patch form against the pull 131 (so includes your Ximon). Removed the transient connection object and created a new one every time in connect() and then updated/added some more logging so i could tell when the terminate was about to fire. I also found a typo in the log by doing this in EC2RetentionStrategy - was treating freeSecondsLeft as milliseconds - not seconds - so fixed that also. — src/main/java/hudson/plugins/ec2/EC2Cloud.java | 9 +++++---- src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java | 6 +++++- 2 files changed, 10 insertions , 5 deletions diff --git a/src/main/java/hudson/plugins/ec2/EC2Cloud.java b/src/main/java/hudson/plugins/ec2/EC2Cloud.java index 8891999..ca102e6 100644 — a/src/main/java/hudson/plugins/ec2/EC2Cloud.java +++ b/src/main/java/hudson/plugins/ec2/EC2Cloud.java @@ -113,7 +113,7 @@ public abstract class EC2Cloud extends Cloud { private final List<? extends SlaveTemplate> templates; private transient KeyPair usableKeyPair; protected transient AmazonEC2 connection; + // protected transient AmazonEC2 connection; private static AWSCredentialsProvider awsCredentialsProvider; @@ -460,9 +460,9 @@ public abstract class EC2Cloud extends Cloud { */ public synchronized AmazonEC2 connect() throws AmazonClientException { try { if (connection == null) { - connection = connect(createCredentialsProvider(), getEc2EndpointUrl()); - } + // if (connection == null) { + AmazonEC2 connection = connect(createCredentialsProvider(), getEc2EndpointUrl()); + // } return connection; } catch (IOException e) { throw new AmazonClientException("Failed to retrieve the endpoint",e); @@ -476,6 +476,7 @@ public abstract class EC2Cloud extends Cloud { public synchronized static AmazonEC2 connect(AWSCredentialsProvider credentialsProvider, URL endpoint) { awsCredentialsProvider = credentialsProvider; ClientConfiguration config = new ClientConfiguration(); + config.setSignerOverride("QueryStringSignerType"); ProxyConfiguration proxyConfig = Jenkins.getInstance().proxy; Proxy proxy = proxyConfig == null ? Proxy.NO_PROXY : proxyConfig.createProxy(endpoint.getHost()); if (! proxy.equals(Proxy.NO_PROXY) && proxy.address() instanceof InetSocketAddress) { diff --git a/src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java b/src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java index 22e185a..52d8f49 100644 a/src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java +++ b/src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java @@ -94,6 +94,8 @@ public class EC2RetentionStrategy extends RetentionStrategy<EC2Computer> { final long idleMilliseconds = System.currentTimeMillis() - c.getIdleStartMilliseconds(); if (idleTerminationMinutes > 0) { // TODO: really think about the right strategy here, see JENKINS-23792 + LOGGER.fine("Test Idle timeout of "+c.getName() + " after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes"); + if (idleMilliseconds > TimeUnit2.MINUTES.toMillis(idleTerminationMinutes)) { LOGGER.info("Idle timeout of "+c.getName() + " after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes"); c.getNode().idleTimeout(); @@ -110,8 +112,10 @@ public class EC2RetentionStrategy extends RetentionStrategy<EC2Computer> Unknown macro: { final int freeSecondsLeft = (60*60) - (int)(TimeUnit2.SECONDS.convert(uptime, TimeUnit2.MILLISECONDS) % (60*60)); // if we have less "free" (aka already paid for) time left than our idle time, stop/terminate the instance // See JENKINS-23821+ LOGGER.fine("Test Idle timeout of "+c.getName()+" after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes, with " + TimeUnit2.SECONDS.toMinutes(freeSecondsLeft) + " minutes remaining in billing period");+ if (freeSecondsLeft <= (Math.abs(idleTerminationMinutes*60))) { - LOGGER.info("Idle timeout of "+c.getName()+" after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes, with " + TimeUnit2.MILLISECONDS.toMinutes(freeSecondsLeft) + " minutes remaining in billing period"); + LOGGER.info("Idle timeout of "+c.getName()+" after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes, with " + TimeUnit2.SECONDS.toMinutes(freeSecondsLeft) + " minutes remaining in billing period"); c.getNode().idleTimeout(); } } 1.9.4.msysgit.2

Vincent Rivellino added a comment - 2015-02-20 12:12

FYI: I'm running into the same issue on Jenkins ver. 1.580.3 with Amazon EC2 Plugin v1.26 on Amazon Linux with an IAM Instance Profile assigned to the instance.

Would be great to get this fixed so we don't have to restart our Jenkins master every 6 hours.

Vincent Rivellino added a comment - 2015-02-20 12:12 FYI: I'm running into the same issue on Jenkins ver. 1.580.3 with Amazon EC2 Plugin v1.26 on Amazon Linux with an IAM Instance Profile assigned to the instance. Would be great to get this fixed so we don't have to restart our Jenkins master every 6 hours.

chris fair added a comment - 2015-02-23 13:55

I built Ximon's branch above, along with Martin's changes, and haven't seen this issue occur again. I did have to add 'config.setSignerOverride("QueryStringSignerType");' and I'm in region us-east-1.

chris fair added a comment - 2015-02-23 13:55 I built Ximon's branch above, along with Martin's changes, and haven't seen this issue occur again. I did have to add 'config.setSignerOverride("QueryStringSignerType");' and I'm in region us-east-1.

Ximon Eighteen added a comment - 2015-02-27 20:28

I've updated my pull request with a new boolean EC2 Cloud parameter to enable using the query string signer, and the fix from Martin (though I just realized I only mentioned the query string signer fix in the commit message, wasn't intentional not to mention that...). Just thought it would be handy to put the changes in one place for anyone else reading this issue until the fix makes it into a proper release build, and to make sure that if the query string signer type is needed that users can enable it.

Ximon Eighteen added a comment - 2015-02-27 20:28 I've updated my pull request with a new boolean EC2 Cloud parameter to enable using the query string signer, and the fix from Martin (though I just realized I only mentioned the query string signer fix in the commit message, wasn't intentional not to mention that...). Just thought it would be handy to put the changes in one place for anyone else reading this issue until the fix makes it into a proper release build, and to make sure that if the query string signer type is needed that users can enable it.

martinfr62 added a comment - 2015-03-02 23:03 - edited

Tried pulling 131 and patching current master - it fails to connect to aws.

Pulled current master - built and depoyed - connects okay (but has normal bug in it)
Pulled 131 and split the patch in two - first half (ie prior 131) works - can connect to aws
Apply second patch as a straight merge - and it builds - but cannot connect

Next tried pull master, apply first half - and then apply my patch alone - this works.

Looks like something in the second patch causes aws to fail.

Let me know if you need a stack trace or anything if you cannot reproduce - but happens every time for me.

Try executing 'Test Connection' from jenkins config page for ec2 - fails immediately for both profile and real keys.

martinfr62 added a comment - 2015-03-02 23:03 - edited Tried pulling 131 and patching current master - it fails to connect to aws. Pulled current master - built and depoyed - connects okay (but has normal bug in it) Pulled 131 and split the patch in two - first half (ie prior 131) works - can connect to aws Apply second patch as a straight merge - and it builds - but cannot connect Next tried pull master, apply first half - and then apply my patch alone - this works. Looks like something in the second patch causes aws to fail. Let me know if you need a stack trace or anything if you cannot reproduce - but happens every time for me. Try executing 'Test Connection' from jenkins config page for ec2 - fails immediately for both profile and real keys.

Ximon Eighteen added a comment - 2015-03-03 06:31 - edited

Hi martinfr62,

I have verified that I have not left any files uncommitted, and that the pull request latest commit ID and commit message match what I have locally. I am now using that pull request on several Jenkins instances in AWS and they are all now working. The pull request introduced a "Use old signing mechanism for authenticating request" tick box in the Amazon EC2 Cloud configuration, above the secret key and access key, did you tick that box? Without ticking that box I get authentication errors.

It was implied by the original AWS staff member who responded to the forum request about the v4 vs query text signer algorithm issue that the default (v4) signer algorithm should work in most cases and thus I have not made the box ticked by default. However, thinking about it, the previous behaviour of the EC2 plugin was to use the old signer algorithm, so perhaps the old behaviour should be the default?

There is currently no description or help for the tick box, I should add that.

'Test Connection' and 'Check AMI' both work for me when the box is ticked.

My Jenkins instances are in the EU (Ireland) AWS region.

If you could give me the error message and stack trace that would be very appreciated, and please verify that you have tested both with and without ticking the tick box.

Ximon Eighteen added a comment - 2015-03-03 06:31 - edited Hi martinfr62, I have verified that I have not left any files uncommitted, and that the pull request latest commit ID and commit message match what I have locally. I am now using that pull request on several Jenkins instances in AWS and they are all now working. The pull request introduced a "Use old signing mechanism for authenticating request" tick box in the Amazon EC2 Cloud configuration, above the secret key and access key, did you tick that box? Without ticking that box I get authentication errors. It was implied by the original AWS staff member who responded to the forum request about the v4 vs query text signer algorithm issue that the default (v4) signer algorithm should work in most cases and thus I have not made the box ticked by default. However, thinking about it, the previous behaviour of the EC2 plugin was to use the old signer algorithm, so perhaps the old behaviour should be the default? There is currently no description or help for the tick box, I should add that. 'Test Connection' and 'Check AMI' both work for me when the box is ticked. My Jenkins instances are in the EU (Ireland) AWS region. If you could give me the error message and stack trace that would be very appreciated, and please verify that you have tested both with and without ticking the tick box.

martinfr62 added a comment - 2015-03-03 17:22

Here's the stack trace I get

using us-east-1 with instance profile enabled - hit test connection and get this in the log.

Process I followed - clone master repo, sync to head, pull down 131 patch and apply using tortoisegit (using windows).Build plugin and deploy to working jenkins.

Hit test connection and it fails as below.

Failed to check EC2 credential
com.amazonaws.AmazonServiceException: AWS was not able to validate the provided access credentials (Service: AmazonEC2; Status Code: 401; Error Code: AuthFailure; Request ID: 43877fc7-b5e0-43ac-8dda-b9f802645343)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225)
at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541)
at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:8665)
at hudson.plugins.ec2.EC2Cloud$DescriptorImpl.doTestConnection(EC2Cloud.java:598)
at hudson.plugins.ec2.AmazonEC2Cloud$DescriptorImpl.doTestConnection(AmazonEC2Cloud.java:159)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:123)
at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:46)
at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:103)
at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:42)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:120)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:114)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

martinfr62 added a comment - 2015-03-03 17:22 Here's the stack trace I get using us-east-1 with instance profile enabled - hit test connection and get this in the log. Process I followed - clone master repo, sync to head, pull down 131 patch and apply using tortoisegit (using windows).Build plugin and deploy to working jenkins. Hit test connection and it fails as below. Failed to check EC2 credential com.amazonaws.AmazonServiceException: AWS was not able to validate the provided access credentials (Service: AmazonEC2; Status Code: 401; Error Code: AuthFailure; Request ID: 43877fc7-b5e0-43ac-8dda-b9f802645343) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:8665) at hudson.plugins.ec2.EC2Cloud$DescriptorImpl.doTestConnection(EC2Cloud.java:598) at hudson.plugins.ec2.AmazonEC2Cloud$DescriptorImpl.doTestConnection(AmazonEC2Cloud.java:159) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:123) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:46) at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:103) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:42) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:120) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:114) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)

Ximon Eighteen added a comment - 2015-03-03 17:43

martinfr62 contacted me privately, we'll work out what the problem is.

Separately to that, it occurred to me that end users won't want to know or care about request signing mechanism, or even having to specify whether or not instance profile credentials should be used. Instead both could be determined by the plugin by testing the connection, checking if it works with instance profile credentials, checking if it works with old or new signing algorithm, and just use what works.

Ximon Eighteen added a comment - 2015-03-03 17:43 martinfr62 contacted me privately, we'll work out what the problem is. Separately to that, it occurred to me that end users won't want to know or care about request signing mechanism, or even having to specify whether or not instance profile credentials should be used. Instead both could be determined by the plugin by testing the connection, checking if it works with instance profile credentials, checking if it works with old or new signing algorithm, and just use what works.

martinfr62 added a comment - 2015-03-03 20:01

So having worked with Ximon offline, the signing step was the issue. I didnt notice it so didnt have it set.

Would prefer that test connection generate an error and indicate to try changing the signing to see if that works - or test both if flag isnt set and indicate that it requires setting.

So user 'has control'.

martinfr62 added a comment - 2015-03-03 20:01 So having worked with Ximon offline, the signing step was the issue. I didnt notice it so didnt have it set. Would prefer that test connection generate an error and indicate to try changing the signing to see if that works - or test both if flag isnt set and indicate that it requires setting. So user 'has control'.

Ximon Eighteen added a comment - 2015-04-03 07:24 - edited

So having worked with Ximon offline, the signing step was the issue. I didnt notice it so didnt have it set.

I should have made it the default as that was effectively the behaviour prior to this new version of the plugin.

However, I think that automated self configuration as suggested by martinfr62 is a better idea. Sadly I haven't had any time for this issue for the last few weeks as the urgency on my side decreased the moment we managed to find a work around, i.e. a custom build of the plugin containing the fix documented in this issue. However, this isn't a sustainable long term solution, this PR is already one version behind the official version as 1.27 was just released and the PR is based on 1.26, so I hope to get back to this issue and increase the quality of my/our pull request.

It bothers me however that there has been no obvious interest from the original plugin authors/maintainers in this issue... if I do improve the pull request I hope that it will be accepted and merged into a new release of the plugin. The plugin does seem to be actively maintained as there was a new 1.27 release a few days ago so there is hope I suppose.

Ximon Eighteen added a comment - 2015-04-03 07:24 - edited So having worked with Ximon offline, the signing step was the issue. I didnt notice it so didnt have it set. I should have made it the default as that was effectively the behaviour prior to this new version of the plugin. However, I think that automated self configuration as suggested by martinfr62 is a better idea. Sadly I haven't had any time for this issue for the last few weeks as the urgency on my side decreased the moment we managed to find a work around, i.e. a custom build of the plugin containing the fix documented in this issue. However, this isn't a sustainable long term solution, this PR is already one version behind the official version as 1.27 was just released and the PR is based on 1.26, so I hope to get back to this issue and increase the quality of my/our pull request. It bothers me however that there has been no obvious interest from the original plugin authors/maintainers in this issue... if I do improve the pull request I hope that it will be accepted and merged into a new release of the plugin. The plugin does seem to be actively maintained as there was a new 1.27 release a few days ago so there is hope I suppose.

Ximon Eighteen added a comment - 2015-04-03 07:41 - edited

Interesting. It seems that ~~JENKINS-26414~~ has updated to the latest Java AWS SDK and added the signer override. Is this issue now obsolete?

Ximon Eighteen added a comment - 2015-04-03 07:41 - edited Interesting. It seems that JENKINS-26414 has updated to the latest Java AWS SDK and added the signer override . Is this issue now obsolete?

Dave Opfer added a comment - 2015-04-10 13:22

Appears that this is still an issue with the latest version.

Dave Opfer added a comment - 2015-04-10 13:22 Appears that this is still an issue with the latest version.

Ximon Eighteen added a comment - 2015-04-14 07:56

Unfortunately https://github.com/jenkinsci/ec2-plugin/pull/131 has now been closed without merging because the closer thought that ec2-1.27 fixed the issue.

Ximon Eighteen added a comment - 2015-04-14 07:56 Unfortunately https://github.com/jenkinsci/ec2-plugin/pull/131 has now been closed without merging because the closer thought that ec2-1.27 fixed the issue.

Vincent Rivellino added a comment - 2015-04-18 13:11

Ximon - not sure if you noticed, but your pull request (#131) has merge conflicts now.

I'd love to see this issue resolved!

Vincent Rivellino added a comment - 2015-04-18 13:11 Ximon - not sure if you noticed, but your pull request ( #131 ) has merge conflicts now. I'd love to see this issue resolved!

James Judd added a comment - 2015-05-21 03:26 - edited

I resolved the conflicts on (#131) and opened a new pull request (#147). Ran into issues using this fix. I think removing the cached connection in EC2Cloud might cause requests to be made too frequently to the metadata service. Our log file is full of these errors and jenkins stops working. The line

 INFO: Excess workload after pending Spot instances: 23

is strange, because we are not using spot instances.

Going to try applying only the first commit from (#131) and seeing if that works, sans the fix that martin made.

.... com.amazonaws.auth.InstanceProfileCredentialsProvider handleError
SEVERE: Unable to load credentials from Amazon EC2 metadata service
com.amazonaws.AmazonClientException: Unable to load credentials from Amazon EC2 metadata service
        at com.amazonaws.auth.InstanceProfileCredentialsProvider.handleError(InstanceProfileCredentialsProvider.java:244)
        at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:225)
        at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:124)
        at com.amazonaws.auth.InstanceProfileCredentialsProvider$1.run(InstanceProfileCredentialsProvider.java:104)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Server returned HTTP response code: 429 for URL: http://169.254.169.254/latest/meta-data/iam/security-credentials/
        at sun.reflect.GeneratedConstructorAccessor460.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1676)
        at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1674)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1672)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1245)
        at com.amazonaws.internal.EC2MetadataClient.readResponse(EC2MetadataClient.java:113)
        at com.amazonaws.internal.EC2MetadataClient.readResource(EC2MetadataClient.java:92)
        at com.amazonaws.internal.EC2MetadataClient.getDefaultCredentials(EC2MetadataClient.java:55)
        at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:186)
        ... 9 more
Caused by: java.io.IOException: Server returned HTTP response code: 429 for URL: http://169.254.169.254/latest/meta-data/iam/security-credentials/
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1627)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
        at com.amazonaws.internal.EC2MetadataClient.readResponse(EC2MetadataClient.java:110)
        ... 12 more

May 21, 2015 3:15:37 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Excess workload after pending Spot instances: 23
May 21, 2015 3:15:37 AM hudson.triggers.SafeTimerTask run
SEVERE: Timer task hudson.slaves.NodeProvisioner$NodeProvisionerInvoker@39ed6b39 failed
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:714)
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
        at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
        at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
        at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
        at com.amazonaws.auth.InstanceProfileCredentialsProvider.<init>(InstanceProfileCredentialsProvider.java:100)
        at hudson.plugins.ec2.EC2Cloud.createCredentialsProvider(EC2Cloud.java:456)
        at hudson.plugins.ec2.EC2Cloud.createCredentialsProvider(EC2Cloud.java:440)
        at hudson.plugins.ec2.EC2Cloud.connect(EC2Cloud.java:468)
        at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:233)
        at hudson.plugins.ec2.EC2Cloud.addProvisionedSlave(EC2Cloud.java:304)
        at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:394)
        at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:683)
        at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:284)
        at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:57)
        at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:778)
        at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

James Judd added a comment - 2015-05-21 03:26 - edited I resolved the conflicts on ( #131 ) and opened a new pull request ( #147 ). Ran into issues using this fix. I think removing the cached connection in EC2Cloud might cause requests to be made too frequently to the metadata service. Our log file is full of these errors and jenkins stops working. The line INFO: Excess workload after pending Spot instances: 23 is strange, because we are not using spot instances. Going to try applying only the first commit from ( #131 ) and seeing if that works, sans the fix that martin made. .... com.amazonaws.auth.InstanceProfileCredentialsProvider handleError SEVERE: Unable to load credentials from Amazon EC2 metadata service com.amazonaws.AmazonClientException: Unable to load credentials from Amazon EC2 metadata service at com.amazonaws.auth.InstanceProfileCredentialsProvider.handleError(InstanceProfileCredentialsProvider.java:244) at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:225) at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:124) at com.amazonaws.auth.InstanceProfileCredentialsProvider$1.run(InstanceProfileCredentialsProvider.java:104) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Server returned HTTP response code: 429 for URL: http://169.254.169.254/latest/meta-data/iam/security-credentials/ at sun.reflect.GeneratedConstructorAccessor460.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1676) at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1674) at java.security.AccessController.doPrivileged(Native Method) at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1672) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1245) at com.amazonaws.internal.EC2MetadataClient.readResponse(EC2MetadataClient.java:113) at com.amazonaws.internal.EC2MetadataClient.readResource(EC2MetadataClient.java:92) at com.amazonaws.internal.EC2MetadataClient.getDefaultCredentials(EC2MetadataClient.java:55) at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:186) ... 9 more Caused by: java.io.IOException: Server returned HTTP response code: 429 for URL: http://169.254.169.254/latest/meta-data/iam/security-credentials/ at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1627) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) at com.amazonaws.internal.EC2MetadataClient.readResponse(EC2MetadataClient.java:110) ... 12 more May 21, 2015 3:15:37 AM hudson.plugins.ec2.EC2Cloud provision INFO: Excess workload after pending Spot instances: 23 May 21, 2015 3:15:37 AM hudson.triggers.SafeTimerTask run SEVERE: Timer task hudson.slaves.NodeProvisioner$NodeProvisionerInvoker@39ed6b39 failed java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590) at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333) at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594) at com.amazonaws.auth.InstanceProfileCredentialsProvider.<init>(InstanceProfileCredentialsProvider.java:100) at hudson.plugins.ec2.EC2Cloud.createCredentialsProvider(EC2Cloud.java:456) at hudson.plugins.ec2.EC2Cloud.createCredentialsProvider(EC2Cloud.java:440) at hudson.plugins.ec2.EC2Cloud.connect(EC2Cloud.java:468) at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:233) at hudson.plugins.ec2.EC2Cloud.addProvisionedSlave(EC2Cloud.java:304) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:394) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:683) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:284) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:57) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:778) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)

martinfr62 added a comment - 2015-05-21 06:45

James - just an fyi the message about pending spot instances is actually n othing to do with spot instances. Weird I know - it just means that you have a pending queue size of 23, after all spot instances have been launched - as you dont have any spot instances - you just have a queue size of 23. That's all the message means. Nothing to do really with spot instances.

martinfr62 added a comment - 2015-05-21 06:45 James - just an fyi the message about pending spot instances is actually n othing to do with spot instances. Weird I know - it just means that you have a pending queue size of 23, after all spot instances have been launched - as you dont have any spot instances - you just have a queue size of 23. That's all the message means. Nothing to do really with spot instances.

James Judd added a comment - 2015-05-22 02:46 - edited

Thanks for the info Martin.

I spent some more time looking into this tonight and I think I found the cause. Even better, I think the fix is quite simple. At the moment, in EC2Cloud: we create an AmazonEC2Client like so

AmazonEC2 client = new AmazonEC2Client(credentialsProvider.getCredentials(), config);

According to the Amazon SDK source this creates a StaticCredentialsProvider using the given credentials. From what I can tell, StaticCredentialsProvider never refreshes its credentials, leading to expiration.

Instead, you can create an AmazonEC2Client with a credentials provider directly. This should, as far as I can tell, refresh the credentials as needed.

AmazonEC2 client = new AmazonEC2Client(credentialsProvider, config);

This is further supported by this amazon documentation, which states

Important

The automatic credentials refresh happens only when you use the default client constructor, which creates its own InstanceProfileCredentialsProvider as part of the default provider chain, or when you pass an InstanceProfileCredentialsProvider instance directly to the client constructor. If you use another method to obtain or pass instance profile credentials, you are responsible for checking for and refreshing expired credentials.

[emphasis mine]

I just uploaded a version of the plugin with this change to our Jenkins server. I'll let it run and report back tomorrow if I see any errors. If it works, I'll create a pull request.

James Judd added a comment - 2015-05-22 02:46 - edited Thanks for the info Martin. I spent some more time looking into this tonight and I think I found the cause. Even better, I think the fix is quite simple. At the moment, in EC2Cloud: we create an AmazonEC2Client like so AmazonEC2 client = new AmazonEC2Client(credentialsProvider.getCredentials(), config); According to the Amazon SDK source this creates a StaticCredentialsProvider using the given credentials. From what I can tell, StaticCredentialsProvider never refreshes its credentials, leading to expiration. Instead, you can create an AmazonEC2Client with a credentials provider directly . This should, as far as I can tell, refresh the credentials as needed. AmazonEC2 client = new AmazonEC2Client(credentialsProvider, config); This is further supported by this amazon documentation , which states Important The automatic credentials refresh happens only when you use the default client constructor, which creates its own InstanceProfileCredentialsProvider as part of the default provider chain, or when you pass an InstanceProfileCredentialsProvider instance directly to the client constructor . If you use another method to obtain or pass instance profile credentials, you are responsible for checking for and refreshing expired credentials. [emphasis mine] I just uploaded a version of the plugin with this change to our Jenkins server. I'll let it run and report back tomorrow if I see any errors. If it works, I'll create a pull request.

James Judd added a comment - 2015-05-22 21:59

It's been almost 24 hours and I have not had any expirations. Created PR #148

James Judd added a comment - 2015-05-22 21:59 It's been almost 24 hours and I have not had any expirations. Created PR #148

SCM/JIRA link daemon added a comment - 2015-05-24 17:33

Code changed in jenkins
User: James Judd
Path:
src/main/java/hudson/plugins/ec2/EC2Cloud.java
http://jenkins-ci.org/commit/ec2-plugin/2855d1d925dcfe92043ec6ce0b58111c116eb330
Log:
Creates an AmazonEC2Client with an AWSCredentialsProvider instead of the
AWSCredentials directly. This is done so the credentials will refresh
instead of expire. Resolves ~~JENKINS-26854~~.

At the moment, in EC2Cloud we create an AmazonEC2Client by passing in
the credentials directly. This creates a StaticCredentialsProvider using
the given credentials. StaticCredentialsProvider never refreshes its
credentials, leading to expiration. Instead, you can create an
AmazonEC2Client with a credentials provider directly. This refreshes the
credentials as needed.

See http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-roles.html
for more information.AWSCredentials directly. This is done so the
credentials will refresh instead of expire.

SCM/JIRA link daemon added a comment - 2015-05-24 17:33 Code changed in jenkins User: James Judd Path: src/main/java/hudson/plugins/ec2/EC2Cloud.java http://jenkins-ci.org/commit/ec2-plugin/2855d1d925dcfe92043ec6ce0b58111c116eb330 Log: Creates an AmazonEC2Client with an AWSCredentialsProvider instead of the AWSCredentials directly. This is done so the credentials will refresh instead of expire. Resolves JENKINS-26854 . At the moment, in EC2Cloud we create an AmazonEC2Client by passing in the credentials directly. This creates a StaticCredentialsProvider using the given credentials. StaticCredentialsProvider never refreshes its credentials, leading to expiration. Instead, you can create an AmazonEC2Client with a credentials provider directly. This refreshes the credentials as needed. See http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-roles.html for more information.AWSCredentials directly. This is done so the credentials will refresh instead of expire.

SCM/JIRA link daemon added a comment - 2015-05-24 17:33

Code changed in jenkins
User: Francis Upton
Path:
src/main/java/hudson/plugins/ec2/EC2Cloud.java
http://jenkins-ci.org/commit/ec2-plugin/1fa2ee1126e007d874cb40d5dee25a031746a635
Log:
Merge pull request #148 from jjudd/request-expired

~~JENKINS-26854~~: Fixing 'RequestExpired'

Compare: https://github.com/jenkinsci/ec2-plugin/compare/b576bb3163db...1fa2ee1126e0

SCM/JIRA link daemon added a comment - 2015-05-24 17:33 Code changed in jenkins User: Francis Upton Path: src/main/java/hudson/plugins/ec2/EC2Cloud.java http://jenkins-ci.org/commit/ec2-plugin/1fa2ee1126e007d874cb40d5dee25a031746a635 Log: Merge pull request #148 from jjudd/request-expired JENKINS-26854 : Fixing 'RequestExpired' Compare: https://github.com/jenkinsci/ec2-plugin/compare/b576bb3163db...1fa2ee1126e0

Ximon Eighteen added a comment - 2015-05-26 10:05 - edited

I see that there is a 1.28 release tag which includes fixes for this issue. However I don't see a 1.28 release at the Jenkins updates site or on the ec2 plugin home page. Also this issue is still marked as 'Open' - has the 1.28 build been tested by anyone following this issue? When will this 1.28 version be released?

Additional: Can I test using this HPI? https://buildhive.cloudbees.com/job/jenkinsci/job/ec2-plugin/115/org.jenkins-ci.plugins$ec2/artifact/org.jenkins-ci.plugins/ec2/1.28-SNAPSHOT/ec2-1.28-SNAPSHOT.hpi

Ximon Eighteen added a comment - 2015-05-26 10:05 - edited I see that there is a 1.28 release tag which includes fixes for this issue. However I don't see a 1.28 release at the Jenkins updates site or on the ec2 plugin home page . Also this issue is still marked as 'Open' - has the 1.28 build been tested by anyone following this issue? When will this 1.28 version be released? Additional: Can I test using this HPI? https://buildhive.cloudbees.com/job/jenkinsci/job/ec2-plugin/115/org.jenkins-ci.plugins$ec2/artifact/org.jenkins-ci.plugins/ec2/1.28-SNAPSHOT/ec2-1.28-SNAPSHOT.hpi

Francis Upton added a comment - 2015-05-26 16:46

@Ximon, I had intended to do the release and I think something went wrong. I will try to fix this today.

Francis Upton added a comment - 2015-05-26 16:46 @Ximon, I had intended to do the release and I think something went wrong. I will try to fix this today.

James Judd added a comment - 2015-06-03 14:38

francisu Just curious when 1.28 will be released.

James Judd added a comment - 2015-06-03 14:38 francisu Just curious when 1.28 will be released.

Francis Upton added a comment - 2015-06-07 03:17

I think the release actually worked this time, so by tomorrow it should be present on the wiki page and available.

Francis Upton added a comment - 2015-06-07 03:17 I think the release actually worked this time, so by tomorrow it should be present on the wiki page and available.

Vincent Rivellino added a comment - 2015-06-09 14:22

I updated to 1.28 yesterday and have disabled my auto-restart job.

After 19 hours of uptime, no expired credentials. I think things are copacetic.

Vincent Rivellino added a comment - 2015-06-09 14:22 I updated to 1.28 yesterday and have disabled my auto-restart job. After 19 hours of uptime, no expired credentials. I think things are copacetic.

Francis Upton added a comment - 2015-06-09 16:18

Fixed in 1.28.

Francis Upton added a comment - 2015-06-09 16:18 Fixed in 1.28.

Jenkins

Details

Description

Attachments

Activity

Collapse comment: Ximon Eighteen added a comment - 2015-02-09 20:34

Expand comment: Ximon Eighteen added a comment - 2015-02-09 20:34

Collapse comment: Ximon Eighteen added a comment - 2015-02-09 22:03, Edited by Ximon Eighteen - 2015-02-09 22:35

Expand comment: Ximon Eighteen added a comment - 2015-02-09 22:03, Edited by Ximon Eighteen - 2015-02-09 22:35

Collapse comment: Ximon Eighteen added a comment - 2015-02-09 22:24, Edited by Ximon Eighteen - 2015-02-10 06:39

Expand comment: Ximon Eighteen added a comment - 2015-02-09 22:24, Edited by Ximon Eighteen - 2015-02-10 06:39

Collapse comment: Ximon Eighteen added a comment - 2015-02-09 23:05, Edited by Ximon Eighteen - 2015-02-10 06:32

Expand comment: Ximon Eighteen added a comment - 2015-02-09 23:05, Edited by Ximon Eighteen - 2015-02-10 06:32

Collapse comment: Ximon Eighteen added a comment - 2015-02-10 06:50, Edited by Ximon Eighteen - 2015-02-10 06:52

Expand comment: Ximon Eighteen added a comment - 2015-02-10 06:50, Edited by Ximon Eighteen - 2015-02-10 06:52

Collapse comment: Ximon Eighteen added a comment - 2015-02-10 07:09

Expand comment: Ximon Eighteen added a comment - 2015-02-10 07:09

Collapse comment: Ximon Eighteen added a comment - 2015-02-10 07:39, Edited by Ximon Eighteen - 2015-02-10 07:45

Expand comment: Ximon Eighteen added a comment - 2015-02-10 07:39, Edited by Ximon Eighteen - 2015-02-10 07:45

Collapse comment: martinfr62 added a comment - 2015-02-10 16:52, Edited by martinfr62 - 2015-02-10 16:54

Expand comment: martinfr62 added a comment - 2015-02-10 16:52, Edited by martinfr62 - 2015-02-10 16:54

Collapse comment: Ximon Eighteen added a comment - 2015-02-10 17:27

Expand comment: Ximon Eighteen added a comment - 2015-02-10 17:27

Collapse comment: martinfr62 added a comment - 2015-02-10 23:23

Expand comment: martinfr62 added a comment - 2015-02-10 23:23

Collapse comment: martinfr62 added a comment - 2015-02-12 16:46

Expand comment: martinfr62 added a comment - 2015-02-12 16:46

Collapse comment: martinfr62 added a comment - 2015-02-12 17:09

Expand comment: martinfr62 added a comment - 2015-02-12 17:09

Collapse comment: martinfr62 added a comment - 2015-02-12 22:53

Expand comment: martinfr62 added a comment - 2015-02-12 22:53

Collapse comment: Vincent Rivellino added a comment - 2015-02-20 12:12

Expand comment: Vincent Rivellino added a comment - 2015-02-20 12:12

Collapse comment: chris fair added a comment - 2015-02-23 13:55

Expand comment: chris fair added a comment - 2015-02-23 13:55

Collapse comment: Ximon Eighteen added a comment - 2015-02-27 20:28

Expand comment: Ximon Eighteen added a comment - 2015-02-27 20:28

Collapse comment: martinfr62 added a comment - 2015-03-02 23:03, Edited by martinfr62 - 2015-03-02 23:03

Expand comment: martinfr62 added a comment - 2015-03-02 23:03, Edited by martinfr62 - 2015-03-02 23:03

Collapse comment: Ximon Eighteen added a comment - 2015-03-03 06:31, Edited by Ximon Eighteen - 2015-03-03 06:33

Expand comment: Ximon Eighteen added a comment - 2015-03-03 06:31, Edited by Ximon Eighteen - 2015-03-03 06:33

Collapse comment: martinfr62 added a comment - 2015-03-03 17:22

Expand comment: martinfr62 added a comment - 2015-03-03 17:22

Collapse comment: Ximon Eighteen added a comment - 2015-03-03 17:43

Expand comment: Ximon Eighteen added a comment - 2015-03-03 17:43

Collapse comment: martinfr62 added a comment - 2015-03-03 20:01

Expand comment: martinfr62 added a comment - 2015-03-03 20:01

Collapse comment: Ximon Eighteen added a comment - 2015-04-03 07:24, Edited by Ximon Eighteen - 2015-04-03 07:25

Expand comment: Ximon Eighteen added a comment - 2015-04-03 07:24, Edited by Ximon Eighteen - 2015-04-03 07:25

Collapse comment: Ximon Eighteen added a comment - 2015-04-03 07:41, Edited by Ximon Eighteen - 2015-04-03 07:41

Expand comment: Ximon Eighteen added a comment - 2015-04-03 07:41, Edited by Ximon Eighteen - 2015-04-03 07:41

Collapse comment: Dave Opfer added a comment - 2015-04-10 13:22

Expand comment: Dave Opfer added a comment - 2015-04-10 13:22

Collapse comment: Ximon Eighteen added a comment - 2015-04-14 07:56

Expand comment: Ximon Eighteen added a comment - 2015-04-14 07:56

Collapse comment: Vincent Rivellino added a comment - 2015-04-18 13:11

Expand comment: Vincent Rivellino added a comment - 2015-04-18 13:11

Collapse comment: James Judd added a comment - 2015-05-21 03:26, Edited by James Judd - 2015-05-21 03:27

Expand comment: James Judd added a comment - 2015-05-21 03:26, Edited by James Judd - 2015-05-21 03:27

Collapse comment: martinfr62 added a comment - 2015-05-21 06:45

Expand comment: martinfr62 added a comment - 2015-05-21 06:45

Collapse comment: James Judd added a comment - 2015-05-22 02:46, Edited by James Judd - 2015-05-22 02:48

Expand comment: James Judd added a comment - 2015-05-22 02:46, Edited by James Judd - 2015-05-22 02:48

Collapse comment: James Judd added a comment - 2015-05-22 21:59

Expand comment: James Judd added a comment - 2015-05-22 21:59

Collapse comment: SCM/JIRA link daemon added a comment - 2015-05-24 17:33

Expand comment: SCM/JIRA link daemon added a comment - 2015-05-24 17:33

Collapse comment: SCM/JIRA link daemon added a comment - 2015-05-24 17:33

Expand comment: SCM/JIRA link daemon added a comment - 2015-05-24 17:33

Collapse comment: Ximon Eighteen added a comment - 2015-05-26 10:05, Edited by Ximon Eighteen - 2015-05-26 10:25

Expand comment: Ximon Eighteen added a comment - 2015-05-26 10:05, Edited by Ximon Eighteen - 2015-05-26 10:25

Collapse comment: Francis Upton added a comment - 2015-05-26 16:46

Expand comment: Francis Upton added a comment - 2015-05-26 16:46

Collapse comment: James Judd added a comment - 2015-06-03 14:38

Expand comment: James Judd added a comment - 2015-06-03 14:38

Collapse comment: Francis Upton added a comment - 2015-06-07 03:17

Expand comment: Francis Upton added a comment - 2015-06-07 03:17

Collapse comment: Vincent Rivellino added a comment - 2015-06-09 14:22

Expand comment: Vincent Rivellino added a comment - 2015-06-09 14:22