-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins Version: 2.387.3.5
Jenkins Vender: Cloudbees
EC2 Plugin Version: 2.0.6
Deployment Details:
We have Jenkins controllers running in EKS on one AWS account. EC2 Agents are being deployed in their own account.
We are using an IAM role to handle permissions with an STS token duration of 900 seconds (this was reduced for testing purposes)Jenkins Version: 2.387.3.5 Jenkins Vender: Cloudbees EC2 Plugin Version: 2.0.6 Deployment Details: We have Jenkins controllers running in EKS on one AWS account. EC2 Agents are being deployed in their own account. We are using an IAM role to handle permissions with an STS token duration of 900 seconds (this was reduced for testing purposes)
Hi there, we have been noticing this issue where we get the following exception sporadically.
hudson.plugins.ec2.EC2Cloud#provision: SlaveTemplate{description='cbci-amznl2-agent', labels='aws-aml2 aws-linux'}. Exception during provisioning
com.amazonaws.services.ec2.model.AmazonEC2Exception: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 3de0d618-7b0c-4ef1-82f4-8eedf5bd8c88; Proxy: null)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:34698)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:34665)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:34654)
at com.amazonaws.services.ec2.AmazonEC2Client.executeDescribeImages(AmazonEC2Client.java:15117)
at com.amazonaws.services.ec2.AmazonEC2Client.describeImages(AmazonEC2Client.java:15085)
at hudson.plugins.ec2.SlaveTemplate.getImage(SlaveTemplate.java:1347)
at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:901)
at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:717)
at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:743)
at hudson.slaves.Cloud.provision(Cloud.java:210)
at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:726)
at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:325)
at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:94)
at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:69)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
In order to unblock ourselves, we have been saving the cloud configuration with no changes, this seems to reestablish the connection to AWS.
We have noticed that in some cases, there have been logs from the EC2ConnectionUpdater between instances of the above stack trace.
I noticed that the EC2ConnectionUpdater works by catching an AmazonClientException and then triggers the reconnectToEc2 method on EC2Cloud.
The only real difference I can see between the calls made to EC2 from this plugin is that the EC2ConnectionUpdater calls describeInstances whereas the SlaveTemplate calls describeImages.
If you need any more information from me please feel free to reach out.
Thanks,
Alan.
