-
Bug
-
Resolution: Fixed
-
Critical
-
Jenkins 2.89.4
Ec2 Plugin 1.38
This issue seems similar to JENKINS-36516. Jenkins is able to launch EC2 build agents and after some time will stop launching new agents. We see the below exception in the system log.
Poking at the cloud settings in "Manage Jenkins" seems to do something that clears this error. I suspect it is refreshing cached or stale credentials/tokens or something similar.
Jenkins version 2.89.4
Ec2 Plugin 1.38
Jenkins is running in a docker container behind an nginx reverse proxy.
Is there a way to troubleshoot or debug this issue further? I would be very interested in assisting with investigation/debug/fix as I have an environment where this happens frequently.
{{Exception during provisioning com.amazonaws.services.ec2.model.AmazonEC2Exception: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 0916d4bd-d804-4ade-af52-0786f9a4aaa0) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:15654) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:15630) at com.amazonaws.services.ec2.AmazonEC2Client.executeDescribeInstances(AmazonEC2Client.java:7621) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:7597) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:7633) at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:363) at hudson.plugins.ec2.EC2Cloud.getPossibleNewSlavesCount(EC2Cloud.java:502) at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:522) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:551) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) }}
I was able to work around/fix this by setting up an appropriate instance profile to allow the Jenkins master to launch build agents using the instance profile instead of using credentials. If anyone else experiences similar behavior, try the instance profile method to see if it resolves the problem.