-
Bug
-
Resolution: Fixed
-
Minor
-
None
Sometimes when ec2-plugin spins up an instance it will stay disconnected and will never be used.
The error on Jenkins log:
INFO: Started provisioning js-ubuntu (ami-d7bhbda9) from ec2-mng-jenkins-slave with 1 executors. Remaining excess workload: 0 Oct 25, 2018 8:31:18 PM hudson.slaves.NodeProvisioner$2 run WARNING: Unexpected exception encountered while provisioning agent js-ubuntu (ami-d7bhbda9) com.amazonaws.services.ec2.model.AmazonEC2Exception: The instance ID 'i-05a000703b7084e53' does not exist (Service: AmazonEC2; Status Code: 400; Error Code: InvalidInstanceID.NotFound; Request ID: 2d9f3866-19d7-482c-a4cb-057c88880e6c) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1658) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1322) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1072) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515) at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:17597) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:17566) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:17555) at com.amazonaws.services.ec2.AmazonEC2Client.executeDescribeInstances(AmazonEC2Client.java:8680) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:8651) at hudson.plugins.ec2.CloudHelper.getInstance(CloudHelper.java:47) at hudson.plugins.ec2.EC2Cloud$1.call(EC2Cloud.java:603) at hudson.plugins.ec2.EC2Cloud$1.call(EC2Cloud.java:586) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Cloudtrail on AWS shows this errocode:
{ "eventVersion": "1.05", "userIdentity": { "type": "IAMUser", "principalId": "", "arn": "arn:aws:iam:::user/_nsof_int_jenkins", "accountId": "", "accessKeyId": "", "userName": "_nsof_int_jenkins" }, "eventTime": "2018-10-25T17:31:16Z", "eventSource": "ec2.amazonaws.com", "eventName": "DescribeInstances", "awsRegion": "us-east-1", "sourceIPAddress": "52.70.50.139", "userAgent": "aws-sdk-java/1.11.403 Linux/4.4.0-1069-aws OpenJDK_64-Bit_Server_VM/25.181-b13 java/1.8.0_181 groovy/2.4.11", "errorCode": "Client.InvalidInstanceID.NotFound", "errorMessage": "The instance ID 'i-05a000703b7084e53' does not exist", "requestParameters": { "instancesSet": { "items": [ { "instanceId": "i-05a000703b7084e53" } ] }, "filterSet": {} }, "responseElements": null, "requestID": "2d9f3866-19d7-482c-a4cb-057c88880e6c", "eventID": "cb5da0a7-15f7-4360-979e-fff11effa285", "eventType": "AwsApiCall", "recipientAccountId": "" }
After chatting with AWS support they say it's due to the nature of their API's eventual consistency
API calls were made before the instance spawned on AWS.
Is there a way to add a retry in these cases?