-
Bug
-
Resolution: Cannot Reproduce
-
Minor
-
None
-
Jenkins 2.60.3
ec2-plugin 1.63
Similar to the previous issues about lost instances (for both spot and demand), the problem occurs for us roughly once per week. As I investigated a bit, I want to share the details here - for EC2 Spot Instances in our case.
Sep 14, 2017 7:26:14 AM WARNING hudson.plugins.ec2.EC2AbstractSlave getInstance Failed to fetch EC2 instance: ESC (sir-xxxxxxx) com.amazonaws.services.ec2.model.AmazonEC2Exception: Invalid id: "<AMI Description> (sir-xxxxxx)" (Service: AmazonEC2; Status Code: 400; Error Code: InvalidInstanceID.Malformed; Request ID: e0563b6f-ce0d-44ae-ba0f-c654934487be) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:13551) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:13527) at com.amazonaws.services.ec2.AmazonEC2Client.executeDescribeInstances(AmazonEC2Client.java:6530) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:6506) at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:278) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:429) at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:397) at hudson.plugins.ec2.EC2SpotSlave.isAlive(EC2SpotSlave.java:52) at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:43) at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:101) at java.lang.Thread.run(Thread.java:748)
Investigating CloudTrail logs confirms that it makes DescribeInstances requests to an instanceID "<AMI Description> (sir-xxxxxx)" (which AWS doesn't know, of course):
"eventSource": "ec2.amazonaws.com", "eventName": "DescribeInstances", "awsRegion": "eu-west-1", "sourceIPAddress": "....", "userAgent": "aws-sdk-java/1.11.119 Linux/4.9.27-14.31.amzn1.x86_64 OpenJDK_64-Bit_Server_VM/25.141-b15/1.8.0_141", "errorCode": "Client.InvalidInstanceID.Malformed", "errorMessage": "Invalid id: \"<AMI Description> (sir-xxxxxx)\"", "requestParameters": { "instancesSet": { "items": [ { "instanceId": "<AMI Description> (sir-xxxxx)" } ] },
Actually, it should look like this
"requestParameters": { "instancesSet": { "items": [ { "instanceId": "i-067209....." } ] },
- is related to
-
JENKINS-26798 EC2 instance not stopped/terminated when slave marked offline
-
- Closed
-
-
JENKINS-43774 Stop expensive runaway EC2 slaves by using the --valid-until option for spot instances
-
- Open
-
Having JENKINS-43774 at least kill the instances after defined runtime.