-
Bug
-
Resolution: Fixed
-
Blocker
-
None
-
Jenkins 2.150.2
ec2-plugin 1.42
When Jenkins attempts to provision spot instances, after spot requests are created (incorrectly, as per JENKINS-55618) Jenkins will try to retrieve the spot instance based on the last working instance id from before we upgraded the plugin, instead of the instance id returned by the spot request.
INFO: Spot instance id in provision: sir-34si4rxjINFO: Spot instance id in provision: sir-34si4rxjJan 17, 2019 4:32:10 AM hudson.plugins.ec2.CloudHelper getInstanceINFO: Unexpected number of reservations reported by EC2 for instance id 'i-0b934d05d88272fc4', expected 1 result, found []. Instance seems to be dead.Jan 17, 2019 4:32:10 AM hudson.plugins.ec2.EC2Cloud provisionWARNING: SlaveTemplate{ami='ami-09009aeca3492b598', labels='agent linux'}. Exception during provisioningcom.amazonaws.AmazonClientException: Unexpected number of reservations reported by EC2 for instance id 'i-0b934d05d88272fc4', expected 1 result, found []. Instance seems to be dead. at hudson.plugins.ec2.CloudHelper.getInstance(CloudHelper.java:54) at hudson.plugins.ec2.CloudHelper.getInstanceWithRetry(CloudHelper.java:25) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:499) at hudson.plugins.ec2.EC2AbstractSlave.<init>(EC2AbstractSlave.java:159) at hudson.plugins.ec2.EC2SpotSlave.<init>(EC2SpotSlave.java:44) at hudson.plugins.ec2.EC2SpotSlave.<init>(EC2SpotSlave.java:37) at hudson.plugins.ec2.SlaveTemplate.newSpotSlave(SlaveTemplate.java:979) at hudson.plugins.ec2.SlaveTemplate.provisionSpot(SlaveTemplate.java:919) at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:464) at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:578) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:594) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
This hasn't happened with on-demand instances.
This may be related to JENKINS-55618; I'm affected by that bug as well, but I wasn't sure how closely related it was so opted to open a new issue.
We also faced this problem on Jenkins 2.157 with ec2-plugin 1.42. Switching our spot instances to regular did not fix the issue, but downgrading the plugin to 1.41 did