Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55639

ec2-plugin only attempts to retrieve spot instance using old id

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • ec2-plugin
    • None
    • Jenkins 2.150.2
      ec2-plugin 1.42

      When Jenkins attempts to provision spot instances, after spot requests are created (incorrectly, as per JENKINS-55618) Jenkins will try to retrieve the spot instance based on the last working instance id from before we upgraded the plugin, instead of the instance id returned by the spot request. 

      INFO: Spot instance id in provision: sir-34si4rxjINFO: Spot instance id in provision: sir-34si4rxjJan 17, 2019 4:32:10 AM hudson.plugins.ec2.CloudHelper getInstanceINFO: Unexpected number of reservations reported by EC2 for instance id 'i-0b934d05d88272fc4', expected 1 result, found []. Instance seems to be dead.Jan 17, 2019 4:32:10 AM hudson.plugins.ec2.EC2Cloud provisionWARNING: SlaveTemplate{ami='ami-09009aeca3492b598', labels='agent linux'}. Exception during provisioningcom.amazonaws.AmazonClientException: Unexpected number of reservations reported by EC2 for instance id 'i-0b934d05d88272fc4', expected 1 result, found []. Instance seems to be dead. at hudson.plugins.ec2.CloudHelper.getInstance(CloudHelper.java:54) at hudson.plugins.ec2.CloudHelper.getInstanceWithRetry(CloudHelper.java:25) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:499) at hudson.plugins.ec2.EC2AbstractSlave.<init>(EC2AbstractSlave.java:159) at hudson.plugins.ec2.EC2SpotSlave.<init>(EC2SpotSlave.java:44) at hudson.plugins.ec2.EC2SpotSlave.<init>(EC2SpotSlave.java:37) at hudson.plugins.ec2.SlaveTemplate.newSpotSlave(SlaveTemplate.java:979) at hudson.plugins.ec2.SlaveTemplate.provisionSpot(SlaveTemplate.java:919) at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:464) at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:578) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:594) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

      This hasn't happened with on-demand instances.

      This may be related to JENKINS-55618; I'm affected by that bug as well, but I wasn't sure how closely related it was so opted to open a new issue. 

          [JENKINS-55639] ec2-plugin only attempts to retrieve spot instance using old id

          We also faced this problem on Jenkins 2.157 with ec2-plugin 1.42.  Switching our spot instances to regular did not fix the issue, but downgrading the plugin to 1.41 did

          Mike Alberghini added a comment - We also faced this problem on Jenkins 2.157 with ec2-plugin 1.42.  Switching our spot instances to regular did not fix the issue, but downgrading the plugin to 1.41 did

          Victor Chavez added a comment - - edited

          On Jenkins 2.150.2 EC2 plugin 1.42 we get this behavior intermittently. We oscillate between this and JENKINS-55556.

          Victor Chavez added a comment - - edited On Jenkins 2.150.2 EC2 plugin 1.42 we get this behavior intermittently. We oscillate between this and  JENKINS-55556 .

          Paul P added a comment - - edited

          EDIT: Looks like I spoke too soon, the problem re-appeared.

          I had this issue and uninstalling the plugin and re-installing it fixed it.

          To be specific:

          1. backup your config.xml
          2. uninstall the plugin
          3. restart jenkins
          4. re-install the plugin (I lost my configuration)
          5. stop jenkins
          6. replace the config.xml
          7. start Jenkins

          I am using Jenkins 2.176 and ec2 plugin 1.42

          Paul P added a comment - - edited EDIT: Looks like I spoke too soon, the problem re-appeared. I had this issue and uninstalling the plugin and re-installing it fixed it. To be specific: backup your config.xml uninstall the plugin restart jenkins re-install the plugin (I lost my configuration) stop jenkins replace the config.xml start Jenkins I am using Jenkins 2.176 and ec2 plugin 1.42

          FABRIZIO MANFREDI added a comment - - edited

          The 1.43 contains a fix on the missing default value of the new options that are not present for people that come from old plugin. Can you test ? 

          FABRIZIO MANFREDI added a comment - - edited The 1.43 contains a fix on the missing default value of the new options that are not present for people that come from old plugin. Can you test ? 

          Yan Corneille added a comment -

          Actully I can't, I'm sorry. I had to rollback and fix my instance EC2 plugin to v1.39 due to https://issues.jenkins-ci.org/browse/JENKINS-52362

          Yan Corneille added a comment - Actully I can't, I'm sorry. I had to rollback and fix my instance EC2 plugin to v1.39 due to https://issues.jenkins-ci.org/browse/JENKINS-52362 . 

          Paul P added a comment -

          I can confirm 1.43 & 2.177 work fine with Spot instances

          Paul P added a comment - I can confirm 1.43 & 2.177 work fine with Spot instances

          Tao Yang added a comment -

          Thanks thoulen, it looks to me like it's working correctly now!

          Tao Yang added a comment - Thanks thoulen , it looks to me like it's working correctly now!

            thoulen FABRIZIO MANFREDI
            tommoyang Tao Yang
            Votes:
            10 Vote for this issue
            Watchers:
            18 Start watching this issue

              Created:
              Updated:
              Resolved: