Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55618

When using spot instances, requests 3-4 nodes when one is needed

XMLWordPrintable

    • Icon: New Feature New Feature
    • Resolution: Unresolved
    • Icon: Blocker Blocker
    • ec2-plugin

      When it is time to request a worker node, more than one can be requested.

      It requests one, then rapidly checks for if its up (node needs a minute or so to come up when using spot instances), and fires another one, then another – until one of spots is finally there on time. This one becomes Jenkins slave, rest linger there until Spot Marketplace kills it.

      Seems to be 1.42 specific: I reverted to version 1.41 and it seems to work fine.

      This is how it looks in Jenkins log file:
      (notice really short delays, only sir-bd78adpq gets into Jenkins slave list)

      Jan 15, 2019 2:17:40 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
      INFO: Spot instance id in provision: sir-dvfib1pn
      Jan 15, 2019 2:17:40 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: SlaveTemplate

      {ami='ami-XXXXX', labels=''}. Attempting provision finished, excess workload: 0
      Jan 15, 2019 2:17:40 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: We have now 1 computers, waiting for 1 more
      Jan 15, 2019 2:17:40 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
      INFO: Started provisioning current_ami (ami-XXXXX) from ec2-ec2 cloud with 1 executors. Remaining excess workload: 0
      Jan 15, 2019 2:17:40 PM hudson.plugins.ec2.EC2Cloud$1 call
      WARNING: SlaveTemplate{ami='ami-XXXXX', labels=''}

      . Node terminated is neither pending, neither running, its

      {2}. Terminate provisioning
      Jan 15, 2019 2:17:48 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: SlaveTemplate{ami='ami-XXXXX', labels=''}. Attempting to provision slave needed by excess workload of 1 units
      Jan 15, 2019 2:17:49 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
      INFO: Launching ami-XXXXX for template current_ami

      Jan 15, 2019 2:17:50 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
      INFO: Spot instance id in provision: sir-bieg943m
      Jan 15, 2019 2:17:50 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: SlaveTemplate{ami='ami-XXXXX', labels=''}. Attempting provision finished, excess workload: 0
      Jan 15, 2019 2:17:50 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: We have now 1 computers, waiting for 1 more
      Jan 15, 2019 2:17:50 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
      INFO: Started provisioning current_ami (ami-XXXXX) from ec2-ec2 cloud with 1 executors. Remaining excess workload: 0
      Jan 15, 2019 2:17:50 PM hudson.plugins.ec2.EC2Cloud$1 call
      WARNING: SlaveTemplate{ami='ami-XXXXX', labels=''}. Node terminated is neither pending, neither running, its {2}

      . Terminate provisioning
      Jan 15, 2019 2:17:58 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: SlaveTemplate

      {ami='ami-XXXXX', labels=''}. Attempting to provision slave needed by excess workload of 1 units

      Jan 15, 2019 2:17:59 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
      INFO: Launching ami-XXXXX for template current_ami

      Jan 15, 2019 2:18:00 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
      INFO: Spot instance id in provision: sir-5mtr9erp
      Jan 15, 2019 2:18:00 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: SlaveTemplate{ami='ami-XXXXX', labels=''}

      . Attempting provision finished, excess workload: 0
      Jan 15, 2019 2:18:00 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: We have now 1 computers, waiting for 1 more
      Jan 15, 2019 2:18:00 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
      INFO: Started provisioning current_ami (ami-XXXXX) from ec2-ec2 cloud with 1 executors. Remaining excess workload: 0
      Jan 15, 2019 2:18:00 PM hudson.plugins.ec2.EC2Cloud$1 call
      WARNING: SlaveTemplate

      {ami='ami-XXXXX', labels=''}. Node terminated is neither pending, neither running, its {2}. Terminate provisioning
      Jan 15, 2019 2:18:08 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: SlaveTemplate{ami='ami-XXXXX', labels=''}

      . Attempting to provision slave needed by excess workload of 1 units
      Jan 15, 2019 2:18:09 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
      INFO: Launching ami-XXXXX for template current_ami
      Jan 15, 2019 2:18:10 PM hudson.plugins.ec2.SlaveTemplate provisionSpot

      INFO: Spot instance id in provision: sir-bd78adpq
      Jan 15, 2019 2:18:10 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: SlaveTemplate

      {ami='ami-XXXXX', labels=''}

      . Attempting provision finished, excess workload: 0
      Jan 15, 2019 2:18:10 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: We have now 1 computers, waiting for 1 more
      Jan 15, 2019 2:18:10 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
      INFO: Started provisioning current_ami (ami-XXXXX) from ec2-ec2 cloud with 1 executors. Remaining excess workload: 0
      Jan 15, 2019 2:18:21 PM hudson.plugins.ec2.EC2Cloud$1 call

            thoulen FABRIZIO MANFREDI
            vnaum Vladislav Naumov
            Votes:
            13 Vote for this issue
            Watchers:
            19 Start watching this issue

              Created:
              Updated: