Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29851

global instance cap not respected for spot instances

      When the build queue piles up new instances are launched. One per waiting job, it seems. I've attached a screenshot of my configuration. The instance cap per AMI doesn't have any effect either.

      I believe this may be related to spot instances not coming up with the Jenkins slave tags, and so not being counted correctly. Though the global cap should apply to the overall number of running instances, no?

          [JENKINS-29851] global instance cap not respected for spot instances

          Arien Kock created issue -
          Arien Kock made changes -
          Description Original: When the build queue piles up new instances are launched. One per waiting job, it seems. I've attached a screenshot of my configuration. The instance cap per AMI doesn't have any effect either. New: When the build queue piles up new instances are launched. One per waiting job, it seems. I've attached a screenshot of my configuration. The instance cap per AMI doesn't have any effect either.

          I believe this may be related to spot instances not coming up with the Jenkins slave tags, and so not being counted correctly. Though the global cap should apply to the overall number of running instances, no?

          Ryan Aslett added a comment -

          This keeps biting us. We had global instance cap of 30, and a per cloud cap of 30, as well as cc28xl caps at aws. Turns out that our aws caps only applied to on demand instances, and so did jenkins. i.e. we had 90 machines running at once.

          Ryan Aslett added a comment - This keeps biting us. We had global instance cap of 30, and a per cloud cap of 30, as well as cc28xl caps at aws. Turns out that our aws caps only applied to on demand instances, and so did jenkins. i.e. we had 90 machines running at once.
          Archie Brentano made changes -
          Priority Original: Minor [ 4 ] New: Major [ 3 ]

          I think this is major, since it causes money to be spent.

          Archie Brentano added a comment - I think this is major, since it causes money to be spent.

          Ryan Aslett added a comment - - edited

          I've narrowed this down to what I think are is the problem in countCurrentEC2Slaves:

          First, it is only looping through the reservations and calling describeInstances : https://github.com/jenkinsci/ec2-plugin/blob/master/src/main/java/hudson/plugins/ec2/EC2Cloud.java#L232-232

          This will only count spot requests that have been fulfilled, not pending spot requests as they have not yet become an instance.

          What is probably needed is a call to describeSpotInstanceRequests() to get the spot instances, and additionally increment n if there are pending spot requests. Either that or there needs to be an entirely separate set of checks for the maximum number of spot requests.

          We have a cron that can sometimes send as many as 100 jobs at once, and we end up with spikes of 100 machines provisioned even though we set our limit to 20. It would be ideal if we could use spot instances + limits on the number running + pending.

          Ryan Aslett added a comment - - edited I've narrowed this down to what I think are is the problem in countCurrentEC2Slaves: First, it is only looping through the reservations and calling describeInstances : https://github.com/jenkinsci/ec2-plugin/blob/master/src/main/java/hudson/plugins/ec2/EC2Cloud.java#L232-232 This will only count spot requests that have been fulfilled, not pending spot requests as they have not yet become an instance. What is probably needed is a call to describeSpotInstanceRequests() to get the spot instances, and additionally increment n if there are pending spot requests. Either that or there needs to be an entirely separate set of checks for the maximum number of spot requests. We have a cron that can sometimes send as many as 100 jobs at once, and we end up with spikes of 100 machines provisioned even though we set our limit to 20. It would be ideal if we could use spot instances + limits on the number running + pending.
          Mohsen K made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]
          Mohsen K made changes -
          Status Original: In Progress [ 3 ] New: Open [ 1 ]

          James Judd added a comment -

          We have run into this as well. Probably won't have time to look into this before January.

          James Judd added a comment - We have run into this as well. Probably won't have time to look into this before January.

          Francis Upton added a comment -

          s/b resolved in 1.30

          Francis Upton added a comment - s/b resolved in 1.30

            francisu Francis Upton
            arienkock Arien Kock
            Votes:
            6 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: