We a have setup with multiple AMI that are tag based. Here is the scenario that is causing a problem.
AMI_1 has tag ONE and only runs Jenkins jobs with that tag. This AMI has an instance cap of 1 and the executor count on the slave is 1 as well, meaning there can be at most one Jenkins job with tag ONE running at a time.
AMI_2 has no tags and is used as much as possible. It also has an instance cap of 6 and each slave has 4 executors.
The overall max cap is 10.
Jenkins is currently running one instance of AMI_1 (at cap) and two running instances and 2 stopped instances of AMI_2 (below cap). All executors are active. In the queue there is 1 job waiting for AMI_1 and 6 jobs waiting for AMI_2.
What happens is the plugin tries provision a new slave for AMI_1 but cannot because it is at cap, and then DOES NOT attempt to provision (start) any AMI_2 slaves.
I looked through the code and saw the following in EC2Cloud.java:
I think that
possibleSlavesCount < 0
possibleSlavesCount <= 0
This issue does resolve itself once the queue for AMI_1 is empty, but it can mean an unnecessary long wait for jobs to complete when there should available capacity.