this is a major issue that is costing me real cash. over at http://jenkins.sqlalchemy.org/ , a new build will add about ten jobs simultaneously, and despite my instance limit of two, I immediately get five or more instances starting up, which I then have to go and kill manually or pay the price, literally. It seems a lot like the EC2 plugin is spawning based on the number of EC2 instances that amazon reports that are running, without taking into account the number of instances the plugin is already starting to spawn but won't yet be reported by amazon.

pull request at https://github.com/jenkinsci/ec2-plugin/pull/25, seems to be working for me.

Code changed in jenkins
User: Mike Bayer
Path:
src/main/java/hudson/plugins/ec2/EC2Cloud.java
http://jenkins-ci.org/commit/ec2-plugin/8d0a9fa1d53d148601e33cb68d50b47b5ff97276
Log:

• add a new mechanism to help count the total number of EC2 instances for a particular AMI. As an EC2Slave
  is being provisioned, a temporary count is placed in a HashMap, which is used in addition to the
  count reported by Amazon itself for a particular ami. This is so that the count of total
  nodes provisioned takes into account those which Amazon doesn't report yet. The count returned
  may be too high, if amazon reports it in addition to the "provision" count, but better to err on the
  side of not spawning a node too soon; it will get spawned on the next go-around.
  Tries to fix JENKINS-6691.

I have another patch coming for this shortly that (in my testing) completely resolves the problem of excess instances being launched when there is work queued, when the underlying reason for the excess launches is that the new instances take a non-zero amount of time to come online and accept work. My fix involves a small change to the Jenkins core API (in the Cloud class) so that will have to be addressed first. Stay tuned.

A related issue is that the nodes launched do not take into account the number of executors that will be provided by the new slave. Something like the below might address that.

diff --git a/src/main/java/hudson/plugins/ec2/EC2Cloud.java b/src/main/java/hudson/plugins/ec2/EC2Cloud.java
index 8952222..06e22b9 100644
--- a/src/main/java/hudson/plugins/ec2/EC2Cloud.java
+++ b/src/main/java/hudson/plugins/ec2/EC2Cloud.java
@@ -319,7 +319,7 @@ public abstract class EC2Cloud extends Cloud {
             final SlaveTemplate t = getTemplate(label);
             int amiCap = t.getInstanceCap();
 
-            for( ; excessWorkload>0; excessWorkload-- ) {
+            while (excessWorkload>0) {
 
                 if (!addProvisionedSlave(t.ami, amiCap)) {
                     break;
@@ -350,6 +350,9 @@ public abstract class EC2Cloud extends Cloud {
                             }
                         })
                         ,t.getNumExecutors()));
+
+                excessWorkload -= t.getNumExecutors();
+
             }
             return r;
         } catch (AmazonClientException e) {

This has already been addressed in the master branch of the plugin, and will be included in the next release. Thanks!

Cool; which commit fixes this? Thanks!

Ahh... oops. I'm wrong

This is fixed in a branch of mine that is queued for the 1.19 release of the plugin; there is a build currently being tested to be the 1.18 release. Once that is pushed out Francis plans to merge my template-instance-cap-fix branch, which corrects this issue among others, but requires Jenkins 1.503 or later. If you'd like to help test my branch, it's at github.com/kpfleming/jenkins-ec2-plugin, called template-instance-cap-fix.

Can some generous soul point me to the changes/diffs that were introduced to remedy this bug?

Thank you!

This appears to be fixed thanks to Mike Bayer <mike_mp@zzzcomputing.com> on 10/9/12 6:42 PM