Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-72361

ec2-fleet-plugin won't provision after scaling down to 0 instances in auto scaling group


    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • ec2-fleet-plugin
    • Jenkins LTS 2.426.1, ec2-fleet-plugin 3.1.0, running on amazon linux 2 in EC2

      I'm currently trying to move from ec2-plugin to ec2-fleet-plugin plugin, but I'm seeing that for some stages, the build doesn't ever provision a node. This seems to happen when trying to scale out shortly after scaling in.

      My build looks roughly like this:

      1. build step on a single node
      2. test in parallel on 77 nodes (these get provisioned and then discarded because I have "Maximum Total Uses" set to 1)
      3. collect coverage reports on a single node

      Everything works nicely until the 3rd step, where it gets stuck on:

      All nodes of label ec2-fleet are offline

      When I check the system logs, I see this on repeat:

      Nov 21, 2023 3:08:12 PM FINE com.amazon.jenkins.ec2fleet.NoDelayProvisionStrategy apply
      label [ec2-fleet]: queueLength 1 availableCapacity 1 (availableExecutors 0 plannedCapacitySnapshot 1 additionalPlannedCapacity 0)
      Nov 21, 2023 3:08:12 PM INFO com.amazon.jenkins.ec2fleet.NoDelayProvisionStrategy apply
      label [ec2-fleet]: No excess workload, provisioning not needed.

      I'm especially suspicious of "plannedCapacitySnapshot 1", which, if I'm reading the source code right, seems to mean that it thinks it's already started scaling up another node (and is waiting for it to come online?) but it never does.

      Other misc info, may or may not be relevant:

      • All steps use the same label ("ec2-fleet") and run one executor per node.
      • cloud configuration includes:
      • Minimum Cluster Size: 0
      • Maximum Cluster Size: 2000
      • Minimum Spare Size: 0
      • Maximum Total Uses: 1

      Sorry for the duplicate, but I also filed this on github issues here: https://github.com/jenkinsci/ec2-fleet-plugin/issues/425

            schmutze Chad Schmutzer
            codyrank Cody Rank
            1 Vote for this issue
            1 Start watching this issue