Problem Statement

On a given Jenkins instance, a desired cloud-node type (e.g. Ubuntu) may not spin up for hours because another cloud-node type (e.g. Win7) has saturated the cloud's quota. Due to node retention times, this can happen even if the existing cloud nodes are idle, resulting in wasted computing power!

Steps to Reproduce

• Start with an idle Jenkins and idle cloud quota of 10 nodes
• Start 10 builds that require Win7 images
• Allow each build to run and complete, e.g. 1 hour
• Start 1 build that requires Ubuntu

Expected Results

• The Ubuntu build enters the build queue
• An idle Win7 node should be killed and spin up with Ubuntu
• Now the cloud quota of 10 is maxed at 9 idle Win7 nodes and 1 running Ubuntu node

Actual Results

• The Ubuntu build enters the build queue
• All 10 Win7 nodes continue to stay idle
• No Ubuntu node is spun up because the cloud quota is maxed out at 10
• The retention time expires after (default) 30 minutes, and the Win7 nodes are killed
• An Ubuntu image spins up and starts building the build requested 30 minutes ago

We prefer a long retention time because we're using our private cloud instead of paying for usage from a 3rd party. We want cloud nodes to live a while so we can avoid spin-up time when reused. But we don't want that behavior at the expense of queuing other builds while cloud nodes are idle.