I'm using the digitalocean2 backend. I have set Max no. of instances to 3 however I am watching it spin up 8 instances in response to a job queue. I'm not sure it has done this all the time but I'm not really sure why it's doing it now. No errors in jenkins.log. the attached screenshots illustrate all 8 nodes happily registered, now idle for the next 30 minutes, in direct contradiction to the 3 instances max setting.
in the logs, I can see a whole bunch of messages for the instance cap reached (seven of them). I would guess that several workers were launched independently, each of them seeing that the instance cap of "3" wasn't reached, yet they were not coordinating with each other. I think we had this problem w/ the AWS plugin years ago also, but as these nodes represent money I'm spending this is a really critical problem.
2020-09-20 18:16:15.780+0000 [id=105463] INFO o.jclouds.logging.jdk.JDKLogger#logInfo: >> adding node location(nyc1) name(basic-672)
image(70119657) hardware(c-4)
2020-09-20 18:16:15.862+0000 [id=105464] INFO o.jclouds.logging.jdk.JDKLogger#logInfo: >> adding node location(nyc1) name(basic-142)
image(70119657) hardware(c-4)
2020-09-20 18:16:25.191+0000 [id=38] INFO j.p.jclouds.compute.JCloudsCloud#provision: Instance cap reached while adding capacity for label fast
2020-09-20 18:16:35.367+0000 [id=34] INFO j.p.jclouds.compute.JCloudsCloud#provision: Instance cap reached while adding capacity for label fast
2020-09-20 18:16:45.193+0000 [id=32] INFO j.p.jclouds.compute.JCloudsCloud#provision: Instance cap reached while adding capacity for label fast
2020-09-20 18:16:55.204+0000 [id=37] INFO j.p.jclouds.compute.JCloudsCloud#provision: Instance cap reached while adding capacity for label fast
2020-09-20 18:17:05.268+0000 [id=39] INFO j.p.jclouds.compute.JCloudsCloud#provision: Instance cap reached while adding capacity for label fast
2020-09-20 18:17:15.185+0000 [id=32] INFO j.p.jclouds.compute.JCloudsCloud#provision: Instance cap reached while adding capacity for label fast
2020-09-20 18:17:25.198+0000 [id=39] INFO j.p.jclouds.compute.JCloudsCloud#provision: Instance cap reached while adding capacity for label fast
yeah I looked in my logs and can see it often has spun up just the three instances, but occasionally it spun up eight.
I know why the number is eight. A worker thread in Jenkins says, "let's spin up three instances, since we have none". Then another worker thread says the same thing at the same time. now there are six provisioning. Then a third worker comes in says "let's spin up three instances! " but it sees one of them, so it only hits two.
I've set the number to two. the theory is now I will get sometimes only two nodes, sometimes five. Or maybe three.