From what I can tell, it has to do with how getActiveSlavePods relies on Kubernetes API to tell it what is currently running. There will always be an inherent delay between launch and this showing pods in the API so this approach will likely never be stable. IMO, a singleton that manages the instantiation of the slaves and their states would be the more appropriate solution. I plan to take a look at this next week as it's killing our Jenkins DR automation when our multibranch projects queue up 2k jobs on the first run after scanning.

I have a similar problem. The plugin execute as much pods as possible whenever it is under the limit. And stop when it is over the limit. But doesn't start just the proper amount of pods to don't exceed the limit. Example:

If the status is this:

  Pods Limit: 30

  Running: 30

  Queued: 50

As soon as one running pod finish, Jenkins is going to execute the 50 queued ones at the same time, so you will finish with:

  Pods Limit: 30

  Running: 30 -1 (finished) + 50 = 79

  Queued: 0

So, the limit is not respected at all

I was expecting that the plugin can manage the internal queue, even running the pods in the correct order

Hi!

Same issue here

Plugin : 1.24.1
Core : 2.204.3

2020-02-29 22:57:35.179+0000 [id=30] INFO o.c.j.p.k.KubernetesCloud#addProvisionedSlave: Maximum number of concurrently running agent pods (10) reached for Kubernetes Cloud kubernetes, not provisioning: 23 running or pending in namespace jenkins with Kubernetes labels {jenkins=slave}
2020-02-29 22:57:35.179+0000 [id=30] INFO o.c.j.p.k.KubernetesCloud#provision: Excess workload after pending Kubernetes agents: 1

Thanks

I have noticed that the limit are better respected when not using the NoDelayProvisioningStrategy (using the system property -Dio.jenkins.plugins.kubernetes.disableNoDelayProvisioning=true)

We are also frequently running into the same issue.

I agree, that in theory the container cap is checked, but as soon as one spot opens up, all jobs waiting in the queue are triggering a "node provision" action. Depending on the actual queue size, this will lead to a massive over-provisioning of agents/pods. As a side effect of the produced system-load, most of the provisioned pods end up in some kind of network-timeout and are never removed from jenkins or the cloud-environment. Afterwards we are forced to manually clean up the mess by hand. Changing the provisioning strategy to NoDelayProvisioningStrategy has no meaningful effect for us. For now, we are forced to deactivate all automatic triggering of our ~ 1.5k jobs.

Plugin : 1.26.2
Core : 2.235.1