After update of kubernetes-plugin to 1.27.3 on our production jenkins instance, we started to observe degraded performance of jobs.

Jobs seem to take long time to schedule pods and the queue size is skyrocketing. We are running 100-200 jobs in parallel at times, during our scheduled end to end tests. Normally they get cleared within couple of minutes.

The problem is not apparent until there is spike in jobs scheduled at the same time.

After updated version of kubernetes jobs get stuck and overload the jenkins queue. Our timeout inside of the pipeline kills them after 30 minutes.

We use podTemplate in pipelines to schedule pod, we are not using the "default" jenkins wide templates for pods.

Kubernetes cloud configuration:

Some statistics gathered during with the problematic plugin version (1.27.3 - 1.27.4 is mistake in screenshots):