Status: Fixed but Unreleased (View Workflow)
Simple steps are taking too long to be finished when we are running multiple stages (10+) in parallel (assuming that each stage creates a pod).
A step like a simple `which` command that under normal conditions should take less than 1s is taking 40s. I attached 2 pictures, one when we are running only 4 stages in parallel and the other when there are 30 stages. Note in the other image attached (conflicting_times) that according to the logs a task runs in ~6s however Jenkins takes 27s to recognize it as done.
The same pipeline with 70 parallel stages running on static pods (without using the plugin) on Kubernetes with the same Docker image takes ~7 min to be completed. Whereas this pipeline with the same 70 stages, is taking ~25 minutes under elastic agents (Kubernetes Plugin).
I've attached the JNLP container logs as well although it does not have much information.
Upgrading Kubernetes plugin (1.28.7) and increasing the connection max number to 128 seem to fixed the issue.
https://github.com/jenkinsci/kubernetes-plugin/pull/826 should have helped with that. Please try with recent versions (1.27.x or later)
Also consider raising the maximum number of connections in the Cloud configuration. It defaults to 64, and each call inside a container block holds a websocket connection while it is running, which can cause additional calls to wait or event time out.