-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins 2.289.2
kubernetes:1.30.1
We are using some complex declarative pipelines that use multiple pods and trigger additional pipelines. We're seeing some cases of deadlock when a job has 2 stages, each with its own pod. When the container cap is hit, those jobs might get stuck waiting for the second pod to be created but since the cap is hit, no pod will every be created. Eventually the job will timeout.
In a simple case:
Set container cap 2
Create a pipeline like this:
pipeline { agent { kubernetes { inheritFrom 'jnlp' } } stages { stage('build') { agent { kubernetes { inheritFrom 'jnlp' } } steps { sh "sleep 600" } } stage('test') { agent { kubernetes { inheritFrom 'jnlp' } } steps { sh "sleep 600" } } } }
Then start two jobs. After the first stage completes, both jobs will be stuck, each holding onto a pod while waiting for a new pod to come up. No pod will come because we're already at the cap.
Still waiting to schedule task All nodes of label ‘test-pipeline_2-vhvn8’ are offline
And the job will wait there until it times out.
Even with this trivial case, we've seen this happen at container cap 10 if enough jobs are queued.