Found one Java-level deadlock: ============================= "Thread-5": waiting to lock monitor 0x00007f0984170b38 (object 0x0000000706fe3aa8, a hudson.model.RunMap), which is held by "Jenkins initialization thread" "Jenkins initialization thread": waiting to lock monitor 0x00007f0988015128 (object 0x00000007066b46c0, a hudson.model.Queue), which is held by "Thread-5" Java stack information for the threads listed above: =================================================== "Thread-5": at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:688) - waiting to lock <0x0000000706fe3aa8> (a hudson.model.RunMap) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:671) at jenkins.model.lazy.AbstractLazyLoadRunMap.getById(AbstractLazyLoadRunMap.java:543) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:523) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:533) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getFlowExecution(CpsStepContext.java:386) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getProgramPromise(CpsStepContext.java:230) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.isReady(CpsStepContext.java:236) at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask.run(ExecutorStepExecution.java:262) at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask.getDisplayName(ExecutorStepExecution.java:281) at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask.getFullDisplayName(ExecutorStepExecution.java:290) at hudson.model.LoadBalancer$1.assignGreedily(LoadBalancer.java:107) at hudson.model.LoadBalancer$1.map(LoadBalancer.java:97) at hudson.model.LoadBalancer$2.map(LoadBalancer.java:148) at hudson.model.Queue.maintain(Queue.java:1053) - locked <0x00000007066b46c0> (a hudson.model.Queue) at hudson.model.Queue$1.call(Queue.java:316) at hudson.model.Queue$1.call(Queue.java:313) at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:94) at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:84) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:104) at java.lang.Thread.run(Thread.java:745) "Jenkins initialization thread": at hudson.model.Queue.schedule2(Queue.java:639) - waiting to lock <0x00000007066b46c0> (a hudson.model.Queue) at org.jenkinsci.plugins.workflow.support.pickles.ExecutorPickle.rehydrate(ExecutorPickle.java:67) at org.jenkinsci.plugins.workflow.support.pickles.serialization.PickleResolver.rehydrate(PickleResolver.java:68) at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverReader.restorePickles(RiverReader.java:128) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.loadProgramAsync(CpsFlowExecution.java:401) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:379) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:300) at hudson.model.RunMap.retrieve(RunMap.java:219) at hudson.model.RunMap.retrieve(RunMap.java:56) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:688) - locked <0x0000000706fe3aa8> (a hudson.model.RunMap) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:671) at jenkins.model.lazy.AbstractLazyLoadRunMap.getById(AbstractLazyLoadRunMap.java:543) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:523) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:533) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:59) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:51) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:165) at jenkins.model.Jenkins.<init>(Jenkins.java:845) at hudson.model.Hudson.<init>(Hudson.java:82) at hudson.model.Hudson.<init>(Hudson.java:78) at hudson.WebAppMain$3.run(WebAppMain.java:222)
Ironically, StepContext.isReady is what is supposed to be breaking deadlocks, yet here it is acquiring a lock.
Since getFlowExecution may block, I think getProgramPromise should be made to return a future which encompasses both getting the execution, and its programPromise.
- depends on
-
JENKINS-26130 Print progress of pending pickles
- Resolved
- is blocking
-
JENKINS-26513 Deserialization error of ExecutorStepExecution
- Resolved
- is related to
-
JENKINS-31614 Deadlocks involving Queue
- Resolved
-
JENKINS-32304 Deadlock between RunMap and Queue in ExecutorPickle.rehydrate
- Resolved
- links to
(1 links to)