Jesse Glick added a comment - 2015-11-17 20:58 Another one involving getEstimatedDuration and possibly related: "Computer.threadPoolForRemoting [#6]": waiting for ownable synchronizer 0x000000070623d790, (a java.util.concurrent.locks.ReentrantLock$NonfairSync), which is held by "AtmostOneTaskExecutor[hudson.model.Queue$1@39a19bf0] [#7]" "AtmostOneTaskExecutor[hudson.model.Queue$1@39a19bf0] [#7]": waiting to lock monitor 0x00007f42f00217d8 (object 0x000000070707f938, a java.util.HashMap), which is held by "Computer.threadPoolForRemoting [#6]" "Computer.threadPoolForRemoting [#6]": at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000070623d790> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at hudson.model.Queue._withLock(Queue.java:1205) at hudson.model.Queue.withLock(Queue.java:1143) at hudson.model.Computer.removeExecutor(Computer.java:977) at hudson.model.Executor.finish2(Executor.java:435) at hudson.model.Executor.completedAsynchronous(Executor.java:448) at jenkins.model.queue.AsynchronousExecution.completed(AsynchronousExecution.java:122) - locked <0x0000000707effcd8> (a org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable$1) at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask.finish(ExecutorStepExecution.java:339) - locked <0x000000070707f938> (a java.util.HashMap) at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask.access$400(ExecutorStepExecution.java:144) at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$Callback.finished(ExecutorStepExecution.java:370) at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:114) at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:311) at com.cloudbees.groovy.cps.impl.ValueBoundContinuation.receive(ValueBoundContinuation.java:21) at com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:73) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:137) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:106) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:164) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:271) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:71) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:180) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:178) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "AtmostOneTaskExecutor[hudson.model.Queue$1@39a19bf0] [#7]": at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask.finish(ExecutorStepExecution.java:333) - waiting to lock <0x000000070707f938> (a java.util.HashMap) at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask.run(ExecutorStepExecution.java:281) at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask.getEstimatedDuration(ExecutorStepExecution.java:307) at hudson.model.queue.MappingWorksheet.<init>(MappingWorksheet.java:335) at hudson.model.queue.MappingWorksheet.<init>(MappingWorksheet.java:318) at hudson.model.Queue.maintain(Queue.java:1358) at hudson.model.Queue$1.call(Queue.java:334) at hudson.model.Queue$1.call(Queue.java:331) at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:101) at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:91) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110) at java.lang.Thread.run(Thread.java:745)

Jesse Glick added a comment - 2016-01-05 21:06 In JENKINS-25890 , Queue is locked while calling PlaceholderTask.getFullDisplayName . That was fixed by making isReady , which getFullDisplayName was calling, nonblocking. The Queue lock was acquired in two reported ways: from WorkflowRun.onLoad , and from ExecutorPickle.rehydrate . Here in JENKINS-31614 , Queue is locked while calling PlaceholderTask.getEstimatedDuration , again with two reported ways in which the Queue lock might be acquired: from WorkflowRun.onLoad (again), and PlaceholderTask.finish . In JENKINS-32304 , Queue is locked while calling AfterRestartTask.getEstimatedDuration , while the Queue lock is acquired from ExecutorPickle.rehydrate . There is no straightforward way to make either getEstimatedDuration implementation not acquire the RunMap lock, so the original fix of isReady does not help in either of these cases. Need to instead ensure that the Queue lock is not acquired synchronously from onLoad , finish , or rehydrate . In other words, accept that Queue.maintain implementations may wind up acquiring other locks.

SCM/JIRA link daemon added a comment - 2016-01-08 19:14 Code changed in jenkins User: Jesse Glick Path: CHANGES.md api/src/main/java/org/jenkinsci/plugins/workflow/pickles/Pickle.java job/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java support/src/main/java/org/jenkinsci/plugins/workflow/support/pickles/ExecutorPickle.java support/src/main/java/org/jenkinsci/plugins/workflow/support/pickles/TryRepeatedly.java support/src/main/java/org/jenkinsci/plugins/workflow/support/steps/ExecutorStepExecution.java http://jenkins-ci.org/commit/workflow-plugin/c89a054375f4e58947e9926d0be74abc5174a478 Log: [FIXED JENKINS-31614] Avoid acquiring the Queue lock while holding other locks.

SCM/JIRA link daemon added a comment - 2016-01-08 19:14 Code changed in jenkins User: Jesse Glick Path: CHANGES.md api/src/main/java/org/jenkinsci/plugins/workflow/pickles/Pickle.java job/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java support/src/main/java/org/jenkinsci/plugins/workflow/support/pickles/ExecutorPickle.java support/src/main/java/org/jenkinsci/plugins/workflow/support/pickles/TryRepeatedly.java support/src/main/java/org/jenkinsci/plugins/workflow/support/steps/ExecutorStepExecution.java http://jenkins-ci.org/commit/workflow-plugin/cb65b316d962df06f9fa8d2c2e0a04fdf60e1d52 Log: Merge pull request #289 from jglick/deadlocks- JENKINS-31614 JENKINS-31614 Avoid acquiring the Queue lock while holding other locks Compare: https://github.com/jenkinsci/workflow-plugin/compare/1c87228afa79...cb65b316d962

SCM/JIRA link daemon added a comment - 2016-04-05 19:52 Code changed in jenkins User: Jesse Glick Path: api/src/main/java/org/jenkinsci/plugins/workflow/pickles/Pickle.java http://jenkins-ci.org/commit/workflow-api-plugin/9dda0bdb3b033ec77cbaf6a5fbe57b074eec9010 Log: [FIXED JENKINS-31614] Avoid acquiring the Queue lock while holding other locks. Originally-Committed-As: c89a054375f4e58947e9926d0be74abc5174a478

SCM/JIRA link daemon added a comment - 2016-04-05 20:03 Code changed in jenkins User: Jesse Glick Path: support/src/main/java/org/jenkinsci/plugins/workflow/support/pickles/TryRepeatedly.java http://jenkins-ci.org/commit/workflow-support-plugin/ff9a05c281a59f456052184f063ab09fc119167a Log: [FIXED JENKINS-31614] Avoid acquiring the Queue lock while holding other locks. Originally-Committed-As: c89a054375f4e58947e9926d0be74abc5174a478

SCM/JIRA link daemon added a comment - 2016-04-05 20:31 Code changed in jenkins User: Jesse Glick Path: job/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java http://jenkins-ci.org/commit/workflow-job-plugin/6648bd916b07ae2e6f4a95ecc8eac7adc052725f Log: [FIXED JENKINS-31614] Avoid acquiring the Queue lock while holding other locks. Originally-Committed-As: c89a054375f4e58947e9926d0be74abc5174a478