Jesse Glick added a comment - 2015-11-17 20:58 Another one involving getEstimatedDuration and possibly related: "Computer.threadPoolForRemoting [#6]": waiting for ownable synchronizer 0x000000070623d790, (a java.util.concurrent.locks.ReentrantLock$NonfairSync), which is held by "AtmostOneTaskExecutor[hudson.model.Queue$1@39a19bf0] [#7]" "AtmostOneTaskExecutor[hudson.model.Queue$1@39a19bf0] [#7]": waiting to lock monitor 0x00007f42f00217d8 (object 0x000000070707f938, a java.util.HashMap), which is held by "Computer.threadPoolForRemoting [#6]" "Computer.threadPoolForRemoting [#6]": at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000070623d790> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at hudson.model.Queue._withLock(Queue.java:1205) at hudson.model.Queue.withLock(Queue.java:1143) at hudson.model.Computer.removeExecutor(Computer.java:977) at hudson.model.Executor.finish2(Executor.java:435) at hudson.model.Executor.completedAsynchronous(Executor.java:448) at jenkins.model.queue.AsynchronousExecution.completed(AsynchronousExecution.java:122) - locked <0x0000000707effcd8> (a org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable$1) at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask.finish(ExecutorStepExecution.java:339) - locked <0x000000070707f938> (a java.util.HashMap) at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask.access$400(ExecutorStepExecution.java:144) at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$Callback.finished(ExecutorStepExecution.java:370) at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:114) at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:311) at com.cloudbees.groovy.cps.impl.ValueBoundContinuation.receive(ValueBoundContinuation.java:21) at com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:73) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:137) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:106) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:164) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:271) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:71) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:180) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:178) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "AtmostOneTaskExecutor[hudson.model.Queue$1@39a19bf0] [#7]": at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask.finish(ExecutorStepExecution.java:333) - waiting to lock <0x000000070707f938> (a java.util.HashMap) at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask.run(ExecutorStepExecution.java:281) at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask.getEstimatedDuration(ExecutorStepExecution.java:307) at hudson.model.queue.MappingWorksheet.<init>(MappingWorksheet.java:335) at hudson.model.queue.MappingWorksheet.<init>(MappingWorksheet.java:318) at hudson.model.Queue.maintain(Queue.java:1358) at hudson.model.Queue$1.call(Queue.java:334) at hudson.model.Queue$1.call(Queue.java:331) at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:101) at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:91) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110) at java.lang.Thread.run(Thread.java:745)

Jesse Glick added a comment - 2016-01-05 21:06 In JENKINS-25890 , Queue is locked while calling PlaceholderTask.getFullDisplayName . That was fixed by making isReady , which getFullDisplayName was calling, nonblocking. The Queue lock was acquired in two reported ways: from WorkflowRun.onLoad , and from ExecutorPickle.rehydrate . Here in JENKINS-31614 , Queue is locked while calling PlaceholderTask.getEstimatedDuration , again with two reported ways in which the Queue lock might be acquired: from WorkflowRun.onLoad (again), and PlaceholderTask.finish . In JENKINS-32304 , Queue is locked while calling AfterRestartTask.getEstimatedDuration , while the Queue lock is acquired from ExecutorPickle.rehydrate . There is no straightforward way to make either getEstimatedDuration implementation not acquire the RunMap lock, so the original fix of isReady does not help in either of these cases. Need to instead ensure that the Queue lock is not acquired synchronously from onLoad , finish , or rehydrate . In other words, accept that Queue.maintain implementations may wind up acquiring other locks.

SCM/JIRA link daemon added a comment - 2016-01-08 19:14 Code changed in jenkins User: Jesse Glick Path: CHANGES.md api/src/main/java/org/jenkinsci/plugins/workflow/pickles/Pickle.java job/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java support/src/main/java/org/jenkinsci/plugins/workflow/support/pickles/ExecutorPickle.java support/src/main/java/org/jenkinsci/plugins/workflow/support/pickles/TryRepeatedly.java support/src/main/java/org/jenkinsci/plugins/workflow/support/steps/ExecutorStepExecution.java http://jenkins-ci.org/commit/workflow-plugin/c89a054375f4e58947e9926d0be74abc5174a478 Log: [FIXED JENKINS-31614] Avoid acquiring the Queue lock while holding other locks.