Jonathan Strickland added a comment - 2014-11-13 17:32 Found that the QueueTaskDispatcher is being called multiple times (LockableResourcesQueueTaskDispatcher) but is appears that there is no distinction in when its being called at different points in the queue. Appears since its being called multiple times and attempting to queue up resources each time that basically get stuck waiting for resources. See below: Nov 13, 2014 11:56:51 AM INFO hudson.WebAppMain$3 run Jenkins is fully up and running Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun Full stack trace is in canRun: [java.lang.Thread.getStackTrace(Thread.java:1568), org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher.canRun(LockableResourcesQueueTaskDispatcher.java:41), hudson.model.Queue.isBuildBlocked(Queue.java:949), hudson.model.Queue.maintain(Queue.java:1018), hudson.model.Queue$1.call(Queue.java:316), hudson.model.Queue$1.call(Queue.java:313), jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:94), jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:84), java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334), java.util.concurrent.FutureTask.run(FutureTask.java:166), hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:104), java.lang.Thread.run(Thread.java:724)] Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun Reserve_Upgrade9k trying to reserve 1 of [ASR9K] Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun PROJECT FULLNAME:Reserve_Upgrade9k NAME:Reserve_Upgrade9k Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Selecting resources for queueItemId:1 queueItemProject:Reserve_Upgrade9k number:1 Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Checking resource ASR9K with project name null with queueItemsId 0 Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Validating rs ASR9K isReserved:false isLocked:false isQueue:false Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Selected size is 1 Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun Reserve_Upgrade9k reserved resources [ASR9K] Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager lock Entering lock resources Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager lock Full stack trace is: [java.lang.Thread.getStackTrace(Thread.java:1568), org.jenkins.plugins.lockableresources.LockableResourcesManager.lock(LockableResourcesManager.java:157), org.jenkins.plugins.lockableresources.queue.LockRunListener.onStarted(LockRunListener.java:48), org.jenkins.plugins.lockableresources.queue.LockRunListener.onStarted(LockRunListener.java:28), hudson.model.listeners.RunListener.fireStarted(RunListener.java:213), hudson.model.Run.execute(Run.java:1755), hudson.matrix.MatrixBuild.run(MatrixBuild.java:306), hudson.model.ResourceController.execute(ResourceController.java:89), hudson.model.Executor.run(Executor.java:240), hudson.model.OneOffExecutor.run(OneOffExecutor.java:43)] Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager lock Exiting lock resources with true Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun Full stack trace is in canRun: [java.lang.Thread.getStackTrace(Thread.java:1568), org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher.canRun(LockableResourcesQueueTaskDispatcher.java:41), hudson.model.Queue.isBuildBlocked(Queue.java:949), hudson.model.Queue.maintain(Queue.java:1018), hudson.model.Queue$1.call(Queue.java:316), hudson.model.Queue$1.call(Queue.java:313), jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:94), jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:84), java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334), java.util.concurrent.FutureTask.run(FutureTask.java:166), hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:104), java.lang.Thread.run(Thread.java:724)] Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun Reserve_Upgrade9k trying to reserve 1 of [ASR9K] Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun PROJECT FULLNAME:Reserve_Upgrade9k NAME:Reserve_Upgrade9k Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Selecting resources for queueItemId:2 queueItemProject:Reserve_Upgrade9k number:1 Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Checking resource ASR9K with project name null with queueItemsId 0 Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Validating rs ASR9K isReserved:false isLocked:true isQueue:false Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Selected size is 0 Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun Reserve_Upgrade9k waiting for resources Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun Full stack trace is in canRun: [java.lang.Thread.getStackTrace(Thread.java:1568), org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher.canRun(LockableResourcesQueueTaskDispatcher.java:41), hudson.model.Queue$BlockedItem.getCauseOfBlockage(Queue.java:1755), hudson.model.Queue$Item.getWhy(Queue.java:1488), hudson.matrix.DefaultMatrixExecutionStrategyImpl.waitForCompletion(DefaultMatrixExecutionStrategyImpl.java:284), hudson.matrix.DefaultMatrixExecutionStrategyImpl.run(DefaultMatrixExecutionStrategyImpl.java:161), hudson.matrix.MatrixBuild$MatrixBuildExecution.doRun(MatrixBuild.java:357), hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:533), hudson.model.Run.execute(Run.java:1759), hudson.matrix.MatrixBuild.run(MatrixBuild.java:306), hudson.model.ResourceController.execute(ResourceController.java:89), hudson.model.Executor.run(Executor.java:240), hudson.model.OneOffExecutor.run(OneOffExecutor.java:43)] Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun Reserve_Upgrade9k trying to reserve 1 of [ASR9K] Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun PROJECT FULLNAME:Reserve_Upgrade9k NAME:Reserve_Upgrade9k Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Selecting resources for queueItemId:2 queueItemProject:Reserve_Upgrade9k number:1 Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Checking resource ASR9K with project name null with queueItemsId 0 Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Validating rs ASR9K isReserved:false isLocked:true isQueue:false Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Selected size is 0 Nov 13, 2014 12:11:41 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun Reserve_Upgrade9k waiting for resources Nov 13, 2014 12:11:42 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun Full stack trace is in canRun: [java.lang.Thread.getStackTrace(Thread.java:1568), org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher.canRun(LockableResourcesQueueTaskDispatcher.java:41), hudson.model.Queue$BlockedItem.getCauseOfBlockage(Queue.java:1755), hudson.model.Queue$Item.getWhy(Queue.java:1488), hudson.matrix.DefaultMatrixExecutionStrategyImpl.waitForCompletion(DefaultMatrixExecutionStrategyImpl.java:284), hudson.matrix.DefaultMatrixExecutionStrategyImpl.run(DefaultMatrixExecutionStrategyImpl.java:161), hudson.matrix.MatrixBuild$MatrixBuildExecution.doRun(MatrixBuild.java:357), hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:533), hudson.model.Run.execute(Run.java:1759), hudson.matrix.MatrixBuild.run(MatrixBuild.java:306), hudson.model.ResourceController.execute(ResourceController.java:89), hudson.model.Executor.run(Executor.java:240), hudson.model.OneOffExecutor.run(OneOffExecutor.java:43)] Nov 13, 2014 12:11:42 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun Reserve_Upgrade9k trying to reserve 1 of [ASR9K] Nov 13, 2014 12:11:42 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun PROJECT FULLNAME:Reserve_Upgrade9k NAME:Reserve_Upgrade9k Nov 13, 2014 12:11:42 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Selecting resources for queueItemId:2 queueItemProject:Reserve_Upgrade9k number:1 Nov 13, 2014 12:11:42 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Checking resource ASR9K with project name null with queueItemsId 0 Nov 13, 2014 12:11:42 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Validating rs ASR9K isReserved:false isLocked:true isQueue:false Nov 13, 2014 12:11:42 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Selected size is 0 Nov 13, 2014 12:11:42 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun Reserve_Upgrade9k waiting for resources Nov 13, 2014 12:11:43 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun Full stack trace is in canRun: [java.lang.Thread.getStackTrace(Thread.java:1568), org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher.canRun(LockableResourcesQueueTaskDispatcher.java:41), hudson.model.Queue.isBuildBlocked(Queue.java:949), hudson.model.Queue.maintain(Queue.java:1001), hudson.model.Queue$MaintainTask.doRun(Queue.java:2050), hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54), java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471), java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351), java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178), java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178), java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293), java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145), java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615), java.lang.Thread.run(Thread.java:724)] Nov 13, 2014 12:11:43 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun Reserve_Upgrade9k trying to reserve 1 of [ASR9K] Nov 13, 2014 12:11:43 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun PROJECT FULLNAME:Reserve_Upgrade9k NAME:Reserve_Upgrade9k Nov 13, 2014 12:11:43 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Selecting resources for queueItemId:2 queueItemProject:Reserve_Upgrade9k number:1 Nov 13, 2014 12:11:43 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Checking resource ASR9K with project name null with queueItemsId 0 Nov 13, 2014 12:11:43 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Validating rs ASR9K isReserved:false isLocked:true isQueue:false Nov 13, 2014 12:11:43 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Selected size is 0 Nov 13, 2014 12:11:43 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun Reserve_Upgrade9k waiting for resources Nov 13, 2014 12:11:43 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun Full stack trace is in canRun: [java.lang.Thread.getStackTrace(Thread.java:1568), org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher.canRun(LockableResourcesQueueTaskDispatcher.java:41), hudson.model.Queue$BlockedItem.getCauseOfBlockage(Queue.java:1755), hudson.model.Queue$Item.getWhy(Queue.java:1488), hudson.matrix.DefaultMatrixExecutionStrategyImpl.waitForCompletion(DefaultMatrixExecutionStrategyImpl.java:284), hudson.matrix.DefaultMatrixExecutionStrategyImpl.run(DefaultMatrixExecutionStrategyImpl.java:161), hudson.matrix.MatrixBuild$MatrixBuildExecution.doRun(MatrixBuild.java:357), hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:533), hudson.model.Run.execute(Run.java:1759), hudson.matrix.MatrixBuild.run(MatrixBuild.java:306), hudson.model.ResourceController.execute(ResourceController.java:89), hudson.model.Executor.run(Executor.java:240), hudson.model.OneOffExecutor.run(OneOffExecutor.java:43)] Nov 13, 2014 12:11:43 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun Reserve_Upgrade9k trying to reserve 1 of [ASR9K] Nov 13, 2014 12:11:43 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun PROJECT FULLNAME:Reserve_Upgrade9k NAME:Reserve_Upgrade9k Nov 13, 2014 12:11:43 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Selecting resources for queueItemId:2 queueItemProject:Reserve_Upgrade9k number:1 Nov 13, 2014 12:11:43 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Checking resource ASR9K with project name null with queueItemsId 0 Nov 13, 2014 12:11:43 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Validating rs ASR9K isReserved:false isLocked:true isQueue:false Nov 13, 2014 12:11:43 PM INFO org.jenkins.plugins.lockableresources.LockableResourcesManager queue Selected size is 0 Nov 13, 2014 12:11:43 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun Reserve_Upgrade9k waiting for resources Nov 13, 2014 12:11:44 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun Full stack trace is in canRun: [java.lang.Thread.getStackTrace(Thread.java:1568), org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher.canRun(LockableResourcesQueueTaskDispatcher.java:41), hudson.model.Queue$BlockedItem.getCauseOfBlockage(Queue.java:1755), hudson.model.Queue$Item.getWhy(Queue.java:1488), hudson.matrix.DefaultMatrixExecutionStrategyImpl.waitForCompletion(DefaultMatrixExecutionStrategyImpl.java:284), hudson.matrix.DefaultMatrixExecutionStrategyImpl.run(DefaultMatrixExecutionStrategyImpl.java:161), hudson.matrix.MatrixBuild$MatrixBuildExecution.doRun(MatrixBuild.java:357), hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:533), hudson.model.Run.execute(Run.java:1759), hudson.matrix.MatrixBuild.run(MatrixBuild.java:306), hudson.model.ResourceController.execute(ResourceController.java:89), hudson.model.Executor.run(Executor.java:240), hudson.model.OneOffExecutor.run(OneOffExecutor.java:43)] Nov 13, 2014 12:11:44 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun Reserve_Upgrade9k trying to reserve 1 of [ASR9K] Nov 13, 2014 12:11:44 PM INFO org.jenkins.plugins.lockableresources.queue.LockableResourcesQueueTaskDispatcher canRun PROJECT FULLNAME:Reserve_Upgrade9k NAME:Reserve_Upgrade9k

Jonathan Strickland added a comment - 2014-11-17 01:25 Seems to be only related to multi-configuration jobs. Captured some debugs: First task called is hudson.matrix.MatrixProject, then multiple hudson.matrix.MatrixConfiguration task. All initiate QueueTaskDispatcher, causing some eventual problems as seen with exhaustion of resources.