[JENKINS-22558] Deadlock condition when provisioning nodes and running the retention strategy to remove nodes - Jenkins Jira

Type: Bug
Resolution: Fixed
Priority: Critical
Component/s: ec2-plugin
Labels:
- deadlock
- threads

Similar Issues:
Powered by SuggestiMate

Show

There's a racing condition between node provisioning and the retention strategy to remove idle nodes.

Using the threadDump, I retrieved the thread conditions when the deadlock happened:

Here's the Retention Strategy Timer thread:

jenkins.util.Timer [#6]

"jenkins.util.Timer [#6]" Id=51 Group=main BLOCKED on hudson.model.Hudson@2fdebb41 owned by "Computer.threadPoolForRemoting [#1450]" Id=13155
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1706)
	-  blocked on hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@5e642b0a
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:66)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)

	Number of locked synchronizers = 1

Here's the Provivisioning thread:

Computer.threadPoolForRemoting [#1450]

"Computer.threadPoolForRemoting [#1450]" Id=13155 Group=main BLOCKED on hudson.plugins.ec2.EC2SpotRetentionStrategy@5e642b0a owned by "jenkins.util.Timer [#6]" Id=51
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:65)
	-  blocked on hudson.plugins.ec2.EC2SpotRetentionStrategy@5e642b0a
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@208a81be
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@74c59c5f
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@1e4b63aa
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@edff3d5
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@6f859bc2
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@4b22eb23
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@4f008c44
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@252a5c30
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@1c5a1b87
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@360fa9ea
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@241929b
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@27f3ec80
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@1d08a504
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@c7c8313
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@3e3587e8
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@7a3ee5db
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@202ba370
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@2a3129be
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@38738704
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@c4748bb
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@654666f2
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@7d635e40
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@35b25a1e
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@e9d7bb2
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@2712efe8
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@712322b
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@5fbd1834
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@57068640
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@bd12c5d
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.removeNode(Jenkins.java:1712)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2SpotSlave.terminate(EC2SpotSlave.java:82)
	at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:249)
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:73)
	-  locked hudson.plugins.ec2.EC2SpotRetentionStrategy@34b01559
	at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:39)
	at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:663)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
	-  locked java.lang.Object@7f564ca9
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
	at jenkins.model.Jenkins.setNodes(Jenkins.java:1717)
	at jenkins.model.Jenkins.addNode(Jenkins.java:1699)
	-  locked hudson.model.Hudson@2fdebb41
	at hudson.plugins.ec2.EC2Cloud$1.call(EC2Cloud.java:360)
	at hudson.plugins.ec2.EC2Cloud$1.call(EC2Cloud.java:355)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)

	Number of locked synchronizers = 1
	- java.util.concurrent.ThreadPoolExecutor$Worker@126cd40b

is related to

JENKINS-15820 JClouds plugin deadlock

Closed

Assignee:: Francis Upton

Reporter:: taksan

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Created:: 2014-04-10 02:08

Updated:: 2016-01-11 20:46

Resolved:: 2016-01-11 20:46

Details

Description

Attachments

Issue Links

Activity

People

Dates