See the attached threaddump.txt which captures a deadlock situation in our Jenkins.

As far as my analysis goes it feels that this is caused by the spotinst plugin.

Two locks are involved:

• the build queue lock
• the lock of the spotinstance agent

And as far as I can see hudson.slaves.ComputerRetentionWork first gets the build queue lock and then tries to get the agent lock.

However hudson.plugins.spotinst.jobs.SpotinstInstancesMonitor first gets the agent lock and then invokes removeNode which tries to get the build queue lock.

So lock ordering is violated which leads to this deadlock situation.

Probably a way to fix that would be to just schedule a separate job to perform "removeNode".