Sometimes our nodes won't be able to start new builds even though there are free slots available.
A workaround for the slaves is to disconnect/connect the slave and it will start to schedule builds again.
I have observed that when this happens for a slave the slave has fewer threads ongoing than an idle slave.
Attaching thread dumps when this happens and after doing an disconnect/connect.
We have seen this issue both on Windows(jlnp) slaves and linux(ssh) slaves as well as on the master node which is running linux.