many listeners are called whilst locks are held which leads to deadlocks (see JENKINS-30057 for one example).
A listener may reasonably want to perform some action based on this which inevitably requires some call back to Jenkins which will require a different lock. However the listener is not at this point in control of what locks are held when it is called so it needs to jump through hoops in order to do things correctly. Whilst looking at listeners I have found zero instances of Listeners that do jump through the hoops - and it would seem like Jenkins should actually be handling this on behalf of the listener.
For example Nodes may be provisioned with the Queue lock held - and if some jenkins configuration needs to be modified will call locking order of Queue -> Jenkins. Whilst someone saving Jenkins Clouds could lock Jenkins -> Queue.
Whilst we can chase our tails unpicking these deadlocks individually I have seen 3 of these recently and it would be much better to ensure that the callbacks happened without any locks held.
- is related to
-
JENKINS-29936 deadlock between OldDataMonitor and AuthorizationStrategy.
-
- Resolved
-
-
-
- Resolved
-
- links to
[JENKINS-30060] Listeners are called back whilst holding locks which is prone to deadlocks
| Link |
New:
This issue is related to |
| Link |
New:
This issue is related to |
| Workflow | Original: JNJira [ 165142 ] | New: JNJira + In-Review [ 181853 ] |
| Remote Link | New: This issue links to "CloudBees Internal OSS-193 (Web Link)" [ 19310 ] |
Agreed. While it is generally a poor idea for a listener implementation to do anything that might acquire a lock, in practice this is hard to do; and using Timer.get to run the body asynchronously just makes for serious headaches in functional tests. The best option, when possible, is to call listeners with no locks held.