-
Bug
-
Resolution: Fixed
-
Critical
-
None
-
Jenkins 1.544 / Jenkins Priority Sorter Plugin 2.2
Found one Java-level deadlock: ============================= "Handling GET /jenkins/view/Portal/ : RequestHandlerThread[#12]": waiting to lock monitor 0x000000001a449d08 (object 0x0000000419d757e8, a hudson.ExtensionList$Lock), which is held by "QueueSorter.installDefaultQueueSorter" "QueueSorter.installDefaultQueueSorter": waiting to lock monitor 0x000000001ae56c48 (object 0x000000041895e0f0, a hudson.model.Queue), which is held by "Queue.init" "Queue.init": waiting to lock monitor 0x000000001a449d08 (object 0x0000000419d757e8, a hudson.ExtensionList$Lock), which is held by "QueueSorter.installDefaultQueueSorter"
- is related to
-
-
- Resolved
-
[JENKINS-21034] Jenkins Startup Deadlock - QueueSorter.installDefaultQueueSorter and Queue.init
| Assignee | Original: bklarson [ bklarson ] | New: Magnus Sandberg [ emsa23 ] |
Does this happen on a "normal" restart or only directly after the upgrade?
Does this only happen after a upgrade of Priority Sorter or in other situations as well?
Did you have a huge queue at restart or does this happen with any queue size?
Rob van Oostrum
added a comment - @Magnus:
- I've had this happen after plugin upgrades as well. Yesterday I restarted after upgrading the JDK on Jenkins' server and it deadlocked. Once it deadlocks, restarting again just results in another deadlock.
- see previous answer
- I believe my queue had only 2 items in it when I restarted yesterday. It may have been a couple more than that, but not a huge number by any stretch.
Rob van Oostrum
added a comment - @Magnus:
I've had this happen after plugin upgrades as well. Yesterday I restarted after upgrading the JDK on Jenkins' server and it deadlocked. Once it deadlocks, restarting again just results in another deadlock.
see previous answer
I believe my queue had only 2 items in it when I restarted yesterday. It may have been a couple more than that, but not a huge number by any stretch.
In my opinion, the code needs some correction even if it works properly in previous Jenkins versions.
Queue.init() initializes QueueSorter extensions, so it is quite strange to retrieve data from it at this stage...
I see the following fix:
- Remove queue handling logic from AdvancedQueueSorter constructor
- Move it to a separate method an call on the first sortBuildableItems() call
- It also makes sense to call this method for new mode only (just don't do anything for legacy)
Yes this is actually already fixed for the 2.3 where the init is done with Initializer (after.JOB_LOADED)(I though that was already in 2.2 but is is not)
https://github.com/emsa23/priority-sorter-plugin/commit/a2447a7e371869cb4ada8ed76a7114b3eb165ee8
@Oleg do you think that fix will do it?
Most probably, it will help with the issue. BTW, any test cases will be appreciated 
Is it guaranteed that initializer's method call will be finished before starting of scheduling?
If no, it makes sense to somehow synchronize methods.
I've added some test cases in the 2.3 pull - testing the queue behaviour is not that easy though.
Regarding the order I'm not sure - one option could be to use before=COMPLETED rather than after=JOB_LOADED but I'm not sure.
I have restarted Jenkins with items in the queue like a gazillion times and I have never seen this so it is a bit tricky for me to to verify this.
I reckon that it cannot be wore than it is so I'm considering releasing 2.3 with the current 2.3 pull (https://github.com/jenkinsci/priority-sorter-plugin/pull/10)
| Description |
Original:
{code} Found one Java-level deadlock: ============================= "Handling GET /jenkins/view/Portal/ : RequestHandlerThread[#12]": waiting to lock monitor 0x000000001a449d08 (object 0x0000000419d757e8, a hudson.ExtensionList$Lock), which is held by "QueueSorter.installDefaultQueueSorter" "QueueSorter.installDefaultQueueSorter": waiting to lock monitor 0x000000001ae56c48 (object 0x000000041895e0f0, a hudson.model.Queue), which is held by "Queue.init" "Queue.init": waiting to lock monitor 0x000000001a449d08 (object 0x0000000419d757e8, a hudson.ExtensionList$Lock), which is held by "QueueSorter.installDefaultQueueSorter" {code} |
New:
{code} Found one Java-level deadlock: ============================= "Handling GET /jenkins/view/Portal/ : RequestHandlerThread[#12]": waiting to lock monitor 0x000000001a449d08 (object 0x0000000419d757e8, a hudson.ExtensionList$Lock), which is held by "QueueSorter.installDefaultQueueSorter" "QueueSorter.installDefaultQueueSorter": waiting to lock monitor 0x000000001ae56c48 (object 0x000000041895e0f0, a hudson.model.Queue), which is held by "Queue.init" "Queue.init": waiting to lock monitor 0x000000001a449d08 (object 0x0000000419d757e8, a hudson.ExtensionList$Lock), which is held by "QueueSorter.installDefaultQueueSorter" {code} |
| Environment | Original: Jenkins 1.544 | New: Jenkins 1.544 / Jenkins Priority Sorter Plugin 2.2 |
Ran into this as well. Seems unrelated to the upgrade. It's simply that the queue that's persisted when the Jenkins service shuts down causes a deadlock when Jenkins re-initializes. The quick fix for me was to rename the file (this is on Ubuntu) from queue.xml into something like queue.xml.keep (or you could just remove it altogether).
I've had this happen both at Jenkins upgrades, and simply restarting Jenkins after plugin updates, since 1.542 or so. Once in this deadlock, downgrading as far as 1.540 didn't do the trick. Ended up upgrading back to 1.544 and removing queue.xml before restarting.