Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21034

Jenkins Startup Deadlock - QueueSorter.installDefaultQueueSorter and Queue.init

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • core, prioritysorter-plugin
    • None
    • Jenkins 1.544 / Jenkins Priority Sorter Plugin 2.2

      Found one Java-level deadlock:
      =============================
      "Handling GET /jenkins/view/Portal/ : RequestHandlerThread[#12]":
        waiting to lock monitor 0x000000001a449d08 (object 0x0000000419d757e8, a hudson.ExtensionList$Lock),
        which is held by "QueueSorter.installDefaultQueueSorter"
      "QueueSorter.installDefaultQueueSorter":
        waiting to lock monitor 0x000000001ae56c48 (object 0x000000041895e0f0, a hudson.model.Queue),
        which is held by "Queue.init"
      "Queue.init":
        waiting to lock monitor 0x000000001a449d08 (object 0x0000000419d757e8, a hudson.ExtensionList$Lock),
        which is held by "QueueSorter.installDefaultQueueSorter"
      

          [JENKINS-21034] Jenkins Startup Deadlock - QueueSorter.installDefaultQueueSorter and Queue.init

          Rob van Oostrum added a comment - - edited

          Ran into this as well. Seems unrelated to the upgrade. It's simply that the queue that's persisted when the Jenkins service shuts down causes a deadlock when Jenkins re-initializes. The quick fix for me was to rename the file (this is on Ubuntu) from queue.xml into something like queue.xml.keep (or you could just remove it altogether).

          I've had this happen both at Jenkins upgrades, and simply restarting Jenkins after plugin updates, since 1.542 or so. Once in this deadlock, downgrading as far as 1.540 didn't do the trick. Ended up upgrading back to 1.544 and removing queue.xml before restarting.

          Rob van Oostrum added a comment - - edited Ran into this as well. Seems unrelated to the upgrade. It's simply that the queue that's persisted when the Jenkins service shuts down causes a deadlock when Jenkins re-initializes. The quick fix for me was to rename the file (this is on Ubuntu) from queue.xml into something like queue.xml.keep (or you could just remove it altogether). I've had this happen both at Jenkins upgrades, and simply restarting Jenkins after plugin updates, since 1.542 or so. Once in this deadlock, downgrading as far as 1.540 didn't do the trick. Ended up upgrading back to 1.544 and removing queue.xml before restarting.

          Does this happen on a "normal" restart or only directly after the upgrade?

          Does this only happen after a upgrade of Priority Sorter or in other situations as well?

          Did you have a huge queue at restart or does this happen with any queue size?

          Magnus Sandberg added a comment - Does this happen on a "normal" restart or only directly after the upgrade? Does this only happen after a upgrade of Priority Sorter or in other situations as well? Did you have a huge queue at restart or does this happen with any queue size?

          @Magnus:

          • I've had this happen after plugin upgrades as well. Yesterday I restarted after upgrading the JDK on Jenkins' server and it deadlocked. Once it deadlocks, restarting again just results in another deadlock.
          • see previous answer
          • I believe my queue had only 2 items in it when I restarted yesterday. It may have been a couple more than that, but not a huge number by any stretch.

          Rob van Oostrum added a comment - @Magnus: I've had this happen after plugin upgrades as well. Yesterday I restarted after upgrading the JDK on Jenkins' server and it deadlocked. Once it deadlocks, restarting again just results in another deadlock. see previous answer I believe my queue had only 2 items in it when I restarted yesterday. It may have been a couple more than that, but not a huge number by any stretch.

          Oleg Nenashev added a comment -

          In my opinion, the code needs some correction even if it works properly in previous Jenkins versions.
          Queue.init() initializes QueueSorter extensions, so it is quite strange to retrieve data from it at this stage...

          I see the following fix:

          • Remove queue handling logic from AdvancedQueueSorter constructor
          • Move it to a separate method an call on the first sortBuildableItems() call
          • It also makes sense to call this method for new mode only (just don't do anything for legacy)

          Oleg Nenashev added a comment - In my opinion, the code needs some correction even if it works properly in previous Jenkins versions. Queue.init() initializes QueueSorter extensions, so it is quite strange to retrieve data from it at this stage... I see the following fix: Remove queue handling logic from AdvancedQueueSorter constructor Move it to a separate method an call on the first sortBuildableItems() call It also makes sense to call this method for new mode only (just don't do anything for legacy)

          Yes this is actually already fixed for the 2.3 where the init is done with Initializer (after.JOB_LOADED)(I though that was already in 2.2 but is is not)

          https://github.com/emsa23/priority-sorter-plugin/commit/a2447a7e371869cb4ada8ed76a7114b3eb165ee8

          @Oleg do you think that fix will do it?

          Magnus Sandberg added a comment - Yes this is actually already fixed for the 2.3 where the init is done with Initializer (after.JOB_LOADED)(I though that was already in 2.2 but is is not) https://github.com/emsa23/priority-sorter-plugin/commit/a2447a7e371869cb4ada8ed76a7114b3eb165ee8 @Oleg do you think that fix will do it?

          Oleg Nenashev added a comment -

          Most probably, it will help with the issue. BTW, any test cases will be appreciated

          Is it guaranteed that initializer's method call will be finished before starting of scheduling?
          If no, it makes sense to somehow synchronize methods.

          Oleg Nenashev added a comment - Most probably, it will help with the issue. BTW, any test cases will be appreciated Is it guaranteed that initializer's method call will be finished before starting of scheduling? If no, it makes sense to somehow synchronize methods.

          I've added some test cases in the 2.3 pull - testing the queue behaviour is not that easy though.

          Regarding the order I'm not sure - one option could be to use before=COMPLETED rather than after=JOB_LOADED but I'm not sure.

          I have restarted Jenkins with items in the queue like a gazillion times and I have never seen this so it is a bit tricky for me to to verify this.

          I reckon that it cannot be wore than it is so I'm considering releasing 2.3 with the current 2.3 pull (https://github.com/jenkinsci/priority-sorter-plugin/pull/10)

          Magnus Sandberg added a comment - I've added some test cases in the 2.3 pull - testing the queue behaviour is not that easy though. Regarding the order I'm not sure - one option could be to use before=COMPLETED rather than after=JOB_LOADED but I'm not sure. I have restarted Jenkins with items in the queue like a gazillion times and I have never seen this so it is a bit tricky for me to to verify this. I reckon that it cannot be wore than it is so I'm considering releasing 2.3 with the current 2.3 pull ( https://github.com/jenkinsci/priority-sorter-plugin/pull/10 )

          Hopefully fixed in 2.3

          The startup sequence is different from 2.2 so if this still occurs please provide new log-file.

          Magnus Sandberg added a comment - Hopefully fixed in 2.3 The startup sequence is different from 2.2 so if this still occurs please provide new log-file.

            emsa23 Magnus Sandberg
            gcummings Geoff Cummings
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: