Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-51600

Build prioritizing plugin gets Jenkins queue stuck even with available executors

      I have created a custom .hpi Jenkins plugin to prioritize (move to the front of the queue) some jobs that arrive in the queue. The logic is very simple and can be found in CustomSorter.java

      In the plugin I am basically just overriding the default sortBuildableItems() method in QueueSorter class and load it at Jenkins initialization.

      After uploading the plugin everything works as expected (builds that meet specific criteria are pushed to the front of the queue) for usually 10-12 hours but then the build queue gets stuck and no jobs are executed even though all executors are free / available.

       

      Later edit:

      In the Attachments Section you can find the main classes used, the .hpi packaged form and a full log file with a thread dump generated at the end of the logs file.

       

        1. build-queue-prioritizer.hpi
          1.17 MB
        2. CustomLogger.java
          2 kB
        3. CustomSorter.java
          5 kB
        4. logs+threadDump.txt
          2.91 MB
        5. PrioritySorterPlugin.java
          2 kB
        6. QueueListener.java
          1 kB

          [JENKINS-51600] Build prioritizing plugin gets Jenkins queue stuck even with available executors

          Daniel Beck added a comment -

          I feel like I've seen this before. Have you posted this to a Jenkins mailing list as well?

          Daniel Beck added a comment - I feel like I've seen this before. Have you posted this to a Jenkins mailing list as well?

          Yes, to jenkins-devs mailing list. With no luck unfortunately. However, I'm going to come back with a thread dump and more info about the bug.

          George Cimpoies added a comment - Yes, to jenkins-devs mailing list. With no luck unfortunately. However, I'm going to come back with a thread dump and more info about the bug.

          Daniel Beck added a comment -

          Note that there appears to be a bug in the code of isTriggeredByReleaseCandidate, a build can have any number of upstream causes. Unrelated, but still FYI

          Daniel Beck added a comment - Note that there appears to be a bug in the code of isTriggeredByReleaseCandidate , a build can have any number of upstream causes. Unrelated, but still FYI

          Daniel Beck added a comment -

          Thread dump once it stops working would be great. Please ping me by name (@ followed by my name) so I get a notification.

          Daniel Beck added a comment - Thread dump once it stops working would be great. Please ping me by name (@ followed by my name) so I get a notification.

          danielbeck thanks for the help! Talk to you soon!

          George Cimpoies added a comment - danielbeck thanks for the help! Talk to you soon!

          danielbeck I've updated the attachmets, added all the source code + hpi + logs + thread dump. 

          George Cimpoies added a comment - danielbeck I've updated the attachmets, added all the source code + hpi + logs + thread dump. 

          Daniel Beck added a comment -
          SEVERE: An exception occurred: java.util.concurrent.ExecutionException
          java.util.concurrent.ExecutionException: java.lang.NullPointerException
          	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
          	at java.util.concurrent.FutureTask.get(FutureTask.java:206)
          	at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter.sortBuildableItems(CustomSorter.java:112)
          	at hudson.model.Queue.maintain(Queue.java:1559)
          	at hudson.model.Queue$1.call(Queue.java:319)
          	at hudson.model.Queue$1.call(Queue.java:316)
          	at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:108)
          	at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:98)
          	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:112)
          	at java.lang.Thread.run(Thread.java:748)
          Caused by: java.lang.NullPointerException
          	at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter.isTriggeredByReleaseCandidate(CustomSorter.java:59)
          	at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter.shouldBePrioritized(CustomSorter.java:44)
          	at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter.compare(CustomSorter.java:71)
          	at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter.access$000(CustomSorter.java:36)
          	at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter$1.lambda$run$0(CustomSorter.java:108)
          	at java.util.TimSort.countRunAndMakeAscending(TimSort.java:360)
          	at java.util.TimSort.sort(TimSort.java:220)
          	at java.util.Arrays.sort(Arrays.java:1512)
          	at java.util.ArrayList.sort(ArrayList.java:1462)
          	at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter$1.run(CustomSorter.java:108)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	... 1 more

          A null pointer exception in your plugin. Probably because there's a build in the queue that's started by another job without build parameters, so getAction is null.

          Daniel Beck added a comment - SEVERE: An exception occurred: java.util.concurrent.ExecutionException java.util.concurrent.ExecutionException: java.lang.NullPointerException at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:206) at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter.sortBuildableItems(CustomSorter.java:112) at hudson.model.Queue.maintain(Queue.java:1559) at hudson.model.Queue$1.call(Queue.java:319) at hudson.model.Queue$1.call(Queue.java:316) at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:108) at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:98) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:112) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter.isTriggeredByReleaseCandidate(CustomSorter.java:59) at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter.shouldBePrioritized(CustomSorter.java:44) at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter.compare(CustomSorter.java:71) at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter.access$000(CustomSorter.java:36) at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter$1.lambda$run$0(CustomSorter.java:108) at java.util.TimSort.countRunAndMakeAscending(TimSort.java:360) at java.util.TimSort.sort(TimSort.java:220) at java.util.Arrays.sort(Arrays.java:1512) at java.util.ArrayList.sort(ArrayList.java:1462) at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter$1.run(CustomSorter.java:108) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more A null pointer exception in your plugin. Probably because there's a build in the queue that's started by another job without build parameters, so getAction is null .

          George Cimpoies added a comment - - edited

          danielbeck In case such an exception is not handled, could it cause the queue to get stuck?

          George Cimpoies added a comment - - edited danielbeck In case such an exception is not handled, could it cause the queue to get stuck?

          Daniel Beck added a comment -

          In this case, the queue sorter throws an exception, so Jenkins cannot determine what to build next. It's not a matter of just skipping a list item. There may be a robustness improvement here, but I'm not sure what that would be.

          Daniel Beck added a comment - In this case, the queue sorter throws an exception, so Jenkins cannot determine what to build next. It's not a matter of just skipping a list item. There may be a robustness improvement here, but I'm not sure what that would be.

          Thanks for all the help!

          George Cimpoies added a comment - Thanks for all the help!

            Unassigned Unassigned
            gcimpoies George Cimpoies
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: