Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-51600

Build prioritizing plugin gets Jenkins queue stuck even with available executors

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      I have created a custom .hpi Jenkins plugin to prioritize (move to the front of the queue) some jobs that arrive in the queue. The logic is very simple and can be found in CustomSorter.java

      In the plugin I am basically just overriding the default sortBuildableItems() method in QueueSorter class and load it at Jenkins initialization.

      After uploading the plugin everything works as expected (builds that meet specific criteria are pushed to the front of the queue) for usually 10-12 hours but then the build queue gets stuck and no jobs are executed even though all executors are free / available.

       

      Later edit:

      In the Attachments Section you can find the main classes used, the .hpi packaged form and a full log file with a thread dump generated at the end of the logs file.

       

        Attachments

        1. build-queue-prioritizer.hpi
          1.17 MB
        2. CustomLogger.java
          2 kB
        3. CustomSorter.java
          5 kB
        4. logs+threadDump.txt
          2.91 MB
        5. PrioritySorterPlugin.java
          2 kB
        6. QueueListener.java
          1 kB

          Activity

          Hide
          gcimpoies George Cimpoies added a comment -

          Daniel Beck I've updated the attachmets, added all the source code + hpi + logs + thread dump. 

          Show
          gcimpoies George Cimpoies added a comment - Daniel Beck I've updated the attachmets, added all the source code + hpi + logs + thread dump. 
          Hide
          danielbeck Daniel Beck added a comment -
          SEVERE: An exception occurred: java.util.concurrent.ExecutionException
          java.util.concurrent.ExecutionException: java.lang.NullPointerException
          	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
          	at java.util.concurrent.FutureTask.get(FutureTask.java:206)
          	at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter.sortBuildableItems(CustomSorter.java:112)
          	at hudson.model.Queue.maintain(Queue.java:1559)
          	at hudson.model.Queue$1.call(Queue.java:319)
          	at hudson.model.Queue$1.call(Queue.java:316)
          	at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:108)
          	at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:98)
          	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:112)
          	at java.lang.Thread.run(Thread.java:748)
          Caused by: java.lang.NullPointerException
          	at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter.isTriggeredByReleaseCandidate(CustomSorter.java:59)
          	at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter.shouldBePrioritized(CustomSorter.java:44)
          	at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter.compare(CustomSorter.java:71)
          	at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter.access$000(CustomSorter.java:36)
          	at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter$1.lambda$run$0(CustomSorter.java:108)
          	at java.util.TimSort.countRunAndMakeAscending(TimSort.java:360)
          	at java.util.TimSort.sort(TimSort.java:220)
          	at java.util.Arrays.sort(Arrays.java:1512)
          	at java.util.ArrayList.sort(ArrayList.java:1462)
          	at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter$1.run(CustomSorter.java:108)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	... 1 more

          A null pointer exception in your plugin. Probably because there's a build in the queue that's started by another job without build parameters, so getAction is null.

          Show
          danielbeck Daniel Beck added a comment - SEVERE: An exception occurred: java.util.concurrent.ExecutionException java.util.concurrent.ExecutionException: java.lang.NullPointerException at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:206) at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter.sortBuildableItems(CustomSorter.java:112) at hudson.model.Queue.maintain(Queue.java:1559) at hudson.model.Queue$1.call(Queue.java:319) at hudson.model.Queue$1.call(Queue.java:316) at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:108) at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:98) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:112) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter.isTriggeredByReleaseCandidate(CustomSorter.java:59) at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter.shouldBePrioritized(CustomSorter.java:44) at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter.compare(CustomSorter.java:71) at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter.access$000(CustomSorter.java:36) at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter$1.lambda$run$0(CustomSorter.java:108) at java.util.TimSort.countRunAndMakeAscending(TimSort.java:360) at java.util.TimSort.sort(TimSort.java:220) at java.util.Arrays.sort(Arrays.java:1512) at java.util.ArrayList.sort(ArrayList.java:1462) at com.ullink.jenkins.plugins.buildqueueprioritizer.CustomSorter$1.run(CustomSorter.java:108) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more A null pointer exception in your plugin. Probably because there's a build in the queue that's started by another job without build parameters, so getAction is null .
          Hide
          gcimpoies George Cimpoies added a comment - - edited

          Daniel Beck In case such an exception is not handled, could it cause the queue to get stuck?

          Show
          gcimpoies George Cimpoies added a comment - - edited Daniel Beck In case such an exception is not handled, could it cause the queue to get stuck?
          Hide
          danielbeck Daniel Beck added a comment -

          In this case, the queue sorter throws an exception, so Jenkins cannot determine what to build next. It's not a matter of just skipping a list item. There may be a robustness improvement here, but I'm not sure what that would be.

          Show
          danielbeck Daniel Beck added a comment - In this case, the queue sorter throws an exception, so Jenkins cannot determine what to build next. It's not a matter of just skipping a list item. There may be a robustness improvement here, but I'm not sure what that would be.
          Hide
          gcimpoies George Cimpoies added a comment -

          Thanks for all the help!

          Show
          gcimpoies George Cimpoies added a comment - Thanks for all the help!

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            gcimpoies George Cimpoies
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: