Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52159

Jenkins Queue DoS when QueueItemSorter extension throws runtime Exception

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • core
    • Jenkins 2.130

      We just upgraded to 2.129. Our jenkins is configured to start nodes as needed; release nodes as upon idle; and run a job on master which turns nodes off if they aren't in use.

      We had the accelerated-build-now-plugin (version 1.0.1) plugin enabled.

      When our normal or pipeline jobs start, they try to pick any node other than master. Jenkins starts the node, it appears in the list of available nodes, but Jenkins doesn't manage to start a build on the node:

      SEVERE: Timer task hudson.model.Queue$MaintainTask@2041414e failed
      java.lang.ClassCastException: org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask cannot be cast to hudson.model.AbstractProject
              at org.terracotta.jenkins.plugins.acceleratedbuildnow.AcceleratedBuildNowComparator.compare(AcceleratedBuildNowComparator.java:21)
              at org.terracotta.jenkins.plugins.acceleratedbuildnow.AcceleratedBuildNowComparator.compare(AcceleratedBuildNowComparator.java:11)
              at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
              at java.util.TimSort.sort(TimSort.java:220)
              at java.util.Arrays.sort(Arrays.java:1512)
              at java.util.ArrayList.sort(ArrayList.java:1462)
              at java.util.Collections.sort(Collections.java:175)
              at org.terracotta.jenkins.plugins.acceleratedbuildnow.AcceleratedBuildNowSorter.sortBuildableItems(AcceleratedBuildNowSorter.java:30)
              at hudson.model.Queue.maintain(Queue.java:1571)
              at hudson.model.Queue$MaintainTask.doRun(Queue.java:2862)
              at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
              at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748) 

      Instead, the jobs sit waiting, and eventually the computers being idle are released by jenkins and then reaped by the job on master.

      Disabling this plugin results in our system working as it used to work.

          [JENKINS-52159] Jenkins Queue DoS when QueueItemSorter extension throws runtime Exception

          Josh Soref created issue -

          Josh Soref added a comment -

          I think that a try/catch block here:

          https://github.com/jenkinsci/jenkins/blob/d4737b2b60438ce0e3cb7a8a6f26675f7c880914/core/src/main/java/hudson/model/Queue.java#L1571

          would protect our jobs from getting lost, but I'm not sure if people would accept that.

          Note, per: https://github.com/Terracotta-OSS/accelerated-build-now-plugin/issues/3#issuecomment-374920842, this plugin appears to be mostly unmaintained.

          Josh Soref added a comment - I think that a try/catch block here: https://github.com/jenkinsci/jenkins/blob/d4737b2b60438ce0e3cb7a8a6f26675f7c880914/core/src/main/java/hudson/model/Queue.java#L1571 would protect our jobs from getting lost, but I'm not sure if people would accept that. Note, per:  https://github.com/Terracotta-OSS/accelerated-build-now-plugin/issues/3#issuecomment-374920842 , this plugin appears to be mostly unmaintained.

          Oleg Nenashev added a comment -

          I'd guess accelerated-build-now-plugin is just incompatible with Jenkins Pipeline.
          Whatever was the configuration before the upgrade, this code is just a bug: https://github.com/jenkinsci/accelerated-build-now-plugin/blob/master/src/main/java/org/terracotta/jenkins/plugins/acceleratedbuildnow/AcceleratedBuildNowComparator.java#L19-L21 . Unchecked casts.

          The plugin also has releases being done outside Jenkins org. It may confuse contributors, but URL in POM is correct.

          danielbeck anthonydahanne maybe should we just remove the fork in Jenkins GitHub

          Oleg Nenashev added a comment - I'd guess accelerated-build-now-plugin is just incompatible with Jenkins Pipeline. Whatever was the configuration before the upgrade, this code is just a bug: https://github.com/jenkinsci/accelerated-build-now-plugin/blob/master/src/main/java/org/terracotta/jenkins/plugins/acceleratedbuildnow/AcceleratedBuildNowComparator.java#L19-L21 . Unchecked casts. The plugin also has releases being done outside Jenkins org. It may confuse contributors, but URL in POM is correct. danielbeck anthonydahanne maybe should we just remove the fork in Jenkins GitHub
          Josh Soref made changes -
          Component/s New: core [ 15593 ]

          Josh Soref added a comment -

          It's possible that someone installed this plugin last week and we only just now hit the rough edge. Anyway. We can split this into two bugs if necessary. (And fix the priority, I agree it doesn't seem like a regression it just apparently triggered coincidentally.)

          Is it possible that the sorting hook only triggers w/ jenkins launch whereas the rest of the plugin can be used w/ an install w/o restart?

          Josh Soref added a comment - It's possible that someone installed this plugin last week and we only just now hit the rough edge. Anyway. We can split this into two bugs if necessary. (And fix the priority, I agree it doesn't seem like a regression it just apparently triggered coincidentally.) Is it possible that the sorting hook only triggers w/ jenkins launch whereas the rest of the plugin can be used w/ an install w/o restart?

          Oleg Nenashev added a comment -

          jsoref I kindly ask to create a separate issue for the Jenkins core robustness.
          Otherwise people may be confused when we release the core and maybe backport the fix.
          With all patches on the core side, this issue will be still there

          Oleg Nenashev added a comment - jsoref I kindly ask to create a separate issue for the Jenkins core robustness. Otherwise people may be confused when we release the core and maybe backport the fix. With all patches on the core side, this issue will be still there

          Oleg Nenashev added a comment -

          Detached the plugin bug to JENKINS-52353, because the issue has been already changelogged

          Oleg Nenashev added a comment - Detached the plugin bug to  JENKINS-52353 , because the issue has been already changelogged
          Oleg Nenashev made changes -
          Link New: This issue relates to JENKINS-52353 [ JENKINS-52353 ]
          Oleg Nenashev made changes -
          Component/s Original: accelerated-build-now-plugin [ 18026 ]
          Oleg Nenashev made changes -
          Summary Original: Jenkins ver. 2.129 is incompatible with accelerated-build-now-plugin 1.0.1 New: Jenkins Queue DoS when QueueItemSorter extension throws runtime Exception

            anthonydahanne Anthony Dahanne
            jsoref Josh Soref
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: