Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52159

Jenkins Queue DoS when QueueItemSorter extension throws runtime Exception

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • core
    • Jenkins 2.130

      We just upgraded to 2.129. Our jenkins is configured to start nodes as needed; release nodes as upon idle; and run a job on master which turns nodes off if they aren't in use.

      We had the accelerated-build-now-plugin (version 1.0.1) plugin enabled.

      When our normal or pipeline jobs start, they try to pick any node other than master. Jenkins starts the node, it appears in the list of available nodes, but Jenkins doesn't manage to start a build on the node:

      SEVERE: Timer task hudson.model.Queue$MaintainTask@2041414e failed
      java.lang.ClassCastException: org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask cannot be cast to hudson.model.AbstractProject
              at org.terracotta.jenkins.plugins.acceleratedbuildnow.AcceleratedBuildNowComparator.compare(AcceleratedBuildNowComparator.java:21)
              at org.terracotta.jenkins.plugins.acceleratedbuildnow.AcceleratedBuildNowComparator.compare(AcceleratedBuildNowComparator.java:11)
              at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
              at java.util.TimSort.sort(TimSort.java:220)
              at java.util.Arrays.sort(Arrays.java:1512)
              at java.util.ArrayList.sort(ArrayList.java:1462)
              at java.util.Collections.sort(Collections.java:175)
              at org.terracotta.jenkins.plugins.acceleratedbuildnow.AcceleratedBuildNowSorter.sortBuildableItems(AcceleratedBuildNowSorter.java:30)
              at hudson.model.Queue.maintain(Queue.java:1571)
              at hudson.model.Queue$MaintainTask.doRun(Queue.java:2862)
              at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
              at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748) 

      Instead, the jobs sit waiting, and eventually the computers being idle are released by jenkins and then reaped by the job on master.

      Disabling this plugin results in our system working as it used to work.

          [JENKINS-52159] Jenkins Queue DoS when QueueItemSorter extension throws runtime Exception

          Josh Soref added a comment -

          I think that a try/catch block here:

          https://github.com/jenkinsci/jenkins/blob/d4737b2b60438ce0e3cb7a8a6f26675f7c880914/core/src/main/java/hudson/model/Queue.java#L1571

          would protect our jobs from getting lost, but I'm not sure if people would accept that.

          Note, per: https://github.com/Terracotta-OSS/accelerated-build-now-plugin/issues/3#issuecomment-374920842, this plugin appears to be mostly unmaintained.

          Josh Soref added a comment - I think that a try/catch block here: https://github.com/jenkinsci/jenkins/blob/d4737b2b60438ce0e3cb7a8a6f26675f7c880914/core/src/main/java/hudson/model/Queue.java#L1571 would protect our jobs from getting lost, but I'm not sure if people would accept that. Note, per:  https://github.com/Terracotta-OSS/accelerated-build-now-plugin/issues/3#issuecomment-374920842 , this plugin appears to be mostly unmaintained.

          Oleg Nenashev added a comment -

          I'd guess accelerated-build-now-plugin is just incompatible with Jenkins Pipeline.
          Whatever was the configuration before the upgrade, this code is just a bug: https://github.com/jenkinsci/accelerated-build-now-plugin/blob/master/src/main/java/org/terracotta/jenkins/plugins/acceleratedbuildnow/AcceleratedBuildNowComparator.java#L19-L21 . Unchecked casts.

          The plugin also has releases being done outside Jenkins org. It may confuse contributors, but URL in POM is correct.

          danielbeck anthonydahanne maybe should we just remove the fork in Jenkins GitHub

          Oleg Nenashev added a comment - I'd guess accelerated-build-now-plugin is just incompatible with Jenkins Pipeline. Whatever was the configuration before the upgrade, this code is just a bug: https://github.com/jenkinsci/accelerated-build-now-plugin/blob/master/src/main/java/org/terracotta/jenkins/plugins/acceleratedbuildnow/AcceleratedBuildNowComparator.java#L19-L21 . Unchecked casts. The plugin also has releases being done outside Jenkins org. It may confuse contributors, but URL in POM is correct. danielbeck anthonydahanne maybe should we just remove the fork in Jenkins GitHub

          Josh Soref added a comment -

          It's possible that someone installed this plugin last week and we only just now hit the rough edge. Anyway. We can split this into two bugs if necessary. (And fix the priority, I agree it doesn't seem like a regression it just apparently triggered coincidentally.)

          Is it possible that the sorting hook only triggers w/ jenkins launch whereas the rest of the plugin can be used w/ an install w/o restart?

          Josh Soref added a comment - It's possible that someone installed this plugin last week and we only just now hit the rough edge. Anyway. We can split this into two bugs if necessary. (And fix the priority, I agree it doesn't seem like a regression it just apparently triggered coincidentally.) Is it possible that the sorting hook only triggers w/ jenkins launch whereas the rest of the plugin can be used w/ an install w/o restart?

          Oleg Nenashev added a comment -

          jsoref I kindly ask to create a separate issue for the Jenkins core robustness.
          Otherwise people may be confused when we release the core and maybe backport the fix.
          With all patches on the core side, this issue will be still there

          Oleg Nenashev added a comment - jsoref I kindly ask to create a separate issue for the Jenkins core robustness. Otherwise people may be confused when we release the core and maybe backport the fix. With all patches on the core side, this issue will be still there

          Oleg Nenashev added a comment -

          Detached the plugin bug to JENKINS-52353, because the issue has been already changelogged

          Oleg Nenashev added a comment - Detached the plugin bug to  JENKINS-52353 , because the issue has been already changelogged

          hello, sorry I have not been pro active with those issues.

          This plugin has been un maintained for a while.

          If I can help making it "officially" not maintained, please let me know. Or if I need to first ask the community if there's someone interested in maintaining it.

          Anthony Dahanne added a comment - hello, sorry I have not been pro active with those issues. This plugin has been un maintained for a while. If I can help making it "officially" not maintained, please let me know. Or if I need to first ask the community if there's someone interested in maintaining it.

          Oleg Nenashev added a comment -

          anthonydahanne no special actions needed, you confirmation is enough.

          I have marked the plugin for adoption. Thanks!

          Oleg Nenashev added a comment - anthonydahanne no special actions needed, you confirmation is enough. I have marked the plugin for adoption. Thanks!

          yes I confirm it's fine ! Adoption is a good idea, I can see you updated the page : https://wiki.jenkins.io/display/JENKINS/Accelerated+Build+Now+Plugin

          Thanks oleg_nenashev !

          Anthony Dahanne added a comment - yes I confirm it's fine ! Adoption is a good idea, I can see you updated the page : https://wiki.jenkins.io/display/JENKINS/Accelerated+Build+Now+Plugin Thanks oleg_nenashev !

            anthonydahanne Anthony Dahanne
            jsoref Josh Soref
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: