Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27650

Page loads slow with hundreds of throttled builds in queue

XMLWordPrintable

      When there are hundreds of throttled builds in the queue, page loads increase by an order of magnitude.

      Steps to reproduce:

      1. Run Jenkins 1.580.2 and latest throttle-concurrent-builds plugin
      2. Create a matrix job with 200 combinations (attached)
      3. In the same job, select "Throttle Concurrent Builds" with a maximum of 7 builds throttled as part of a category called 'semaphore'
      4. Set number of executors on the 'master' queue to 200
      5. Run the job. There should only be 7 builds running due to the throttling

      Page load times will increase by an order of magnitude – I observed 10 seconds from 

      time curl http://localhost:8080/jenkins/ajaxBuildQueue

      If you remove the throttling in the job configuration, the page load times will be under 50 ms.

        1. threaddump.txt
          139 kB
        2. config.xml
          7 kB

          [JENKINS-27650] Page loads slow with hundreds of throttled builds in queue

          Ryan Campbell created issue -

          Oleg Nenashev added a comment -

          Hi Ryan,

          This issue is being caused by the behavior of Queue cache in Jenkins core.
          See the discussion in JENKINS-20046

          It's possible to somehow optimize the performance of the plugin (e.g. by internal caching), but in general I would recommend to improve the behavior of Queue caching for web interfaces.

          Oleg Nenashev added a comment - Hi Ryan, This issue is being caused by the behavior of Queue cache in Jenkins core. See the discussion in JENKINS-20046 It's possible to somehow optimize the performance of the plugin (e.g. by internal caching), but in general I would recommend to improve the behavior of Queue caching for web interfaces.
          Oleg Nenashev made changes -
          Link New: This issue is related to JENKINS-20046 [ JENKINS-20046 ]

          Daniel Beck added a comment -

          Oleg: Would be interesting to see the effect Stephen's change has starting with 1.607.

          http://jenkins-ci.org/changelog
          https://github.com/jenkinsci/jenkins/pull/1596

          Daniel Beck added a comment - Oleg: Would be interesting to see the effect Stephen's change has starting with 1.607. http://jenkins-ci.org/changelog https://github.com/jenkinsci/jenkins/pull/1596

          Oleg Nenashev added a comment -

          Yes, we had a discussion with Ryan on this topic in Skype.

          My opinion:

          • Changes from stephenc will definitely improve the performance
          • Queue::CachedItemList still needs an improvement to provide a better caching
          • I would move the periodic cache refresh to an independent thread with a configurable priority (high by default)

          Oleg Nenashev added a comment - Yes, we had a discussion with Ryan on this topic in Skype. My opinion: Changes from stephenc will definitely improve the performance Queue::CachedItemList still needs an improvement to provide a better caching I would move the periodic cache refresh to an independent thread with a configurable priority (high by default)

          Ryan Campbell added a comment -

          The thread dump seen when reproducing the issue.

          Ryan Campbell added a comment - The thread dump seen when reproducing the issue.
          Ryan Campbell made changes -
          Attachment New: threaddump.txt [ 28908 ]

          Ryan Campbell added a comment -

          Is it possible to do this calculation without holding a lock on the Queue the entire time?

          For instance, could it be done asynchronously in some executor pool? As it stands, the entire process keeps the queue locked because it is called from Queue.maintain(). Until the result is ready, the plugin could return a Blockage reason of "Calculating..." But this is just speculation – I'm casting about for a solution limited to the plugin.

          I realize this can be fixed in core, but for many people on older LTS, this plugin is basically broken for high-end usage.

          Ryan Campbell added a comment - Is it possible to do this calculation without holding a lock on the Queue the entire time? For instance, could it be done asynchronously in some executor pool? As it stands, the entire process keeps the queue locked because it is called from Queue.maintain(). Until the result is ready, the plugin could return a Blockage reason of "Calculating..." But this is just speculation – I'm casting about for a solution limited to the plugin. I realize this can be fixed in core, but for many people on older LTS, this plugin is basically broken for high-end usage.
          Jesse Glick made changes -
          Assignee Original: Oleg Nenashev [ oleg_nenashev ] New: Jesse Glick [ jglick ]

          Jesse Glick added a comment -

          Will see if https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/26 can be improved upon. From thread dumps it is clear that this plugin is to blame for wasting resources. Yes 1.607+ will no longer block the web UI due to problems like this, but the backend will still be excessively loaded.

          Jesse Glick added a comment - Will see if https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/26 can be improved upon. From thread dumps it is clear that this plugin is to blame for wasting resources. Yes 1.607+ will no longer block the web UI due to problems like this, but the backend will still be excessively loaded.

            Unassigned Unassigned
            recampbell Ryan Campbell
            Votes:
            12 Vote for this issue
            Watchers:
            20 Start watching this issue

              Created:
              Updated: