Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19623

Jobs throttling causes performance issues in case of enabled security

      • QueueTaskDispatcher locks the queue
      • ThrottleQueueTaskDispatcher invokes Jenkins.Instance().getAllItems() method, which has a security check for the each item
      • Every dispatching attempt causes security check for all jobs in the system
      • Due to big delays, so that dispatching almost blocks the queue and web interfaces

      Workaround - use Hudson.getInstance().getItemMap(), which has not security checks

      "Jenkins cron thread" prio=10 tid=0x0000000005d28000 nid=0x667b runnable [0x0000000041df4000]
      java.lang.Thread.State: RUNNABLE
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$2.perform(RoleMap.java:270)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$RoleWalker.walk(RoleMap.java:327)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$RoleWalker.<init>(RoleMap.java:316)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$2.<init>(RoleMap.java:268)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.getMatchingRoles(RoleMap.java:268)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.newMatchingRoleMap(RoleMap.java:224)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:111)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:128)
      at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:123)
      at hudson.model.Job.getACL(Job.java:1324)
      at hudson.model.AbstractItem.hasPermission(AbstractItem.java:448)
      at jenkins.model.Jenkins.getAllItems(Jenkins.java:1397)
      at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.getCategoryProjects(ThrottleQueueTaskDispatcher.java:211)
      at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:121)
      at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canTake(ThrottleQueueTaskDispatcher.java:36)
      at hudson.model.queue.QueueTaskDispatcher.canTake(QueueTaskDispatcher.java:101)
      at hudson.model.Queue$JobOffer.canTake(Queue.java:254)
      at hudson.model.Queue.maintain(Queue.java:1032)

      • locked <0x00000006805417d8> (a hudson.model.Queue)
        at hudson.model.Queue$MaintainTask.doRun(Queue.java:1762)
        at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)

          [JENKINS-19623] Jobs throttling causes performance issues in case of enabled security

          Oleg Nenashev added a comment -

          Upd. Issue affects categories only

          Oleg Nenashev added a comment - Upd. Issue affects categories only

          Marco Miller added a comment -

          Hi Oleg; we're interested in fixing this issue too (Ericsson).
          How is it going from your end, as we speak?
          Thx+ for letting me know! =)

          Marco Miller added a comment - Hi Oleg; we're interested in fixing this issue too (Ericsson). How is it going from your end, as we speak? Thx+ for letting me know! =)

          Oleg Nenashev added a comment -

          Hello Marco,

          Pull request is waiting for review (it has been created about month ago).
          https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/6
          Seems I need to send an another message to Andrew.

          The fix successfully runs on my installations, so we just need to merge it.

          Best regards,
          Oleg Nenashev

          Oleg Nenashev added a comment - Hello Marco, Pull request is waiting for review (it has been created about month ago). https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/6 Seems I need to send an another message to Andrew. The fix successfully runs on my installations, so we just need to merge it. Best regards, Oleg Nenashev

          Jesse Glick added a comment -

          Working on a different fix.

          Jesse Glick added a comment - Working on a different fix.

          Jesse Glick added a comment -

          https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/10 offered as an alternative. (I have no installation to check it on, so relying on automated tests.)

          Jesse Glick added a comment - https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/10 offered as an alternative. (I have no installation to check it on, so relying on automated tests.)

          Code changed in jenkins
          User: Jesse Glick
          Path:
          src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java
          src/test/java/hudson/plugins/throttleconcurrents/ThrottleJobPropertyTest.java
          http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/9fc87ab447a60552129b4e1580e188b8a7df4aa7
          Log:
          [FIXED JENKINS-19623] Avoid getAllItems during getCategoryProjects as it checks permissions and so can be expensive, whereas QueueTaskDispatcher.canTake must be fast.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java src/test/java/hudson/plugins/throttleconcurrents/ThrottleJobPropertyTest.java http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/9fc87ab447a60552129b4e1580e188b8a7df4aa7 Log: [FIXED JENKINS-19623] Avoid getAllItems during getCategoryProjects as it checks permissions and so can be expensive, whereas QueueTaskDispatcher.canTake must be fast.

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java
          src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java
          src/test/java/hudson/plugins/throttleconcurrents/ThrottleJobPropertyTest.java
          http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/18c7b4f136eed46bbeb07f9661dae9075921c43c
          Log:
          Merge pull request #10 from jglick/slow-getCategoryProjects-JENKINS-19623

          [FIXED JENKINS-19623] getCategoryProjects without permission checks

          Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/1bc16cd4c9e4...18c7b4f136ee

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java src/test/java/hudson/plugins/throttleconcurrents/ThrottleJobPropertyTest.java http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/18c7b4f136eed46bbeb07f9661dae9075921c43c Log: Merge pull request #10 from jglick/slow-getCategoryProjects- JENKINS-19623 [FIXED JENKINS-19623] getCategoryProjects without permission checks Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/1bc16cd4c9e4...18c7b4f136ee

            jglick Jesse Glick
            oleg_nenashev Oleg Nenashev
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: