Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-18377

Improve the performance when listing many jobs (GSoC 2019, coding phase 2)

    • RoleStrategy-Performance

      This affects a jenkins installation with 750+ jobs.

      When loading the "Overview" to list all jobs, the loading time is over 60 seconds.

      With the role-strategy plugin disabled, the loading time goes down to 5 seconds.

          [JENKINS-18377] Improve the performance when listing many jobs (GSoC 2019, coding phase 2)

          Oliver Gondža added a comment - Related PR submitted https://github.com/jenkinsci/role-strategy-plugin/pull/28/

          Code changed in jenkins
          User: Oliver Gondža
          Path:
          src/main/java/com/michelin/cio/hudson/plugins/rolestrategy/Role.java
          http://jenkins-ci.org/commit/role-strategy-plugin/161c0358179a4c845d688b3e42db09f8aa91739d
          Log:
          JENKINS-18377 Cache Role#hashCode to speed up RoleMap#getRolesHavingPermission

          I have experienced severe slowdown caused by following stacktrace:
          at java.util.AbstractSet.hashCode(AbstractSet.java:126)
          at com.michelin.cio.hudson.plugins.rolestrategy.Role.hashCode(Role.java:149)
          at java.util.HashMap.hash(HashMap.java:338)
          at java.util.HashMap.put(HashMap.java:611)
          at java.util.HashSet.add(HashSet.java:219)
          at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$1.perform(RoleMap.java:310)
          at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$RoleWalker.walk(RoleMap.java:387)
          at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$RoleWalker.<init>(RoleMap.java:376)
          at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$1.<init>(RoleMap.java:307)
          at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.getRolesHavingPermission(RoleMap.java:307)
          at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.hasPermission(RoleMap.java:107)
          at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.access$000(RoleMap.java:75)
          at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$AclImpl.hasPermission(RoleMap.java:362)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: src/main/java/com/michelin/cio/hudson/plugins/rolestrategy/Role.java http://jenkins-ci.org/commit/role-strategy-plugin/161c0358179a4c845d688b3e42db09f8aa91739d Log: JENKINS-18377 Cache Role#hashCode to speed up RoleMap#getRolesHavingPermission I have experienced severe slowdown caused by following stacktrace: at java.util.AbstractSet.hashCode(AbstractSet.java:126) at com.michelin.cio.hudson.plugins.rolestrategy.Role.hashCode(Role.java:149) at java.util.HashMap.hash(HashMap.java:338) at java.util.HashMap.put(HashMap.java:611) at java.util.HashSet.add(HashSet.java:219) at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$1.perform(RoleMap.java:310) at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$RoleWalker.walk(RoleMap.java:387) at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$RoleWalker.<init>(RoleMap.java:376) at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$1.<init>(RoleMap.java:307) at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.getRolesHavingPermission(RoleMap.java:307) at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.hasPermission(RoleMap.java:107) at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.access$000(RoleMap.java:75) at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$AclImpl.hasPermission(RoleMap.java:362)

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          src/main/java/com/michelin/cio/hudson/plugins/rolestrategy/Role.java
          http://jenkins-ci.org/commit/role-strategy-plugin/165538fc1e85d9ff3bae99ab85035f34a4039440
          Log:
          Merge pull request #28 from olivergondza/optimize-RoleMap-getRolesHavingPermission

          JENKINS-18377 Cache Role#hashCode to speed up RoleMap#getRolesHavingPermission

          Compare: https://github.com/jenkinsci/role-strategy-plugin/compare/8dbec9ea93d1...165538fc1e85

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/com/michelin/cio/hudson/plugins/rolestrategy/Role.java http://jenkins-ci.org/commit/role-strategy-plugin/165538fc1e85d9ff3bae99ab85035f34a4039440 Log: Merge pull request #28 from olivergondza/optimize-RoleMap-getRolesHavingPermission JENKINS-18377 Cache Role#hashCode to speed up RoleMap#getRolesHavingPermission Compare: https://github.com/jenkinsci/role-strategy-plugin/compare/8dbec9ea93d1...165538fc1e85

          Oleg Nenashev added a comment -

          Unassigning the issue for now. We have added two Role Strategy plugin project ideas to GSoC 2019: https://jenkins.io/projects/gsoc/2019/project-ideas/. If somebody is interested in co-mentoring the ideas (including these tickets), please let us know

          Oleg Nenashev added a comment - Unassigning the issue for now. We have added two Role Strategy plugin project ideas to GSoC 2019: https://jenkins.io/projects/gsoc/2019/project-ideas/ . If somebody is interested in co-mentoring the ideas (including these tickets), please let us know

          Oleg Nenashev added a comment -

          I have converted this issue to EPIC, because there are multiple low-hanging fruits left to be done

          Oleg Nenashev added a comment - I have converted this issue to EPIC, because there are multiple low-hanging fruits left to be done

          Oleg Nenashev added a comment -

          abhyudaya is currently working on this EPIC as a part of hist GSoC project. https://jenkins.io/projects/gsoc/2019/role-strategy-performance/ .Some stories have been already addressed in the master branch.

          abhyudaya could you please take a look at the EPIc and update tasks or add missing ones?

          Oleg Nenashev added a comment - abhyudaya is currently working on this EPIC as a part of hist GSoC project.  https://jenkins.io/projects/gsoc/2019/role-strategy-performance/  .Some stories have been already addressed in the master branch. abhyudaya could you please take a look at the EPIc and update tasks or add missing ones?

          runze xia added a comment -

          I thought about our authentication model. First we need to get the acl (traversing all the roles), and in the second step to determine whether the role in the acl has the appropriate permissions.
          Can we simplify the acquisition of acl content, we construct the roleMap in the second step, so we can reduce the traversal of some roles.

          This idea may require redesigning the ACL object. The way the ACL is cached may also change.

          runze xia added a comment - I thought about our authentication model. First we need to get the acl (traversing all the roles), and in the second step to determine whether the role in the acl has the appropriate permissions. Can we simplify the acquisition of acl content, we construct the roleMap in the second step, so we can reduce the traversal of some roles. This idea may require redesigning the ACL object. The way the ACL is cached may also change.

          runzexia I have just committed https://github.com/jenkinsci/role-strategy-plugin/pull/89/commits/a02ea97b0bd512da1973f502c6ab3faca69845d8 for the folder based authorization. This changes how global roles work. Is it something like what you're looking for?

          Abhyudaya Sharma added a comment - runzexia  I have just committed  https://github.com/jenkinsci/role-strategy-plugin/pull/89/commits/a02ea97b0bd512da1973f502c6ab3faca69845d8  for the folder based authorization. This changes how global roles work. Is it something like what you're looking for?

          runze xia added a comment -

          abhyudaya   Yes, it is very similar to the approach I mentioned. If we operate in havePermission func, can we reduce some memory consumption? And we might be very simple to break the loop?

          runze xia added a comment - abhyudaya    Yes, it is very similar to the approach I mentioned. If we operate in havePermission func, can we reduce some memory consumption? And we might be very simple to break the loop?

          Jesse Glick added a comment -

          When loading the "Overview" to list all jobs, the loading time is over 60 seconds.

          Certainly it is good to optimize particular bottlenecks as we find them, but in tandem I would suggest changing the rendering of the dashboard to be progressive so that server-side bottlenecks do not completely prevent display nor block the user from initiating unrelated actions (like clicking on the sidebar). JENKINS-25075 suggests doing this for individual *ListView/column.jelly cells, though you could just as easily render whole rows lazily, ameliorating a bunch of performance problems at once.

          Jesse Glick added a comment - When loading the "Overview" to list all jobs, the loading time is over 60 seconds. Certainly it is good to optimize particular bottlenecks as we find them, but in tandem I would suggest changing the rendering of the dashboard to be progressive so that server-side bottlenecks do not completely prevent display nor block the user from initiating unrelated actions (like clicking on the sidebar). JENKINS-25075 suggests doing this for individual *ListView/column.jelly cells, though you could just as easily render whole rows lazily, ameliorating a bunch of performance problems at once.

            abhyudaya Abhyudaya Sharma
            croesus Croesus Kall
            Votes:
            14 Vote for this issue
            Watchers:
            26 Start watching this issue

              Created:
              Updated: