• Type: Bug
• Resolution: Unresolved
• Priority: Major
• Component/s: cloudbees-folder-plugin, core, role-strategy-plugin
• Labels:

  None
• Environment:

  Hide

  Ubuntu 14.04.5 LTS
  OpenJDK 8u111
  Jenkins v2.32.1
  Folders Plugin v5.16
  Multi-Branch Project Plugin v0.5.1
  Role-based Authorization Strategy v2.3.2
  
  Intel Xeon E5-2630
  24 GB RAM
  500 GB SSD x 5 (RAID-5)
  1897 jobs in total on the server – (1799 Matrix jobs)
  Role Strategy Project Role: 119
  Total Users: 464 users
  
  Client tested on Ubuntu 16.04 (64-bit)
  Chromium-browser 55.0.2883.87
  Firefox 50.1.0

  Show

  Ubuntu 14.04.5 LTS OpenJDK 8u111 Jenkins v2.32.1 Folders Plugin v5.16 Multi-Branch Project Plugin v0.5.1 Role-based Authorization Strategy v2.3.2 Intel Xeon E5-2630 24 GB RAM 500 GB SSD x 5 (RAID-5) 1897 jobs in total on the server – (1799 Matrix jobs) Role Strategy Project Role: 119 Total Users: 464 users Client tested on Ubuntu 16.04 (64-bit) Chromium-browser 55.0.2883.87 Firefox 50.1.0

1. jenkins_open_AC_folder_Current_Thread.log

   172 kB

   2017-02-25 03:23

2. jenkins_open_AC_folder.log.tgz

   2.01 MB

   2017-02-25 03:23

3. jenkins-open-folder.tar.gz

   9.54 MB

   2017-02-24 04:01

4. jenkins-startup.tar.gz

   4.39 MB

   2017-02-24 04:01

5. jenkins-startup-repeat05-0631.log

   2.74 MB

   2017-02-23 03:36

6. jenkins-startup-repeat04-0630.log

   2.74 MB

   2017-02-23 03:36

7. jenkins-startup-repeat03-0629.log

   2.74 MB

   2017-02-23 03:36

8. jenkins-startup-repeat02-0628.log

   2.74 MB

   2017-02-23 03:36

9. jenkins-startup-repeat01-0627.log

   5.47 MB

   2017-02-23 03:36

10. jenkins-startup.log.gz

    487 kB

    2017-02-23 03:36

[JENKINS-41722] Job folder loading performance scalability

Rick Liu created issue - 2017-02-04 02:55

Rick Liu made changes - 2017-02-04 02:57

Description

Original: I have a tree-like structure job layout as below: || "Projects" Cloudbees folder | contains *34* subproject folder | || || subproject: "Project-A" | contains *6* branch folders ( *5* multibranch job folder and *1* cloudbees folder ) | || || || multibranch-job1 | contains 295 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job2 | contains 317 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job3 | contains 189 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job4 | contains 15 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job5 | contains 5 matrix jobs (75 x 2 matrix combination) | || || || Cloudbees folder | contains 4 matrix jobs (75 x 2 matrix combination) | || || subproject: Project-B | .... | || || || multibranch-job1 | ... | || || || ... | ... | I found somehow no matter what kind of authorization strategy I choose, it's always taking longer time to open "{{*Project-A*}}" cloudbees folder (contains *6* items) than "{{*Projects*}}" folder (contains *34* items). Below is my benchmark: || || Chrome || Chrome || Firefox || Firefox || || Authorization Method || Open Projects Folder || Open Project-A Folder || Open Projects Folder || Open Project-A Folder || || Anyone can do anything | 2.10 sec | 8.43 sec | 5.99 sec | 17.05 sec | || Logged-in users can do anything | 3.97 sec | 9.94 sec | 7.97 sec | 20.06 sec | || Initial default Role-based configuration - admin role | 8.22 sec | 14.54 sec | 16.47 sec | 28.55 sec | || Role-based configuration - admin role with 119 Project roles | 18.81 sec | 31.95 sec | 37.7 sec | 65 |

New: I have a tree-like structure job layout as below: || "Projects" Cloudbees folder | contains *34* subproject folder | || || subproject: "Project-A" Cloudbees folder | contains *6* branch folders ( *5* multibranch job folder and *1* cloudbees folder ) | || || || multibranch-job1 | contains 295 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job2 | contains 317 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job3 | contains 189 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job4 | contains 15 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job5 | contains 5 matrix jobs (75 x 2 matrix combination) | || || || Cloudbees folder | contains 4 matrix jobs (75 x 2 matrix combination) | || || subproject: Project-B Cloudbees folder | .... | || || || multibranch-job1 | ... | || || || ... | ... | || || ... || ... | ... | I found somehow no matter what kind of authorization strategy I choose, it's always taking longer time to open "{{*Project-A*}}" cloudbees folder (contains *6* items) than "{{*Projects*}}" folder (contains *34* items). Below is my benchmark: || || Chrome || Chrome || Firefox || Firefox || || Authorization Method || Open Projects Folder || Open Project-A Folder || Open Projects Folder || Open Project-A Folder || || Anyone can do anything | 2.10 sec | 8.43 sec | 5.99 sec | 17.05 sec | || Logged-in users can do anything | 3.97 sec | 9.94 sec | 7.97 sec | 20.06 sec | || Initial default Role-based configuration - admin role | 8.22 sec | 14.54 sec | 16.47 sec | 28.55 sec | || Role-based configuration - admin role with 119 Project roles | 18.81 sec | 31.95 sec | 37.7 sec | 65 |

Rick Liu made changes - 2017-02-04 02:58

Summary

Original: Job folder loading performance difference in tree-like job structure

New: Job folder loading performance is worse when folder contains fewer items

Rick Liu made changes - 2017-02-04 03:01

Description

Original: I have a tree-like structure job layout as below: || "Projects" Cloudbees folder | contains *34* subproject folder | || || subproject: "Project-A" Cloudbees folder | contains *6* branch folders ( *5* multibranch job folder and *1* cloudbees folder ) | || || || multibranch-job1 | contains 295 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job2 | contains 317 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job3 | contains 189 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job4 | contains 15 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job5 | contains 5 matrix jobs (75 x 2 matrix combination) | || || || Cloudbees folder | contains 4 matrix jobs (75 x 2 matrix combination) | || || subproject: Project-B Cloudbees folder | .... | || || || multibranch-job1 | ... | || || || ... | ... | || || ... || ... | ... | I found somehow no matter what kind of authorization strategy I choose, it's always taking longer time to open "{{*Project-A*}}" cloudbees folder (contains *6* items) than "{{*Projects*}}" folder (contains *34* items). Below is my benchmark: || || Chrome || Chrome || Firefox || Firefox || || Authorization Method || Open Projects Folder || Open Project-A Folder || Open Projects Folder || Open Project-A Folder || || Anyone can do anything | 2.10 sec | 8.43 sec | 5.99 sec | 17.05 sec | || Logged-in users can do anything | 3.97 sec | 9.94 sec | 7.97 sec | 20.06 sec | || Initial default Role-based configuration - admin role | 8.22 sec | 14.54 sec | 16.47 sec | 28.55 sec | || Role-based configuration - admin role with 119 Project roles | 18.81 sec | 31.95 sec | 37.7 sec | 65 |

New: I have a tree-like structure job layout as below: || "Projects" Cloudbees folder | contains *34* subproject folder | || || subproject: "Project-A" Cloudbees folder | contains *6* branch folders ( *5* multibranch job folder and *1* cloudbees folder ) | || || || multibranch-job1 | contains 295 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job2 | contains 317 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job3 | contains 189 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job4 | contains 15 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job5 | contains 5 matrix jobs (75 x 2 matrix combination) | || || || Cloudbees folder | contains 4 matrix jobs (75 x 2 matrix combination) | || || subproject: Project-B Cloudbees folder | .... | || || || multibranch-job1 | ... | || || || ... | ... | || || ... || ... | ... | I found somehow no matter what kind of authorization strategy I choose, it's always taking longer time to open "{{*Project-A*}}" cloudbees folder (contains *6* items) than "{{*Projects*}}" folder (contains *34* items). I'm expecting it should be faster to open "{{*Project-A*}}" folder because it needs to process permission and load only *6* items, and slower to open "{{*Projects*}}" folder because it needs to process permissions and load for *34* items. Below is my benchmark: || || Chrome || Chrome || Firefox || Firefox || || Authorization Method || Open Projects Folder || Open Project-A Folder || Open Projects Folder || Open Project-A Folder || || Anyone can do anything | 2.10 sec | 8.43 sec | 5.99 sec | 17.05 sec | || Logged-in users can do anything | 3.97 sec | 9.94 sec | 7.97 sec | 20.06 sec | || Initial default Role-based configuration - admin role | 8.22 sec | 14.54 sec | 16.47 sec | 28.55 sec | || Role-based configuration - admin role with 119 Project roles | 18.81 sec | 31.95 sec | 37.7 sec | 65 |

Rick Liu made changes - 2017-02-04 03:01

Description

Original: I have a tree-like structure job layout as below: || "Projects" Cloudbees folder | contains *34* subproject folder | || || subproject: "Project-A" Cloudbees folder | contains *6* branch folders ( *5* multibranch job folder and *1* cloudbees folder ) | || || || multibranch-job1 | contains 295 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job2 | contains 317 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job3 | contains 189 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job4 | contains 15 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job5 | contains 5 matrix jobs (75 x 2 matrix combination) | || || || Cloudbees folder | contains 4 matrix jobs (75 x 2 matrix combination) | || || subproject: Project-B Cloudbees folder | .... | || || || multibranch-job1 | ... | || || || ... | ... | || || ... || ... | ... | I found somehow no matter what kind of authorization strategy I choose, it's always taking longer time to open "{{*Project-A*}}" cloudbees folder (contains *6* items) than "{{*Projects*}}" folder (contains *34* items). I'm expecting it should be faster to open "{{*Project-A*}}" folder because it needs to process permission and load only *6* items, and slower to open "{{*Projects*}}" folder because it needs to process permissions and load for *34* items. Below is my benchmark: || || Chrome || Chrome || Firefox || Firefox || || Authorization Method || Open Projects Folder || Open Project-A Folder || Open Projects Folder || Open Project-A Folder || || Anyone can do anything | 2.10 sec | 8.43 sec | 5.99 sec | 17.05 sec | || Logged-in users can do anything | 3.97 sec | 9.94 sec | 7.97 sec | 20.06 sec | || Initial default Role-based configuration - admin role | 8.22 sec | 14.54 sec | 16.47 sec | 28.55 sec | || Role-based configuration - admin role with 119 Project roles | 18.81 sec | 31.95 sec | 37.7 sec | 65 |

New: I have a tree-like structure job layout as below: || "Projects" Cloudbees folder | contains *34* subproject folder | || || subproject: "Project-A" Cloudbees folder | contains *6* branch folders ( *5* multibranch job folder and *1* cloudbees folder ) | || || || multibranch-job1 | contains 295 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job2 | contains 317 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job3 | contains 189 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job4 | contains 15 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job5 | contains 5 matrix jobs (75 x 2 matrix combination) | || || || Cloudbees folder | contains 4 matrix jobs (75 x 2 matrix combination) | || || subproject: Project-B Cloudbees folder | .... | || || || multibranch-job1 | ... | || || || ... | ... | || || ... || ... | ... | I found somehow no matter what kind of authorization strategy I choose, it's always taking longer time to open "{{*Project-A*}}" cloudbees folder (contains *6* items) than "{{*Projects*}}" folder (contains *34* items). Below is my benchmark: || || Chrome || Chrome || Firefox || Firefox || || Authorization Method || Open Projects Folder || Open Project-A Folder || Open Projects Folder || Open Project-A Folder || || Anyone can do anything | 2.10 sec | 8.43 sec | 5.99 sec | 17.05 sec | || Logged-in users can do anything | 3.97 sec | 9.94 sec | 7.97 sec | 20.06 sec | || Initial default Role-based configuration - admin role | 8.22 sec | 14.54 sec | 16.47 sec | 28.55 sec | || Role-based configuration - admin role with 119 Project roles | 18.81 sec | 31.95 sec | 37.7 sec | 65 | I'm expecting it should be faster to open "{{*Project-A*}}" folder because it needs to process permission and load only *6* items, and slower to open "{{*Projects*}}" folder because it needs to process permissions and load for *34* items.

Rick Liu made changes - 2017-02-04 03:02

Description

Original: I have a tree-like structure job layout as below: || "Projects" Cloudbees folder | contains *34* subproject folder | || || subproject: "Project-A" Cloudbees folder | contains *6* branch folders ( *5* multibranch job folder and *1* cloudbees folder ) | || || || multibranch-job1 | contains 295 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job2 | contains 317 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job3 | contains 189 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job4 | contains 15 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job5 | contains 5 matrix jobs (75 x 2 matrix combination) | || || || Cloudbees folder | contains 4 matrix jobs (75 x 2 matrix combination) | || || subproject: Project-B Cloudbees folder | .... | || || || multibranch-job1 | ... | || || || ... | ... | || || ... || ... | ... | I found somehow no matter what kind of authorization strategy I choose, it's always taking longer time to open "{{*Project-A*}}" cloudbees folder (contains *6* items) than "{{*Projects*}}" folder (contains *34* items). Below is my benchmark: || || Chrome || Chrome || Firefox || Firefox || || Authorization Method || Open Projects Folder || Open Project-A Folder || Open Projects Folder || Open Project-A Folder || || Anyone can do anything | 2.10 sec | 8.43 sec | 5.99 sec | 17.05 sec | || Logged-in users can do anything | 3.97 sec | 9.94 sec | 7.97 sec | 20.06 sec | || Initial default Role-based configuration - admin role | 8.22 sec | 14.54 sec | 16.47 sec | 28.55 sec | || Role-based configuration - admin role with 119 Project roles | 18.81 sec | 31.95 sec | 37.7 sec | 65 | I'm expecting it should be faster to open "{{*Project-A*}}" folder because it needs to process permission and load only *6* items, and slower to open "{{*Projects*}}" folder because it needs to process permissions and load for *34* items.

New: I have a tree-like structure job layout as below: || "Projects" Cloudbees folder | contains *34* subproject folder | || || subproject: "Project-A" Cloudbees folder | contains *6* branch folders ( *5* multibranch job folder and *1* cloudbees folder ) | || || || multibranch-job1 | contains 295 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job2 | contains 317 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job3 | contains 189 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job4 | contains 15 matrix jobs (75 x 2 matrix combination) | || || || multibranch-job5 | contains 5 matrix jobs (75 x 2 matrix combination) | || || || Cloudbees folder | contains 4 matrix jobs (75 x 2 matrix combination) | || || subproject: Project-B Cloudbees folder | .... | || || || multibranch-job1 | ... | || || || ... | ... | || || ... || ... | ... | I found somehow no matter what kind of authorization strategy I choose, it's always taking longer time to open "{{*Project-A*}}" cloudbees folder (contains *6* items) than "{{*Projects*}}" folder (contains *34* items). Below is my benchmark: || || Chrome || Chrome || Firefox || Firefox || || Authorization Method || Open Projects Folder || Open Project-A Folder || Open Projects Folder || Open Project-A Folder || || Anyone can do anything | 2.10 sec | 8.43 sec | 5.99 sec | 17.05 sec | || Logged-in users can do anything | 3.97 sec | 9.94 sec | 7.97 sec | 20.06 sec | || Initial default Role-based configuration - admin role | 8.22 sec | 14.54 sec | 16.47 sec | 28.55 sec | || Role-based configuration - admin role with 119 Project roles | 18.81 sec | 31.95 sec | 37.7 sec | 65 | I'm expecting it should be faster to open "{{*Project-A*}}" folder because it needs to process permission and load only *6* items, and slower to open "{{*Projects*}}" folder because it needs to process permissions and load for *34* items.

Collapse comment: Rick Liu added a comment - 2017-02-04 03:04, Edited by Rick Liu - 2017-02-04 03:11

Rick Liu added a comment - 2017-02-04 03:04 - edited

Does the Jenkins core somehow process more items than needed for Multibranch jobs?
eg.
Projects folder contains 34 cloudbees folder and processed only this 34 items.
Project-A folder contains 6 items,
but somehow Jenkins core processed more to include Multibranch branch jobs (those hundred of matrix jobs).

Expand comment: Rick Liu added a comment - 2017-02-04 03:04, Edited by Rick Liu - 2017-02-04 03:11

Rick Liu added a comment - 2017-02-04 03:04 - edited Does the Jenkins core somehow process more items than needed for Multibranch jobs? eg. Projects folder contains 34 cloudbees folder and processed only this 34 items. Project-A folder contains 6 items, but somehow Jenkins core processed more to include Multibranch branch jobs (those hundred of matrix jobs).

Collapse comment: Rick Liu added a comment - 2017-02-04 03:07

Rick Liu added a comment - 2017-02-04 03:07

this issue is somewhat similar to JENKINS-18377 but not identical
because the phenomenon I see apply to not only Role-base strategy
but also to other different authorization.

Expand comment: Rick Liu added a comment - 2017-02-04 03:07

Rick Liu added a comment - 2017-02-04 03:07 this issue is somewhat similar to JENKINS-18377 but not identical because the phenomenon I see apply to not only Role-base strategy but also to other different authorization.

Collapse comment: Daniel Beck added a comment - 2017-02-04 14:46

Daniel Beck added a comment - 2017-02-04 14:46

There's no bug here. It's more expensive to compute the status of matrix jobs as each combination is considered an individual build and must be loaded.

Expand comment: Daniel Beck added a comment - 2017-02-04 14:46

Daniel Beck added a comment - 2017-02-04 14:46 There's no bug here. It's more expensive to compute the status of matrix jobs as each combination is considered an individual build and must be loaded.

Daniel Beck made changes - 2017-02-04 14:46

Resolution		New: Not A Defect [ 7 ]
Status	Original: Open [ 1 ]	New: Resolved [ 5 ]

Load more newer events