Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-26746

Very slow when there are many folders and projects.

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • None
    • Jenkins 1.580.2, delivery-pipeline-plugin 0.8.8, build-pipeline-plugin 1.4.5, Ubuntu 12.04, Chrome, Oracle Java 1.7.

    Description

      Performance of the delivery pipeline view degrades significantly when using a high number (100+) of folders and projects. A single user request will cause the Jenkins java process to tie up a full CPU. A thread dump during this situation shows usually all threads idle except those with stacks similar to the snippet below.

      Example thread stack:

              at java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1066)
              at com.cloudbees.hudson.plugins.folder.Folder.getItems(Folder.java:503)
              at se.diabol.jenkins.pipeline.util.ProjectUtil.find(ProjectUtil.java:108)
              at se.diabol.jenkins.pipeline.util.ProjectUtil.find(ProjectUtil.java:110)
              at se.diabol.jenkins.pipeline.util.ProjectUtil.getProject(ProjectUtil.java:88)
              at se.diabol.jenkins.pipeline.util.BuildUtil.getUpstreamBuild(BuildUtil.java:40)
              at se.diabol.jenkins.pipeline.util.BuildUtil.getFirstUpstreamBuild(BuildUtil.java:65)
              at se.diabol.jenkins.pipeline.util.BuildUtil.match(BuildUtil.java:83)
              at se.diabol.jenkins.pipeline.domain.Task.getLatestTask(Task.java:142)
              at se.diabol.jenkins.pipeline.domain.Stage.createLatestStage(Stage.java:188)
              at se.diabol.jenkins.pipeline.domain.Pipeline.createPipelineLatest(Pipeline.java:149)
              at se.diabol.jenkins.pipeline.DeliveryPipelineView.getComponent(DeliveryPipelineView.java:328)
              at se.diabol.jenkins.pipeline.DeliveryPipelineView.getPipelines(DeliveryPipelineView.java:293)
      

      I have created pull request 94 to address this. This has been verified in a production Jenkins environment. Also, CPU load has been shown to be significantly reduced.

      Attachments

        Activity

          westse Steve West created issue -
          westse Steve West added a comment -

          Updated description to reference pull request that fixes the problem.

          westse Steve West added a comment - Updated description to reference pull request that fixes the problem.
          westse Steve West made changes -
          Field Original Value New Value
          Description Performance of the delivery pipeline view degrades significantly when using a high number (100+) of folders and projects. A single user request will cause the Jenkins java process to tie up a full CPU. A thread dump during this situation shows usually all threads idle except those with stacks similar to the snippet below.

          Example thread stack:
          {code}
                  at java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1066)
                  at com.cloudbees.hudson.plugins.folder.Folder.getItems(Folder.java:503)
                  at se.diabol.jenkins.pipeline.util.ProjectUtil.find(ProjectUtil.java:108)
                  at se.diabol.jenkins.pipeline.util.ProjectUtil.find(ProjectUtil.java:110)
                  at se.diabol.jenkins.pipeline.util.ProjectUtil.getProject(ProjectUtil.java:88)
                  at se.diabol.jenkins.pipeline.util.BuildUtil.getUpstreamBuild(BuildUtil.java:40)
                  at se.diabol.jenkins.pipeline.util.BuildUtil.getFirstUpstreamBuild(BuildUtil.java:65)
                  at se.diabol.jenkins.pipeline.util.BuildUtil.match(BuildUtil.java:83)
                  at se.diabol.jenkins.pipeline.domain.Task.getLatestTask(Task.java:142)
                  at se.diabol.jenkins.pipeline.domain.Stage.createLatestStage(Stage.java:188)
                  at se.diabol.jenkins.pipeline.domain.Pipeline.createPipelineLatest(Pipeline.java:149)
                  at se.diabol.jenkins.pipeline.DeliveryPipelineView.getComponent(DeliveryPipelineView.java:328)
                  at se.diabol.jenkins.pipeline.DeliveryPipelineView.getPipelines(DeliveryPipelineView.java:293)
          {code}
          Performance of the delivery pipeline view degrades significantly when using a high number (100+) of folders and projects. A single user request will cause the Jenkins java process to tie up a full CPU. A thread dump during this situation shows usually all threads idle except those with stacks similar to the snippet below.

          Example thread stack:
          {code}
                  at java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1066)
                  at com.cloudbees.hudson.plugins.folder.Folder.getItems(Folder.java:503)
                  at se.diabol.jenkins.pipeline.util.ProjectUtil.find(ProjectUtil.java:108)
                  at se.diabol.jenkins.pipeline.util.ProjectUtil.find(ProjectUtil.java:110)
                  at se.diabol.jenkins.pipeline.util.ProjectUtil.getProject(ProjectUtil.java:88)
                  at se.diabol.jenkins.pipeline.util.BuildUtil.getUpstreamBuild(BuildUtil.java:40)
                  at se.diabol.jenkins.pipeline.util.BuildUtil.getFirstUpstreamBuild(BuildUtil.java:65)
                  at se.diabol.jenkins.pipeline.util.BuildUtil.match(BuildUtil.java:83)
                  at se.diabol.jenkins.pipeline.domain.Task.getLatestTask(Task.java:142)
                  at se.diabol.jenkins.pipeline.domain.Stage.createLatestStage(Stage.java:188)
                  at se.diabol.jenkins.pipeline.domain.Pipeline.createPipelineLatest(Pipeline.java:149)
                  at se.diabol.jenkins.pipeline.DeliveryPipelineView.getComponent(DeliveryPipelineView.java:328)
                  at se.diabol.jenkins.pipeline.DeliveryPipelineView.getPipelines(DeliveryPipelineView.java:293)
          {code}

          I have created [pull request 94|https://github.com/Diabol/delivery-pipeline-plugin/pull/94] to address this. This has been verified in a production Jenkins environment. Also, CPU load has been shown to be significantly reduced.

          We are facing the same issue in our Jenkins production environment, which contains many folders and jobs. Looking forward to a new release containing this fix.

          damien_coraboeuf Damien Coraboeuf added a comment - We are facing the same issue in our Jenkins production environment, which contains many folders and jobs. Looking forward to a new release containing this fix.
          westse Steve West made changes -
          Priority Minor [ 4 ] Major [ 3 ]

          I just verified that the PR fix improves this dramatically. Well done @SteveWest.

          checketts Clint Checketts added a comment - I just verified that the PR fix improves this dramatically. Well done @SteveWest.

          Great job Steve! PR is merged. Thanks for testing Clint!
          Will release as soon as possible.

          patbos Patrik Boström added a comment - Great job Steve! PR is merged. Thanks for testing Clint! Will release as soon as possible.

          Will be release in 0.8.9

          patbos Patrik Boström added a comment - Will be release in 0.8.9
          patbos Patrik Boström made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]

          Released

          patbos Patrik Boström added a comment - Released
          patbos Patrik Boström made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          patbos Patrik Boström made changes -
          Resolution Fixed [ 1 ]
          Status Reopened [ 4 ] Resolved [ 5 ]
          patbos Patrik Boström made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          westse Steve West added a comment -

          Thanks for the quick turnaround. Impressive!

          westse Steve West added a comment - Thanks for the quick turnaround. Impressive!
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 160913 ] JNJira + In-Review [ 208402 ]

          People

            patbos Patrik Boström
            westse Steve West
            Votes:
            2 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: