Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-26746

Very slow when there are many folders and projects.

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: delivery-pipeline-plugin
    • Labels:
      None
    • Environment:
      Jenkins 1.580.2, delivery-pipeline-plugin 0.8.8, build-pipeline-plugin 1.4.5, Ubuntu 12.04, Chrome, Oracle Java 1.7.

    • Similar Issues:

      Description

      Performance of the delivery pipeline view degrades significantly when using a high number (100+) of folders and projects. A single user request will cause the Jenkins java process to tie up a full CPU. A thread dump during this situation shows usually all threads idle except those with stacks similar to the snippet below.

      Example thread stack:

              at java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1066)
        at com.cloudbees.hudson.plugins.folder.Folder.getItems(Folder.java:503)
        at se.diabol.jenkins.pipeline.util.ProjectUtil.find(ProjectUtil.java:108)
        at se.diabol.jenkins.pipeline.util.ProjectUtil.find(ProjectUtil.java:110)
        at se.diabol.jenkins.pipeline.util.ProjectUtil.getProject(ProjectUtil.java:88)
        at se.diabol.jenkins.pipeline.util.BuildUtil.getUpstreamBuild(BuildUtil.java:40)
        at se.diabol.jenkins.pipeline.util.BuildUtil.getFirstUpstreamBuild(BuildUtil.java:65)
        at se.diabol.jenkins.pipeline.util.BuildUtil.match(BuildUtil.java:83)
        at se.diabol.jenkins.pipeline.domain.Task.getLatestTask(Task.java:142)
        at se.diabol.jenkins.pipeline.domain.Stage.createLatestStage(Stage.java:188)
        at se.diabol.jenkins.pipeline.domain.Pipeline.createPipelineLatest(Pipeline.java:149)
        at se.diabol.jenkins.pipeline.DeliveryPipelineView.getComponent(DeliveryPipelineView.java:328)
        at se.diabol.jenkins.pipeline.DeliveryPipelineView.getPipelines(DeliveryPipelineView.java:293)

      I have created pull request 94 to address this. This has been verified in a production Jenkins environment. Also, CPU load has been shown to be significantly reduced.

        Attachments

          Activity

          Ascending order - Click to sort in descending order
          westse Steve West created issue -
          Hide
          Permalink
          westse Steve West added a comment -

          Updated description to reference pull request that fixes the problem.

          Show
          westse Steve West added a comment - Updated description to reference pull request that fixes the problem.
          westse Steve West made changes -
          Field Original Value New Value
          Description Performance of the delivery pipeline view degrades significantly when using a high number (100+) of folders and projects. A single user request will cause the Jenkins java process to tie up a full CPU. A thread dump during this situation shows usually all threads idle except those with stacks similar to the snippet below.

          Example thread stack:
          {code}
                  at java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1066)
                  at com.cloudbees.hudson.plugins.folder.Folder.getItems(Folder.java:503)
                  at se.diabol.jenkins.pipeline.util.ProjectUtil.find(ProjectUtil.java:108)
                  at se.diabol.jenkins.pipeline.util.ProjectUtil.find(ProjectUtil.java:110)
                  at se.diabol.jenkins.pipeline.util.ProjectUtil.getProject(ProjectUtil.java:88)
                  at se.diabol.jenkins.pipeline.util.BuildUtil.getUpstreamBuild(BuildUtil.java:40)
                  at se.diabol.jenkins.pipeline.util.BuildUtil.getFirstUpstreamBuild(BuildUtil.java:65)
                  at se.diabol.jenkins.pipeline.util.BuildUtil.match(BuildUtil.java:83)
                  at se.diabol.jenkins.pipeline.domain.Task.getLatestTask(Task.java:142)
                  at se.diabol.jenkins.pipeline.domain.Stage.createLatestStage(Stage.java:188)
                  at se.diabol.jenkins.pipeline.domain.Pipeline.createPipelineLatest(Pipeline.java:149)
                  at se.diabol.jenkins.pipeline.DeliveryPipelineView.getComponent(DeliveryPipelineView.java:328)
                  at se.diabol.jenkins.pipeline.DeliveryPipelineView.getPipelines(DeliveryPipelineView.java:293)
          {code}           		Performance of the delivery pipeline view degrades significantly when using a high number (100+) of folders and projects. A single user request will cause the Jenkins java process to tie up a full CPU. A thread dump during this situation shows usually all threads idle except those with stacks similar to the snippet below.

          Example thread stack:
          {code}
                  at java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1066)
                  at com.cloudbees.hudson.plugins.folder.Folder.getItems(Folder.java:503)
                  at se.diabol.jenkins.pipeline.util.ProjectUtil.find(ProjectUtil.java:108)
                  at se.diabol.jenkins.pipeline.util.ProjectUtil.find(ProjectUtil.java:110)
                  at se.diabol.jenkins.pipeline.util.ProjectUtil.getProject(ProjectUtil.java:88)
                  at se.diabol.jenkins.pipeline.util.BuildUtil.getUpstreamBuild(BuildUtil.java:40)
                  at se.diabol.jenkins.pipeline.util.BuildUtil.getFirstUpstreamBuild(BuildUtil.java:65)
                  at se.diabol.jenkins.pipeline.util.BuildUtil.match(BuildUtil.java:83)
                  at se.diabol.jenkins.pipeline.domain.Task.getLatestTask(Task.java:142)
                  at se.diabol.jenkins.pipeline.domain.Stage.createLatestStage(Stage.java:188)
                  at se.diabol.jenkins.pipeline.domain.Pipeline.createPipelineLatest(Pipeline.java:149)
                  at se.diabol.jenkins.pipeline.DeliveryPipelineView.getComponent(DeliveryPipelineView.java:328)
                  at se.diabol.jenkins.pipeline.DeliveryPipelineView.getPipelines(DeliveryPipelineView.java:293)
          {code}

          I have created [pull request 94|https://github.com/Diabol/delivery-pipeline-plugin/pull/94] to address this. This has been verified in a production Jenkins environment. Also, CPU load has been shown to be significantly reduced.
          Hide
          Permalink
          damien_coraboeuf Damien Coraboeuf added a comment -

          We are facing the same issue in our Jenkins production environment, which contains many folders and jobs. Looking forward to a new release containing this fix.

          Show
          damien_coraboeuf Damien Coraboeuf added a comment - We are facing the same issue in our Jenkins production environment, which contains many folders and jobs. Looking forward to a new release containing this fix.
          westse Steve West made changes -
          Priority Minor [ 4 ] Major [ 3 ]
          Hide
          Permalink
          checketts Clint Checketts added a comment -

          I just verified that the PR fix improves this dramatically. Well done @SteveWest.

          Show
          checketts Clint Checketts added a comment - I just verified that the PR fix improves this dramatically. Well done @SteveWest.
          Hide
          Permalink
          patbos Patrik Boström added a comment -

          Great job Steve! PR is merged. Thanks for testing Clint!
          Will release as soon as possible.

          Show
          patbos Patrik Boström added a comment - Great job Steve! PR is merged. Thanks for testing Clint! Will release as soon as possible.
          Hide
          Permalink
          patbos Patrik Boström added a comment -

          Will be release in 0.8.9

          Show
          patbos Patrik Boström added a comment - Will be release in 0.8.9
          patbos Patrik Boström made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]
          Hide
          Permalink
          patbos Patrik Boström added a comment -

          Released

          Show
          patbos Patrik Boström added a comment - Released
          patbos Patrik Boström made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          patbos Patrik Boström made changes -
          Resolution Fixed [ 1 ]
          Status Reopened [ 4 ] Resolved [ 5 ]
          patbos Patrik Boström made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Permalink
          westse Steve West added a comment -

          Thanks for the quick turnaround. Impressive!

          Show
          westse Steve West added a comment - Thanks for the quick turnaround. Impressive!
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 160913 ] JNJira + In-Review [ 208402 ]

            People

            Assignee:
            patbos Patrik Boström
            Reporter:
            westse Steve West
            Votes:
            2 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: