Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49347

BlueOcean unusably slow, maybe related to having a lot of builds stored in history

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • blueocean-plugin
    • None
    • Blue Ocean 1.5 - beta 3

      During FOSDEM discussion about a problem we had with BlueOcean being prohibitively slow, a reasonable guess from markwaite was that our system in question stores a long history of builds - tens of thousands of those. Indeed, when the browser sends a request for BlueOcean (main panel, particular project, a build) the server spends a lot of time (minutes) in "wait" state for one of the CPU cores while others are idle or in small user time spending.

      Rendering the list of recent 50-100 builds or details of a particular build should not require inspecting all histories of all builds, right?

          [JENKINS-49347] BlueOcean unusably slow, maybe related to having a lot of builds stored in history

          Jim Klimov created issue -

          Vivek Pandey added a comment -

          There were few underlying issues. One being related to how favorites were rendered (affects logged in user with long build history) and the other had to do with unnecessary and expensive data returned as part of latestRun field.

          Fix is in review stage and should be merged soon. See https://github.com/jenkinsci/blueocean-plugin/pull/1638 and  https://github.com/jenkinsci/blueocean-plugin/pull/1635.

          Vivek Pandey added a comment - There were few underlying issues. One being related to how favorites were rendered (affects logged in user with long build history) and the other had to do with unnecessary and expensive data returned as part of latestRun field. Fix is in review stage and should be merged soon. See https://github.com/jenkinsci/blueocean-plugin/pull/1638  and   https://github.com/jenkinsci/blueocean-plugin/pull/1635 .

          James Dumay added a comment -

          jimklimov Ill close this against the issues quoted by vivek (likely to be the cause).

          However, would be great to capture a HAR file of specific page loads and capture a support bundle to help us diagnose these problems. We know there are optimizations to be make but we need specific info

          James Dumay added a comment - jimklimov Ill close this against the issues quoted by vivek (likely to be the cause). However, would be great to capture a HAR file of specific page loads and capture a support bundle to help us diagnose these problems. We know there are optimizations to be make but we need specific info
          James Dumay made changes -
          Link New: This issue duplicates JENKINS-48868 [ JENKINS-48868 ]
          James Dumay made changes -
          Link New: This issue duplicates JENKINS-44995 [ JENKINS-44995 ]
          James Dumay made changes -
          Resolution New: Duplicate [ 3 ]
          Status Original: Open [ 1 ] New: Resolved [ 5 ]
          Jim Klimov made changes -
          Attachment New: pipeline-BO-loadtimes-console.png [ 41519 ]
          Attachment New: pipeline-BO-loadtimes.7z [ 41520 ]
          Attachment New: pipeline-BO-loadtimes.png [ 41521 ]

          Jim Klimov added a comment - - edited

          Added HAR and respective screenshots (console is rather empty and uninteresting, but load times are spectacular).

          UPDATE: Added somewhat redacted support data.

          Statistics-wise, at the moment there are about 40k builds stored in history (and yes, much of the time spent in rendering BO can be Java walking the filesystem to find and parse all that build data):

          [root@jenkins2 jenkins/jobs]# sync; echo 3 > /proc/sys/vm/drop_caches
          [root@jenkins2 jenkins/jobs]# find . -name build.xml | wc -l
          39847
          
          real    12m40.689s
          user    0m12.259s
          sys     0m36.710s
          

          and 94 known "users" (many of which were auto-registered from upstream/3rd party projects' commits that were built here; about 20 actual internal users). Listing $JENKINS_URL/view/Masters/asynchPeople/ or going into the active users and listing builds they are associated with at $JENKINS_URL/user/myusername/builds also takes several minutes, at least first time after a filesystem cache flush for cleaner results (and comparably long with a "dirty" cache too).

          The system is a KVM VM, has 4Gb RAM assigned and about 1.3Gb of that is free. I'll try bumping it to 8Gb so more FS (meta)data can be cached, at least...

          Jim Klimov added a comment - - edited Added HAR and respective screenshots (console is rather empty and uninteresting, but load times are spectacular). UPDATE: Added somewhat redacted support data. Statistics-wise, at the moment there are about 40k builds stored in history (and yes, much of the time spent in rendering BO can be Java walking the filesystem to find and parse all that build data): [root@jenkins2 jenkins/jobs]# sync; echo 3 > /proc/sys/vm/drop_caches [root@jenkins2 jenkins/jobs]# find . -name build.xml | wc -l 39847 real 12m40.689s user 0m12.259s sys 0m36.710s and 94 known "users" (many of which were auto-registered from upstream/3rd party projects' commits that were built here; about 20 actual internal users). Listing $JENKINS_URL/view/Masters/asynchPeople/ or going into the active users and listing builds they are associated with at $JENKINS_URL/user/myusername/builds also takes several minutes, at least first time after a filesystem cache flush for cleaner results (and comparably long with a "dirty" cache too). The system is a KVM VM, has 4Gb RAM assigned and about 1.3Gb of that is free. I'll try bumping it to 8Gb so more FS (meta)data can be cached, at least...
          Jim Klimov made changes -
          Attachment New: support_2018-02-19_12.57.04.zip [ 41522 ]

          James Dumay added a comment -

          vivek - jimklimov has provided some HARs and a support bundle, so it should be possible to see the problem he is experiencing now.

          James Dumay added a comment - vivek - jimklimov has provided some HARs and a support bundle, so it should be possible to see the problem he is experiencing now.
          James Dumay made changes -
          Assignee New: Vivek Pandey [ vivek ]
          Resolution Original: Duplicate [ 3 ]
          Status Original: Resolved [ 5 ] New: Reopened [ 4 ]

            vivek Vivek Pandey
            jimklimov Jim Klimov
            Votes:
            1 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: