Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49347

BlueOcean unusably slow, maybe related to having a lot of builds stored in history

    XMLWordPrintable

Details

    • Blue Ocean 1.5 - beta 3

    Description

      During FOSDEM discussion about a problem we had with BlueOcean being prohibitively slow, a reasonable guess from markwaite was that our system in question stores a long history of builds - tens of thousands of those. Indeed, when the browser sends a request for BlueOcean (main panel, particular project, a build) the server spends a lot of time (minutes) in "wait" state for one of the CPU cores while others are idle or in small user time spending.

      Rendering the list of recent 50-100 builds or details of a particular build should not require inspecting all histories of all builds, right?

      Attachments

        Issue Links

          Activity

            vivek Vivek Pandey added a comment -

            There were few underlying issues. One being related to how favorites were rendered (affects logged in user with long build history) and the other had to do with unnecessary and expensive data returned as part of latestRun field.

            Fix is in review stage and should be merged soon. See https://github.com/jenkinsci/blueocean-plugin/pull/1638 and  https://github.com/jenkinsci/blueocean-plugin/pull/1635.

            vivek Vivek Pandey added a comment - There were few underlying issues. One being related to how favorites were rendered (affects logged in user with long build history) and the other had to do with unnecessary and expensive data returned as part of latestRun field. Fix is in review stage and should be merged soon. See https://github.com/jenkinsci/blueocean-plugin/pull/1638  and   https://github.com/jenkinsci/blueocean-plugin/pull/1635 .
            jamesdumay James Dumay added a comment -

            jimklimov Ill close this against the issues quoted by vivek (likely to be the cause).

            However, would be great to capture a HAR file of specific page loads and capture a support bundle to help us diagnose these problems. We know there are optimizations to be make but we need specific info

            jamesdumay James Dumay added a comment - jimklimov Ill close this against the issues quoted by vivek (likely to be the cause). However, would be great to capture a HAR file of specific page loads and capture a support bundle to help us diagnose these problems. We know there are optimizations to be make but we need specific info
            jimklimov Jim Klimov added a comment - - edited

            Added HAR and respective screenshots (console is rather empty and uninteresting, but load times are spectacular).

            UPDATE: Added somewhat redacted support data.

            Statistics-wise, at the moment there are about 40k builds stored in history (and yes, much of the time spent in rendering BO can be Java walking the filesystem to find and parse all that build data):

            [root@jenkins2 jenkins/jobs]# sync; echo 3 > /proc/sys/vm/drop_caches
            [root@jenkins2 jenkins/jobs]# find . -name build.xml | wc -l
            39847
            
            real    12m40.689s
            user    0m12.259s
            sys     0m36.710s
            

            and 94 known "users" (many of which were auto-registered from upstream/3rd party projects' commits that were built here; about 20 actual internal users). Listing $JENKINS_URL/view/Masters/asynchPeople/ or going into the active users and listing builds they are associated with at $JENKINS_URL/user/myusername/builds also takes several minutes, at least first time after a filesystem cache flush for cleaner results (and comparably long with a "dirty" cache too).

            The system is a KVM VM, has 4Gb RAM assigned and about 1.3Gb of that is free. I'll try bumping it to 8Gb so more FS (meta)data can be cached, at least...

            jimklimov Jim Klimov added a comment - - edited Added HAR and respective screenshots (console is rather empty and uninteresting, but load times are spectacular). UPDATE: Added somewhat redacted support data. Statistics-wise, at the moment there are about 40k builds stored in history (and yes, much of the time spent in rendering BO can be Java walking the filesystem to find and parse all that build data): [root@jenkins2 jenkins/jobs]# sync; echo 3 > /proc/sys/vm/drop_caches [root@jenkins2 jenkins/jobs]# find . -name build.xml | wc -l 39847 real 12m40.689s user 0m12.259s sys 0m36.710s and 94 known "users" (many of which were auto-registered from upstream/3rd party projects' commits that were built here; about 20 actual internal users). Listing $JENKINS_URL/view/Masters/asynchPeople/ or going into the active users and listing builds they are associated with at $JENKINS_URL/user/myusername/builds also takes several minutes, at least first time after a filesystem cache flush for cleaner results (and comparably long with a "dirty" cache too). The system is a KVM VM, has 4Gb RAM assigned and about 1.3Gb of that is free. I'll try bumping it to 8Gb so more FS (meta)data can be cached, at least...
            jamesdumay James Dumay added a comment -

            vivek - jimklimov has provided some HARs and a support bundle, so it should be possible to see the problem he is experiencing now.

            jamesdumay James Dumay added a comment - vivek - jimklimov has provided some HARs and a support bundle, so it should be possible to see the problem he is experiencing now.
            vivek Vivek Pandey added a comment -

            I looked at the attached HAR file. Most slowness are around how favorites are pre-fetched and loaded subsequently.

            Fixes for optimized favorites loading (https://github.com/jenkinsci/blueocean-plugin/pull/1638) and minimizing returns in latestRun in pipeline should fixed this issue (https://github.com/jenkinsci/blueocean-plugin/pull/1635).

            These fixes went in 1.5.0-beta-1. I recommend trying latest 1.5.0-beta-2, and report back/reopen if the problem persists.

            vivek Vivek Pandey added a comment - I looked at the attached HAR file. Most slowness are around how favorites are pre-fetched and loaded subsequently. Fixes for optimized favorites loading ( https://github.com/jenkinsci/blueocean-plugin/pull/1638 ) and minimizing returns in latestRun in pipeline should fixed this issue ( https://github.com/jenkinsci/blueocean-plugin/pull/1635 ). These fixes went in 1.5.0-beta-1. I recommend trying latest 1.5.0-beta-2, and report back/reopen if the problem persists.
            smekkley smek added a comment -

            Looks like pull requests are merged. But I find it still a lot slower compared to normal pipeline view, when I open up a job.

            smekkley smek added a comment - Looks like pull requests are merged. But I find it still a lot slower compared to normal pipeline view, when I open up a job.

            We are fully up to date with Jenkins and plugins and Blue Ocean is still terrible to have to sit and wait through all of the stuff it loads.

            brianjmurrell Brian J Murrell added a comment - We are fully up to date with Jenkins and plugins and Blue Ocean is still terrible to have to sit and wait through all of the stuff it loads.
            mwos Michał Woś added a comment -

            Same here. Pipeline views load acceptably fast. However job view (.../blue/organizations/jenkins/JOB-NAME/activity/) is terribly slow. We keep only last 50 builds.

            mwos Michał Woś added a comment - Same here. Pipeline views load acceptably fast. However job view (.../blue/organizations/jenkins/JOB-NAME/activity/) is terribly slow. We keep only last 50 builds.
            brianjmurrell Brian J Murrell added a comment - - edited

            I have a theory about why Blue Ocean page loads can seem so slow.  If you look at the last, 3rd last (the highlighted one), 4th last and 7th last items in: you can see that it takes 30s (yes, 30 seconds!). Every Blue Ocean page load does this, AFAICT. It seems to be some kind of GET that is meant to take 30s for some reason.

            The problem with this kind of behaviour by the (Jenkins web) server is that modern browsers limit the number of sockets (6 for Chrome, IIRC) that are allowed to be open to any single server. When you have a page that ties up a socket for 30s, repeatedly in a "live refresh" (i.e. AJAX) type of web page it doesn't take too many tabs open to Blue Ocean pages to tie up all of the limited number of sockets the browser will open to a server with these 30s page loads. All other fetches to the Jenkins server then block waiting for one of these sockets to come out of their 30 GET.

            This can be seen by going to a Classic View Jenkins page and watching the status bar on the lower right on Chrome saying that it's waiting for available sockets.  I get that very frequently when I have a number of open Blue Ocean pages.  And it goes away as I close down the number of open Blue Ocean pages.

            Having a limit on the number of open Blue Ocean pages should NOT be considered the solution to this problem.

            brianjmurrell Brian J Murrell added a comment - - edited I have a theory about why Blue Ocean page loads can seem so slow.  If you look at the last, 3rd last (the highlighted one), 4th last and 7th last items in: you can see that it takes 30s (yes, 30 seconds!). Every Blue Ocean page load does this, AFAICT. It seems to be some kind of GET that is meant to take 30s for some reason. The problem with this kind of behaviour by the (Jenkins web) server is that modern browsers limit the number of sockets (6 for Chrome, IIRC) that are allowed to be open to any single server. When you have a page that ties up a socket for 30s, repeatedly in a "live refresh" (i.e. AJAX) type of web page it doesn't take too many tabs open to Blue Ocean pages to tie up all of the limited number of sockets the browser will open to a server with these 30s page loads. All other fetches to the Jenkins server then block waiting for one of these sockets to come out of their 30 GET . This can be seen by going to a Classic View Jenkins page and watching the status bar on the lower right on Chrome saying that it's waiting for available sockets.  I get that very frequently when I have a number of open Blue Ocean pages.  And it goes away as I close down the number of open Blue Ocean pages. Having a limit on the number of open Blue Ocean pages should NOT be considered the solution to this problem.

            People

              vivek Vivek Pandey
              jimklimov Jim Klimov
              Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: