Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45614

Activity tab is really slow

    XMLWordPrintable

    Details

    • Similar Issues:
    • Epic Link:
    • Sprint:
      Blue Ocean 1.2-beta3, Blue Ocean 1.2-beta4, Blue Ocean 1.2, Blue Ocean 1.6 - beta 2, Blue Ocean - 1.6 - beta 4

      Description

      Je activity page is really slow, it takes around 10s to load and it's related to blue/organizations/jenkins/[PROJECT]/activity/?start=0&limit=26 API call that took ~90% of the time.
       
      The project is a Github organisation project with a Jenkinsfile
       
      Blue Ocean / Jenkins version: 1.1.4 · Core 2.68 · abed268 · (no branch) · 3rd July 2017 09:54 PM
       
       

        Attachments

          Issue Links

            Activity

            Hide
            jamesdumay James Dumay added a comment -

            Hi Julien Maitrehenry,

            Thanks for the report.

            Would you be able to do two things that would help me debug this issue?

            1. Capture a HAR file of a full page load of the activity screen
            2. Generate a support bundle using the Support Core Plugin.

            Then upload both of these to our secure drop. I should be able to use this info to track down your problem

            Thanks
            James

            Show
            jamesdumay James Dumay added a comment - Hi Julien Maitrehenry , Thanks for the report. Would you be able to do two things that would help me debug this issue? Capture a HAR file of a full page load of the activity screen Generate a support bundle using the Support Core Plugin . Then upload both of these to our secure drop . I should be able to use this info to track down your problem Thanks James
            Hide
            jamesdumay James Dumay added a comment -

            Hi Julien Maitrehenry,

            I haven't received any information that can help us reproduce the issue you have reported. If you are able to provide the data we need to reproduce, please reopen this ticket.

            Thanks
            James

            Show
            jamesdumay James Dumay added a comment - Hi Julien Maitrehenry , I haven't received any information that can help us reproduce the issue you have reported. If you are able to provide the data we need to reproduce, please reopen this ticket. Thanks James
            Hide
            jmaitrehenry Julien Maitrehenry added a comment - - edited

            Hi James,

             I'm sorry, I was on vacation, I will check today if I can reproduce it after upgrading plugins and Jenkins.

            If I still have an issue, I will reopen this ticket.

             Thanks,
            Julien

            Show
            jmaitrehenry Julien Maitrehenry added a comment - - edited Hi James,  I'm sorry, I was on vacation, I will check today if I can reproduce it after upgrading plugins and Jenkins. If I still have an issue, I will reopen this ticket.  Thanks, Julien
            Hide
            jamesdumay James Dumay added a comment -

            No problems Julien Maitrehenry. We've done some profiling of the rest endpoint that provides data for Activity and found some low hanging fruit (we think its 50-60% faster!) See https://github.com/jenkinsci/blueocean-plugin/pull/1323.

            Show
            jamesdumay James Dumay added a comment - No problems Julien Maitrehenry . We've done some profiling of the rest endpoint that provides data for Activity and found some low hanging fruit (we think its 50-60% faster!) See https://github.com/jenkinsci/blueocean-plugin/pull/1323 .
            Hide
            jmaitrehenry Julien Maitrehenry added a comment -

            Hi James,

            I have updated jenkins and all plugins and made some tests:
            1.1.7 · Core 2.74 · 9d9e8b6 · (no branch) · 11th August 2017 06:30 PM

            The 2 following API are slow:

            - /blue/rest/organizations/jenkins/pipelines/Github/[project]/branches/?filter=origin&start=0&limit=26 ~10s
            - /blue/rest/organizations/jenkins/pipelines/Github/[project]/runs/?start=0&limit=26 ~10s

            Thanks,
            Julien

            Show
            jmaitrehenry Julien Maitrehenry added a comment - Hi James, I have updated jenkins and all plugins and made some tests: 1.1.7 · Core 2.74 · 9d9e8b6 · (no branch) · 11th August 2017 06:30 PM The 2 following API are slow: - /blue/rest/organizations/jenkins/pipelines/Github/ [project] /branches/?filter=origin&start=0&limit=26 ~10s - /blue/rest/organizations/jenkins/pipelines/Github/ [project] /runs/?start=0&limit=26 ~10s Thanks, Julien
            Show
            jmaitrehenry Julien Maitrehenry added a comment - HARs: https://dockerswarm1.blob.core.windows.net/julien/jenkins-branches.har https://dockerswarm1.blob.core.windows.net/julien/jenkins-activity.har
            Hide
            jamesdumay James Dumay added a comment -

            Julien Maitrehenry could I also please get a support bundle? You can read about how to generate one here

            Show
            jamesdumay James Dumay added a comment - Julien Maitrehenry could I also please get a support bundle? You can read about how to generate one here
            Hide
            jmaitrehenry Julien Maitrehenry added a comment -

            James Dumay: done. Let me know if you need anything else.

            Show
            jmaitrehenry Julien Maitrehenry added a comment - James Dumay : done. Let me know if you need anything else.
            Hide
            jamesdumay James Dumay added a comment -

            OK thanks for the info!

            Show
            jamesdumay James Dumay added a comment - OK thanks for the info!
            Hide
            djviking Sverre Moe added a comment -

            This is really a cause for concern for us. It is hard to sell Jenkins to our developers with this load time.
            It was still slow after upgrading to Blue Ocean v1.5, running Jenkins 2.117 on Java 8u144. I sent Vivek Pandey a HAR recording from our Jenkins a few days ago.

            I am going to try a fresh install of Jenkins and set up all our pipelines and see if Blue Ocean is still slow after a few runs. Considering that Cloudbees own public dogfood Jenkins server loads Blue Ocean pretty fast, it might perhaps be some problem with our configuration or some of the installed plugins.

            Show
            djviking Sverre Moe added a comment - This is really a cause for concern for us. It is hard to sell Jenkins to our developers with this load time. It was still slow after upgrading to Blue Ocean v1.5, running Jenkins 2.117 on Java 8u144. I sent Vivek Pandey a HAR recording from our Jenkins a few days ago. I am going to try a fresh install of Jenkins and set up all our pipelines and see if Blue Ocean is still slow after a few runs. Considering that Cloudbees own public dogfood Jenkins server loads Blue Ocean pretty fast, it might perhaps be some problem with our configuration or some of the installed plugins.
            Hide
            vivek Vivek Pandey added a comment -

            Sverre Moe Sorry for not getting back on the activity tab HAR. It could be some plugins causing it as well, hard to tell.

            Have been busy with other commitments. We hope to make progress on it but can't give you a timeline at the moment.

            Thanks for patience.

            Show
            vivek Vivek Pandey added a comment - Sverre Moe Sorry for not getting back on the activity tab HAR. It could be some plugins causing it as well, hard to tell. Have been busy with other commitments. We hope to make progress on it but can't give you a timeline at the moment. Thanks for patience.
            Hide
            olamy Olivier Lamy added a comment -

            Sverre Moe I wonder if you guys have a lot of test results per build? I'd like to ensure my findings are right.

            Show
            olamy Olivier Lamy added a comment - Sverre Moe I wonder if you guys have a lot of test results per build? I'd like to ensure my findings are right.
            Hide
            olamy Olivier Lamy added a comment -

            As far I can see the bottleneck of the first load is due to loading unittest results which can take a while for large projects with a lot of branches, history and lot of unit tests.
            Note it is cache after the first call.

            Show
            olamy Olivier Lamy added a comment - As far I can see the bottleneck of the first load is due to loading unittest results which can take a while for large projects with a lot of branches, history and lot of unit tests. Note it is cache after the first call.
            Hide
            olamy Olivier Lamy added a comment -

            Some ideas I'm experimenting.
            Changing the cache to a cache with persistence (finished build are immutable).
            Loading tests information in parallel.

            Show
            olamy Olivier Lamy added a comment - Some ideas I'm experimenting. Changing the cache to a cache with persistence (finished build are immutable). Loading tests information in parallel.
            Hide
            olamy Olivier Lamy added a comment -

            After more tests/search/experimentation/discussions/coffee/beers.... :

            • Loading runs in parallel will be done (but optional with a feature flag to activate it)
            • remove the testSummary field to avoid heavy calculation on first activity screen
            • add an extra rest call in runDetails page to get the testSummary data
            Show
            olamy Olivier Lamy added a comment - After more tests/search/experimentation/discussions/coffee/beers.... : Loading runs in parallel will be done (but optional with a feature flag to activate it) remove the testSummary field to avoid heavy calculation on first activity screen add an extra rest call in runDetails page to get the testSummary data
            Hide
            djviking Sverre Moe added a comment - - edited

            Olivier Lamy Very few of our projects have actually test results that are published by Jenkins with JUnitResultArchiver. Most of our projects are C/C++ with Google Test, but few creates a test output to publish in Jenkins. Even projects without any tests results recorded by Jenkins take a long time to load initially.

            Show
            djviking Sverre Moe added a comment - - edited Olivier Lamy Very few of our projects have actually test results that are published by Jenkins with JUnitResultArchiver. Most of our projects are C/C++ with Google Test, but few creates a test output to publish in Jenkins. Even projects without any tests results recorded by Jenkins take a long time to load initially.
            Hide
            olamy Olivier Lamy added a comment -

            pr https://github.com/jenkinsci/blueocean-plugin/pull/1744
            note the optional feature with system property

            -Dblueocean.collectRuns.threads= a number of parallel threads to load blueRun data
            
            Show
            olamy Olivier Lamy added a comment - pr https://github.com/jenkinsci/blueocean-plugin/pull/1744 note the optional feature with system property -Dblueocean.collectRuns.threads= a number of parallel threads to load blueRun data
            Show
            olamy Olivier Lamy added a comment - pr merged https://github.com/jenkinsci/blueocean-plugin/commit/6b8ed888e37b86eebce7e843eea9987259703783
            Hide
            olamy Olivier Lamy added a comment -

            Sverre Moe fyi 1.6.0 has been released. Please let us know if it's faster for you. thanks!

            Show
            olamy Olivier Lamy added a comment - Sverre Moe fyi 1.6.0 has been released. Please let us know if it's faster for you. thanks!
            Hide
            djviking Sverre Moe added a comment -

            Olivier Lamy I will perform the update as soon as possible and try it out.

            Show
            djviking Sverre Moe added a comment - Olivier Lamy I will perform the update as soon as possible and try it out.
            Hide
            djviking Sverre Moe added a comment -

            I have just performed the upgrade. Blue Ocean is much faster, and it responds nicely.
            No more errors and warnings in the Jenkins log file regarding Blue Ocean either.
            Great work

            Show
            djviking Sverre Moe added a comment - I have just performed the upgrade. Blue Ocean is much faster, and it responds nicely. No more errors and warnings in the Jenkins log file regarding Blue Ocean either. Great work
            Hide
            olamy Olivier Lamy added a comment -

            Good to have happy users
            Did you try the property

            -Dblueocean.collectRuns.threads=
            

            or default setup only?

            Show
            olamy Olivier Lamy added a comment - Good to have happy users Did you try the property -Dblueocean.collectRuns.threads= or default setup only?
            Hide
            djviking Sverre Moe added a comment -

            Tried both
            As mentioned on the gitter chat, the property caused an IllegalArgumentException.

            Show
            djviking Sverre Moe added a comment - Tried both As mentioned on the gitter chat, the property caused an IllegalArgumentException.
            Hide
            dragoonis Paul Dragoonis added a comment -

            I'd like to know more information about collectRuns. I've reviewed the Java code behind this config value (https://github.com/jenkinsci/blueocean-plugin/pull/1744/files#diff-ade0bbe8660bd0c91243650f643ffc8eR168) and would like to know what the recommended number of threads should be?

             

            Has any performance results been published around threads vs cores ? any benchmarks to publish?

            I'm hesitant to enable this value until more results have been published.

            Please advise.

            Show
            dragoonis Paul Dragoonis added a comment - I'd like to know more information about collectRuns. I've reviewed the Java code behind this config value ( https://github.com/jenkinsci/blueocean-plugin/pull/1744/files#diff-ade0bbe8660bd0c91243650f643ffc8eR168)  and would like to know what the recommended number of threads should be?   Has any performance results been published around threads vs cores ? any benchmarks to publish? I'm hesitant to enable this value until more results have been published. Please advise.
            Hide
            dragoonis Paul Dragoonis added a comment -
            Show
            dragoonis Paul Dragoonis added a comment -  /cc Olivier Lamy James Dumay
            Hide
            olamy Olivier Lamy added a comment -

            Paul Dragoonishard to say an exact number as this will depend on your hardware. Maybe start with 8.

            Show
            olamy Olivier Lamy added a comment - Paul Dragoonis hard to say an exact number as this will depend on your hardware. Maybe start with 8.
            Hide
            hoppej Jan Hoppe added a comment -

            Hey Guys, 

            you did a great job to enhance the blue ocen performance!

             

            Thanks a lot!

            Show
            hoppej Jan Hoppe added a comment - Hey Guys,  you did a great job to enhance the blue ocen performance!   Thanks a lot!

              People

              Assignee:
              olamy Olivier Lamy
              Reporter:
              jmaitrehenry Julien Maitrehenry
              Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: