Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56773

High master CPU usage introduced by Blue Ocean 1.14.0

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • blueocean-plugin
    • None
    • BlueOcean 1.14.0
      Jenkins 1.168
    • 1.15.0

      We went from BO 1.13.2 to BO 1.14.0.
      After the upgrade, we have seen a major CPU increase on our Jenkins master, leading to really slow UI and job execution, with threads being locked - see screenshot.

      Our master is configured to not run any job ("only build jobs with label expression matching this node").
      Using Jenkins monitor, we saw that ~90% of the CPU time was consumed by calls to the rest endpoint /blue/rest/organizations/jenkins/computers - see screenshot.

      Looking at the release notes from 1.14.0, it seems that JENKINS-52825 has introduced that endpoint and should be an area to investigate.

      We had to rollback BO to 1.13.2 which does not exhibit that behavior.

          [JENKINS-56773] High master CPU usage introduced by Blue Ocean 1.14.0

          /cc rodrigc - in case this can indeed be caused by JENKINS-52825.

          Olivier Jacques added a comment - /cc rodrigc - in case this can indeed be caused by JENKINS-52825 .

          Gavin Mogan added a comment -

          So had a bit of a chat with keith (original executor plugin author) and it was never really designed for more than a proof of concept on how to write a BlueOcean plugin.

          We are pretty sure its caused by https://github.com/jenkinsci/blueocean-plugin/blob/e2dd04dcb3015063a2e31c8d37c90ee5c1ef4d98/blueocean-executor-info/src/main/js/ExecutorInfoService.js#L14 (which will grab executor info every time any pipeline does work)

          So Plans:

          • I think the plan is going to be to drop making the executor plugin as a requirement, it'll have to be manually installed again (Sorry Craig)
          • Put a throttle on the lookup so it doesn't call nearly as often.

          Since there is a work around in place until the next release, and I personally have a few pressing issues to deal with first, i'm going to leave it unassigned, but we'll get it fixed before we do the next release.

          Gavin Mogan added a comment - So had a bit of a chat with keith (original executor plugin author) and it was never really designed for more than a proof of concept on how to write a BlueOcean plugin. We are pretty sure its caused by  https://github.com/jenkinsci/blueocean-plugin/blob/e2dd04dcb3015063a2e31c8d37c90ee5c1ef4d98/blueocean-executor-info/src/main/js/ExecutorInfoService.js#L14  (which will grab executor info every time any pipeline does work) So Plans: I think the plan is going to be to drop making the executor plugin as a requirement, it'll have to be manually installed again (Sorry Craig) Put a throttle on the lookup so it doesn't call nearly as often. Since there is a work around in place until the next release, and I personally have a few pressing issues to deal with first, i'm going to leave it unassigned, but we'll get it fixed before we do the next release.

          Thanks halkeye.

          Olivier Jacques added a comment - Thanks halkeye .

          Gavin Mogan added a comment -

          Gavin Mogan added a comment - https://github.com/jenkinsci/blueocean-plugin/pull/1947

          Devyn Goetsch added a comment -

          This impacted our company and we had to do the same rollback.

          we went from 1.6.x to 1.14.0, the next work day jenkins became unresponsive under load, and our jvm monitoring told us that the blue ocean rest computers endpoint was using most of the cpu time.

          We rolled back to 1.13.2 and disabled the Blue Ocean Executor Info plugin and our Jenkins master is far more responsive and page loads for blue ocean are much lower.

          Our builds have 3 stages, the first one is a single stage, the second one has 3 parallel stages, and the third one has 50 parallel stages, 32 of which are computed dynamically and launched with the "node" step.

          Please revert the change to make the Blue Ocean Executor Info plugin a required dependency.  I'm sure it would be okay on a smaller system, but for a system like ours it is untenable.

          Devyn Goetsch added a comment - This impacted our company and we had to do the same rollback. we went from 1.6.x to 1.14.0, the next work day jenkins became unresponsive under load, and our jvm monitoring told us that the blue ocean rest computers endpoint was using most of the cpu time. We rolled back to 1.13.2 and disabled the Blue Ocean Executor Info plugin and our Jenkins master is far more responsive and page loads for blue ocean are much lower. Our builds have 3 stages, the first one is a single stage, the second one has 3 parallel stages, and the third one has 50 parallel stages, 32 of which are computed dynamically and launched with the "node" step. Please revert the change to make the Blue Ocean Executor Info plugin a required dependency.  I'm sure it would be okay on a smaller system, but for a system like ours it is untenable.

          Søren Friis added a comment -

          We are still seeing a very high amount of CPU time spend servicing the following endpoint on Blue Ocean 1.17.0

          /blue/rest/organizations/jenkins/computers/
          

          computers_GET.png

          I tried blocking the endpoint in my browser, and everything seemed to function normally on Blue Ocean pages.
          Can the polling of this endpoint be disabled or reduced?

          Søren Friis added a comment - We are still seeing a very high amount of CPU time spend servicing the following endpoint on Blue Ocean 1.17.0 /blue/rest/organizations/jenkins/computers/ computers_GET.png I tried blocking the endpoint in my browser, and everything seemed to function normally on Blue Ocean pages. Can the polling of this endpoint be disabled or reduced?

          Gavin Mogan added a comment -

          disable the executor plugin, its no longer required

          Gavin Mogan added a comment - disable the executor plugin, its no longer required

          Daniel Beck added a comment -

          Removing a plugin from the aggregator/meta plugin doesn't remove the broken functionality from any instances that have it installed.

          Like ci.jenkins.io which got shit for the last several weeks for its recent stability problems. Imagine my surprise seeing dozens of threads hanging in the endpoint mentioned in this issue, and how this issue has been resolved.

          At the very least, admins that generally take care of their instance need to be informed about this with an admin monitor.

          Daniel Beck added a comment - Removing a plugin from the aggregator/meta plugin doesn't remove the broken functionality from any instances that have it installed. Like ci.jenkins.io which got shit for the last several weeks for its recent stability problems. Imagine my surprise seeing dozens of threads hanging in the endpoint mentioned in this issue, and how this issue has been resolved. At the very least, admins that generally take care of their instance need to be informed about this with an admin monitor.

          Mark Waite added a comment -

          As far as I can tell, https://github.com/jenkinsci/blueocean-plugin/releases/tag/blueocean-parent-1.21.0 includes this change. Thanks jglick for that pull request! I think this can be marked 'Closed' in release blue ocean 1.21.0 with a code fix to prevent the issue.

          Can one of the users that encountered this problem confirm that it is now resolved?

          Mark Waite added a comment - As far as I can tell, https://github.com/jenkinsci/blueocean-plugin/releases/tag/blueocean-parent-1.21.0 includes this change. Thanks jglick for that pull request! I think this can be marked 'Closed' in release blue ocean 1.21.0 with a code fix to prevent the issue. Can one of the users that encountered this problem confirm that it is now resolved?

          Michał Woś added a comment - - edited

          We use Blue Ocean and Blue Ocean Executor Info plugins in version 1.22.0 and still have a problem with 

          ```

          /blue/rest/organizations/jenkins/computers/

          ```

          endpoint. It is called very often and hangs for ~5sec each time. Not sure why informations about all executors is needed in the first place when opened blue ocean for a given job and a run. Those informations do not seem to be utilised anywhere in the view.

          Michał Woś added a comment - - edited We use Blue Ocean and Blue Ocean Executor Info plugins in version 1.22.0 and still have a problem with  ``` /blue/rest/organizations/jenkins/computers/ ``` endpoint. It is called very often and hangs for ~5sec each time. Not sure why informations about all executors is needed in the first place when opened blue ocean for a given job and a run. Those informations do not seem to be utilised anywhere in the view.

          Mark Waite added a comment -

          mwos sorry that you've encountered the problem. Per the comment from Gavin Mogan, the e Blue Ocean Executor Info plugin is no longer required. You might be able to resolve the issue on your installation by removing the Blue Ocean Executor Info plugin. It is not installed on ci.jenkins.io (likely because Daniel Beck removed it). I was able to remove it from my Jenkins instance without seeing any harmful side effects.

          I agree with Daniel that it would be much better to have an administrative monitor or a Jenkins Health Advisor warning that Blue Ocean Executor Info is not required and may cause performance issues.

          Mark Waite added a comment - mwos sorry that you've encountered the problem. Per the comment from Gavin Mogan , the e Blue Ocean Executor Info plugin is no longer required. You might be able to resolve the issue on your installation by removing the Blue Ocean Executor Info plugin. It is not installed on ci.jenkins.io (likely because Daniel Beck removed it). I was able to remove it from my Jenkins instance without seeing any harmful side effects. I agree with Daniel that it would be much better to have an administrative monitor or a Jenkins Health Advisor warning that Blue Ocean Executor Info is not required and may cause performance issues.

          Chris Bayliss added a comment -

          Same issue. Enabling the executor plugin made our Jenkins instance unusable for a couple of days until we tracked the issue down to this bug and disabled it. Our Jenkins master took a massive performance hit with this plugin enabled. Please remove it if it's not required. 

          Chris Bayliss added a comment - Same issue. Enabling the executor plugin made our Jenkins instance unusable for a couple of days until we tracked the issue down to this bug and disabled it. Our Jenkins master took a massive performance hit with this plugin enabled. Please remove it if it's not required. 

          Gavin Mogan added a comment -

          remove it from what?

          Gavin Mogan added a comment - remove it from what?

          Remove the plugin from your Jenkins installation.

          Damien Coraboeuf added a comment - Remove the plugin from your Jenkins installation.

          Daniel Beck added a comment - - edited

          It's no longer a dependency in recent BO releases. You can just click the buttons.

          I don't think the Jenkins project should be able to decide what gets uninstalled on people's instances.

          Daniel Beck added a comment - - edited It's no longer a dependency in recent BO releases. You can just click the buttons. I don't think the Jenkins project should be able to decide what gets uninstalled on people's instances.

          efo plo added a comment -

          We've been hit by this issue too, with the (unwelcome) addition of it causing any endpoint (browser) to waste CPU cycles and eventually get stuck in any installation having over ~30 nodes. We have hundreds of nodes, so any call to `/computers` endpoint is bringing 4-5 MB each, and there are several requests each couple of seconds from each computer having any BlueOcean interface open (of which we have above 150).

          While I agree that Jenkins should not voluntarily remove plugins from people who installed them, this one deserves a separate treatment in my opinion, as a) it does not seem to bring any functionality to Jenkins by the virtue of it being a "demo" of how to write a BlueOcean plugin; and b) I believe most installations of it come from the people just selecting everything with "blueocean" in its name.

          efo plo added a comment - We've been hit by this issue too, with the (unwelcome) addition of it causing any endpoint (browser) to waste CPU cycles and eventually get stuck in any installation having over ~30 nodes. We have hundreds of nodes, so any call to `/computers` endpoint is bringing 4-5 MB each, and there are several requests each couple of seconds from each computer having any BlueOcean interface open (of which we have above 150). While I agree that Jenkins should not voluntarily remove plugins from people who installed them, this one deserves a separate treatment in my opinion, as a) it does not seem to bring any functionality to Jenkins by the virtue of it being a "demo" of how to write a BlueOcean plugin; and b) I believe most installations of it come from the people just selecting everything with "blueocean" in its name.

          Daniel Beck added a comment -

          this one deserves a separate treatment in my opinion

          I would rather not add an elaborate and certain to be controversial feature to Jenkins (core) for it to be used only once for an issue that was resolved a year ago, and then never again.

          It would be plenty good enough for new Blue Ocean releases to show a warning to administrators if this plugin is still installed (and enabled), but that's up to its maintainers.

           

          Daniel Beck added a comment - this one deserves a separate treatment in my opinion I would rather not add an elaborate and certain to be controversial feature to Jenkins (core) for it to be used only once for an issue that was resolved a year ago, and then never again. It would be plenty good enough for new Blue Ocean releases to show a warning to administrators if this plugin is still installed (and enabled), but that's up to its maintainers.  

          Jesse Glick added a comment -

          Should suffice for BO maintainers (halkeye? olamy?) to just cut a “tombstone” release of this plugin with zero code and prominently marked as deprecated. Then people who simply update all plugins routinely would get the tombstone and no longer encounter issues; they could uninstall the plugin at some point for purposes of cleanup.

          Jesse Glick added a comment - Should suffice for BO maintainers ( halkeye ? olamy ?) to just cut a “tombstone” release of this plugin with zero code and prominently marked as deprecated. Then people who simply update all plugins routinely would get the tombstone and no longer encounter issues; they could uninstall the plugin at some point for purposes of cleanup.

          Daniel Beck added a comment -

          marked as deprecated

          Good point, Jenkins 2.248 and newer also prominently shows which plugins are labeled deprecated; we could add this one: https://github.com/jenkins-infra/update-center2/pull/451

          (TBH it seems crazy to me that it's still getting new releases because it's still in the multimodule project…)

           

          Daniel Beck added a comment - marked as deprecated Good point, Jenkins 2.248 and newer also prominently shows which plugins are labeled deprecated; we could add this one: https://github.com/jenkins-infra/update-center2/pull/451 (TBH it seems crazy to me that it's still getting new releases because it's still in the multimodule project…)  

          ethorsa added a comment -

          I can't remove the plugin since it's still shown as a dependency of Blue Ocean. The enable and uninstall control are both greyed out. The Plugin is marked as deprecated though.

          Jenkins 2.249.2, Blue Ocean / Blue Ocean Executor Info v1.24.2.

          ethorsa added a comment - I can't remove the plugin since it's still shown as a dependency of Blue Ocean. The enable and uninstall control are both greyed out. The Plugin is marked as deprecated though. Jenkins 2.249.2, Blue Ocean / Blue Ocean Executor Info v1.24.2.

          Gavin Mogan added a comment -

          Please open a new issue if this is happening again, reviving old bugs is not going to get things addressed.

          Gavin Mogan added a comment - Please open a new issue if this is happening again, reviving old bugs is not going to get things addressed.

          Olivier Lamy added a comment - - edited

          The plugin does NOTHING now and will be removed with next release.

          Olivier Lamy added a comment - - edited The plugin does NOTHING now and will be removed with next release.

          Mark Waite added a comment - - edited

          I'm seeing the same condition that is described by ethorsa. Prior to Blue Ocean 1.24.1, I was able to uninstall the blueocean executor info plugin so that I could avoid this issue. With the release of blueocean I'm unable to uninstall the blueocean executor info plugin.

          I'm not seeing any issue from the blueocean-executor-info plugin itself, since (as noted by olamy) it now does nothing. It is an empty plugin.

          I assume the next release of Blue Ocean (possibly 1.24.3) will remove the dependency on blueocean executor info plugin so that I can uninstall it. Until then, the admin monitor warning will continue to be visible that tells me I have a deprecated plugin in my Jenkins installation. I prefer to have no admin monitors raising alerts on my Jenkins installation, but there does not appear to be any way to resolve this one other than waiting for the next release of Blue Ocean.

          Mark Waite added a comment - - edited I'm seeing the same condition that is described by ethorsa . Prior to Blue Ocean 1.24.1, I was able to uninstall the blueocean executor info plugin so that I could avoid this issue. With the release of blueocean I'm unable to uninstall the blueocean executor info plugin. I'm not seeing any issue from the blueocean-executor-info plugin itself, since (as noted by olamy ) it now does nothing. It is an empty plugin. I assume the next release of Blue Ocean (possibly 1.24.3) will remove the dependency on blueocean executor info plugin so that I can uninstall it. Until then, the admin monitor warning will continue to be visible that tells me I have a deprecated plugin in my Jenkins installation. I prefer to have no admin monitors raising alerts on my Jenkins installation, but there does not appear to be any way to resolve this one other than waiting for the next release of Blue Ocean.

          Olivier Lamy added a comment -

          I agree it's annoying... I will cut a new release next week so people will have enough time to notice this plugin has been deprecated

          Olivier Lamy added a comment - I agree it's annoying... I will cut a new release next week so people will have enough time to notice this plugin has been deprecated

          Paul Bruce added a comment -

          Also getting this in the past few days. To olamy 's point, would like guidance on what to do here.

          Paul Bruce added a comment - Also getting this in the past few days. To olamy 's point, would like guidance on what to do here.

          Mark Waite added a comment -

          paulsbruce if you've upgraded to blue ocean 1.24.2 and restarted your Jenkins server since the upgrade, then the blueocean-executor-info plugin described in this bug report cannot be causing the high CPU usage. With release 1.24.2, the blueocean-executor-info plugin contains no code.

          The contents of the plugin file (hpi / jpi file) are:

          M Filemode      Length  Date         Time      File
          - ----------  --------  -----------  --------  --------------------------------------------------------------------------
            drwxr-xr-x         0  27-Oct-2020  08:59:58  META-INF/
            -rw-r--r--       855  27-Oct-2020  08:59:56  META-INF/MANIFEST.MF
            drwxr-xr-x         0  27-Oct-2020  08:59:58  WEB-INF/
            drwxr-xr-x         0  27-Oct-2020  08:59:58  WEB-INF/lib/
            -rw-r--r--       531  27-Oct-2020  08:59:58  WEB-INF/licenses.xml
            -rw-r--r--      3648  27-Oct-2020  08:59:58  WEB-INF/lib/blueocean-executor-info.jar
            drwxr-xr-x         0  27-Oct-2020  08:59:58  META-INF/maven/
            drwxr-xr-x         0  27-Oct-2020  08:59:58  META-INF/maven/io.jenkins.blueocean/
            drwxr-xr-x         0  27-Oct-2020  08:59:58  META-INF/maven/io.jenkins.blueocean/blueocean-executor-info/
            -rw-r--r--      4010  27-Oct-2020  08:59:56  META-INF/maven/io.jenkins.blueocean/blueocean-executor-info/pom.xml
            -rw-r--r--       130  27-Oct-2020  08:59:58  META-INF/maven/io.jenkins.blueocean/blueocean-executor-info/pom.properties
          - ----------  --------  -----------  --------  --------------------------------------------------------------------------
                            9174                         11 files
          

          The contents of the jar file packaged in that plugin are:

          M Filemode      Length  Date         Time      File
          - ----------  --------  -----------  --------  --------------------------------------------------------------------------
            drwxr-xr-x         0  27-Oct-2020  08:59:58  META-INF/
            -rw-r--r--       855  27-Oct-2020  08:59:56  META-INF/MANIFEST.MF
            -rw-r--r--        93  27-Oct-2020  08:59:56  jenkins-js-extension.json
            -rw-r--r--      1290  27-Oct-2020  08:36:28  package.json
            -rw-r--r--        70  27-Oct-2020  08:59:56  index.jelly
            drwxr-xr-x         0  27-Oct-2020  08:59:58  META-INF/maven/
            drwxr-xr-x         0  27-Oct-2020  08:59:58  META-INF/maven/io.jenkins.blueocean/
            drwxr-xr-x         0  27-Oct-2020  08:59:58  META-INF/maven/io.jenkins.blueocean/blueocean-executor-info/
            -rw-r--r--      4010  27-Oct-2020  08:59:56  META-INF/maven/io.jenkins.blueocean/blueocean-executor-info/pom.xml
            -rw-r--r--       130  27-Oct-2020  08:59:58  META-INF/maven/io.jenkins.blueocean/blueocean-executor-info/pom.properties
          - ----------  --------  -----------  --------  --------------------------------------------------------------------------
                            6448                         10 files
          

          If you're seeing high cpu usage with blue ocean 1.24.2 installed, it is not a result of the blueocean-executor-info plugin described in this bug report.

          Mark Waite added a comment - paulsbruce if you've upgraded to blue ocean 1.24.2 and restarted your Jenkins server since the upgrade, then the blueocean-executor-info plugin described in this bug report cannot be causing the high CPU usage. With release 1.24.2, the blueocean-executor-info plugin contains no code. The contents of the plugin file (hpi / jpi file) are: M Filemode Length Date Time File - ---------- -------- ----------- -------- -------------------------------------------------------------------------- drwxr-xr-x 0 27-Oct-2020 08:59:58 META-INF/ -rw-r--r-- 855 27-Oct-2020 08:59:56 META-INF/MANIFEST.MF drwxr-xr-x 0 27-Oct-2020 08:59:58 WEB-INF/ drwxr-xr-x 0 27-Oct-2020 08:59:58 WEB-INF/lib/ -rw-r--r-- 531 27-Oct-2020 08:59:58 WEB-INF/licenses.xml -rw-r--r-- 3648 27-Oct-2020 08:59:58 WEB-INF/lib/blueocean-executor-info.jar drwxr-xr-x 0 27-Oct-2020 08:59:58 META-INF/maven/ drwxr-xr-x 0 27-Oct-2020 08:59:58 META-INF/maven/io.jenkins.blueocean/ drwxr-xr-x 0 27-Oct-2020 08:59:58 META-INF/maven/io.jenkins.blueocean/blueocean-executor-info/ -rw-r--r-- 4010 27-Oct-2020 08:59:56 META-INF/maven/io.jenkins.blueocean/blueocean-executor-info/pom.xml -rw-r--r-- 130 27-Oct-2020 08:59:58 META-INF/maven/io.jenkins.blueocean/blueocean-executor-info/pom.properties - ---------- -------- ----------- -------- -------------------------------------------------------------------------- 9174 11 files The contents of the jar file packaged in that plugin are: M Filemode Length Date Time File - ---------- -------- ----------- -------- -------------------------------------------------------------------------- drwxr-xr-x 0 27-Oct-2020 08:59:58 META-INF/ -rw-r--r-- 855 27-Oct-2020 08:59:56 META-INF/MANIFEST.MF -rw-r--r-- 93 27-Oct-2020 08:59:56 jenkins-js-extension.json -rw-r--r-- 1290 27-Oct-2020 08:36:28 package.json -rw-r--r-- 70 27-Oct-2020 08:59:56 index.jelly drwxr-xr-x 0 27-Oct-2020 08:59:58 META-INF/maven/ drwxr-xr-x 0 27-Oct-2020 08:59:58 META-INF/maven/io.jenkins.blueocean/ drwxr-xr-x 0 27-Oct-2020 08:59:58 META-INF/maven/io.jenkins.blueocean/blueocean-executor-info/ -rw-r--r-- 4010 27-Oct-2020 08:59:56 META-INF/maven/io.jenkins.blueocean/blueocean-executor-info/pom.xml -rw-r--r-- 130 27-Oct-2020 08:59:58 META-INF/maven/io.jenkins.blueocean/blueocean-executor-info/pom.properties - ---------- -------- ----------- -------- -------------------------------------------------------------------------- 6448 10 files If you're seeing high cpu usage with blue ocean 1.24.2 installed, it is not a result of the blueocean-executor-info plugin described in this bug report.

            olamy Olivier Lamy
            ojacques Olivier Jacques
            Votes:
            7 Vote for this issue
            Watchers:
            17 Start watching this issue

              Created:
              Updated:
              Resolved: