Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56773

High master CPU usage introduced by Blue Ocean 1.14.0

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • blueocean-plugin
    • None
    • BlueOcean 1.14.0
      Jenkins 1.168
    • 1.15.0

      We went from BO 1.13.2 to BO 1.14.0.
      After the upgrade, we have seen a major CPU increase on our Jenkins master, leading to really slow UI and job execution, with threads being locked - see screenshot.

      Our master is configured to not run any job ("only build jobs with label expression matching this node").
      Using Jenkins monitor, we saw that ~90% of the CPU time was consumed by calls to the rest endpoint /blue/rest/organizations/jenkins/computers - see screenshot.

      Looking at the release notes from 1.14.0, it seems that JENKINS-52825 has introduced that endpoint and should be an area to investigate.

      We had to rollback BO to 1.13.2 which does not exhibit that behavior.

          [JENKINS-56773] High master CPU usage introduced by Blue Ocean 1.14.0

          Olivier Jacques created issue -

          /cc rodrigc - in case this can indeed be caused by JENKINS-52825.

          Olivier Jacques added a comment - /cc rodrigc - in case this can indeed be caused by JENKINS-52825 .
          Gavin Mogan made changes -
          Link New: This issue is related to JENKINS-56833 [ JENKINS-56833 ]

          Gavin Mogan added a comment -

          So had a bit of a chat with keith (original executor plugin author) and it was never really designed for more than a proof of concept on how to write a BlueOcean plugin.

          We are pretty sure its caused by https://github.com/jenkinsci/blueocean-plugin/blob/e2dd04dcb3015063a2e31c8d37c90ee5c1ef4d98/blueocean-executor-info/src/main/js/ExecutorInfoService.js#L14 (which will grab executor info every time any pipeline does work)

          So Plans:

          • I think the plan is going to be to drop making the executor plugin as a requirement, it'll have to be manually installed again (Sorry Craig)
          • Put a throttle on the lookup so it doesn't call nearly as often.

          Since there is a work around in place until the next release, and I personally have a few pressing issues to deal with first, i'm going to leave it unassigned, but we'll get it fixed before we do the next release.

          Gavin Mogan added a comment - So had a bit of a chat with keith (original executor plugin author) and it was never really designed for more than a proof of concept on how to write a BlueOcean plugin. We are pretty sure its caused by  https://github.com/jenkinsci/blueocean-plugin/blob/e2dd04dcb3015063a2e31c8d37c90ee5c1ef4d98/blueocean-executor-info/src/main/js/ExecutorInfoService.js#L14  (which will grab executor info every time any pipeline does work) So Plans: I think the plan is going to be to drop making the executor plugin as a requirement, it'll have to be manually installed again (Sorry Craig) Put a throttle on the lookup so it doesn't call nearly as often. Since there is a work around in place until the next release, and I personally have a few pressing issues to deal with first, i'm going to leave it unassigned, but we'll get it fixed before we do the next release.

          Thanks halkeye.

          Olivier Jacques added a comment - Thanks halkeye .

          Gavin Mogan added a comment -

          Gavin Mogan added a comment - https://github.com/jenkinsci/blueocean-plugin/pull/1947

          Devyn Goetsch added a comment -

          This impacted our company and we had to do the same rollback.

          we went from 1.6.x to 1.14.0, the next work day jenkins became unresponsive under load, and our jvm monitoring told us that the blue ocean rest computers endpoint was using most of the cpu time.

          We rolled back to 1.13.2 and disabled the Blue Ocean Executor Info plugin and our Jenkins master is far more responsive and page loads for blue ocean are much lower.

          Our builds have 3 stages, the first one is a single stage, the second one has 3 parallel stages, and the third one has 50 parallel stages, 32 of which are computed dynamically and launched with the "node" step.

          Please revert the change to make the Blue Ocean Executor Info plugin a required dependency.  I'm sure it would be okay on a smaller system, but for a system like ours it is untenable.

          Devyn Goetsch added a comment - This impacted our company and we had to do the same rollback. we went from 1.6.x to 1.14.0, the next work day jenkins became unresponsive under load, and our jvm monitoring told us that the blue ocean rest computers endpoint was using most of the cpu time. We rolled back to 1.13.2 and disabled the Blue Ocean Executor Info plugin and our Jenkins master is far more responsive and page loads for blue ocean are much lower. Our builds have 3 stages, the first one is a single stage, the second one has 3 parallel stages, and the third one has 50 parallel stages, 32 of which are computed dynamically and launched with the "node" step. Please revert the change to make the Blue Ocean Executor Info plugin a required dependency.  I'm sure it would be okay on a smaller system, but for a system like ours it is untenable.
          Gavin Mogan made changes -
          Link New: This issue is related to JENKINS-56833 [ JENKINS-56833 ]
          Gavin Mogan made changes -
          Released As New: 1.15.0
          Resolution New: Fixed [ 1 ]
          Status Original: Open [ 1 ] New: Resolved [ 5 ]
          Gavin Mogan made changes -
          Assignee New: Gavin Mogan [ halkeye ]

            olamy Olivier Lamy
            ojacques Olivier Jacques
            Votes:
            7 Vote for this issue
            Watchers:
            17 Start watching this issue

              Created:
              Updated:
              Resolved: