Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59706

Threads blocked in /blue/rest/organizations/jenkins/computers/ due to loop in PipelineRunImpl.getCommitUrl

    XMLWordPrintable

Details

    Description

      ci.jenkins.io froze, and danielbeck found that hundreds of threads were blocked in

      "Handling GET /blue/rest/organizations/jenkins/computers/ from ... : ..." #... prio=5 os_prio=0 cpu=9992.49ms elapsed=2517.82s tid=... nid=... waiting for monitor entry  [...]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:370)
	- waiting to lock <...> (a hudson.model.RunMap)
	at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:228)
	at org.jenkinsci.plugins.workflow.job.WorkflowJob.getBuildByNumber(WorkflowJob.java:232)
	at io.jenkins.blueocean.rest.impl.pipeline.PipelineRunImpl.getCommitUrl(PipelineRunImpl.java:197)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(java.base@11.0.4/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/....invoke(java.base@11.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/....invoke_MT(java.base@11.0.4/LambdaForm$MH)
	at io.jenkins.blueocean.commons.stapler.export.MethodProperty.getValue(MethodProperty.java:72)
	at io.jenkins.blueocean.commons.stapler.export.ExportInterceptor$1.getValue(ExportInterceptor.java:46)
	at io.jenkins.blueocean.commons.stapler.Export$BlueOceanExportInterceptor.getValue(Export.java:196)
	at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:136)
	at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228)
	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:303)
	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:169)
	at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:154)
	at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228)
	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:303)
	at io.jenkins.blueocean.commons.stapler.export.Property.writeBuffered(Property.java:175)
	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:235)
	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:169)
	at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:154)
	at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228)
	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:303)
	at io.jenkins.blueocean.commons.stapler.export.Property.writeBuffered(Property.java:175)
	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:219)
	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:169)
	at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:154)
	at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228)
	at io.jenkins.blueocean.commons.stapler.export.Model.writeTo(Model.java:199)
	at io.jenkins.blueocean.commons.stapler.Export.writeOne(Export.java:177)
	at io.jenkins.blueocean.commons.stapler.Export.serveExposedBean(Export.java:168)
	at io.jenkins.blueocean.commons.stapler.Export.doJson(Export.java:108)
	at io.jenkins.blueocean.commons.stapler.TreeResponse$Processor$1.generateResponse(TreeResponse.java:48)
	at ...

      Looking at the code, this seems like a classic lazy-loading violation: looping over possibly every historical build of a job, each load of which involves expensive disk I/O. Under no circumstances may a plugin do this.

      This seems to have been noted in JENKINS-56773 but the response was just to disable the blueocean-executor-info plugin. How are users who already have this plugin installed supposed to know this? Some throttling was added, but evidently not enough.

      Attachments

        Issue Links

          is caused by

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-45011 Commit has in run result header is not clickable

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-56773 High master CPU usage introduced by Blue Ocean 1.14.0

          • Major - Major loss of function.
          • Closed
          links to

          Web Link blueocean #2029

          Activity

            Ascending order - Click to sort in descending order
            markewaite Mark Waite added a comment -

            As far as I can tell, https://github.com/jenkinsci/blueocean-plugin/releases/tag/blueocean-parent-1.21.0 includes this change. Thanks jglick for that pull request! I think this can be marked 'Closed' in release blue ocean 1.21.0

            markewaite Mark Waite added a comment - As far as I can tell, https://github.com/jenkinsci/blueocean-plugin/releases/tag/blueocean-parent-1.21.0 includes this change. Thanks jglick for that pull request! I think this can be marked 'Closed' in release blue ocean 1.21.0
            damien_coraboeuf Damien Coraboeuf added a comment - - edited

            Hi,

            We're using version 1.21.0 of BlueOcean on Jenkins LTS 2.204.1 and the problem persists

            Calls to /blue/rest/organizations/jenkins/computers/ are so heavy that they freeze the browsers when accessing BlueOcean. We had to disable those calls at proxy level but this makes some features of BlueOcean not working any longer (like triggering a build from the branch view).

            We do have an installation with up to 125 transient EC2 nodes and thousands of jobs running every day.

            What could I provide which helps fixing the issue?

            Thanks,
            Damien Coraboeuf

            damien_coraboeuf Damien Coraboeuf added a comment - - edited Hi, We're using version 1.21.0 of BlueOcean on Jenkins LTS 2.204.1 and the problem persists Calls to /blue/rest/organizations/jenkins/computers/ are so heavy that they freeze the browsers when accessing BlueOcean. We had to disable those calls at proxy level but this makes some features of BlueOcean not working any longer (like triggering a build from the branch view). We do have an installation with up to 125 transient EC2 nodes and thousands of jobs running every day. What could I provide which helps fixing the issue? Thanks, Damien Coraboeuf
            damien_coraboeuf Damien Coraboeuf added a comment -

            I've now seen JENKINS-56773 and we will uninstall the "Blue Ocean Executor Info" plugin and check.

            damien_coraboeuf Damien Coraboeuf added a comment - I've now seen JENKINS-56773 and we will uninstall the "Blue Ocean Executor Info" plugin and check.
            bhogan4 Brian hogan added a comment -

            damien_coraboeuf Did this resolve your performance issues in blue ocean?

            bhogan4 Brian hogan added a comment - damien_coraboeuf Did this resolve your performance issues in blue ocean?
            damien_coraboeuf Damien Coraboeuf added a comment -

            Somehow, yes. I followed instructions at JENKINS-56773 and uninstalled the Blue Ocean Executor Info plugin. This solved the issue immediately.

            damien_coraboeuf Damien Coraboeuf added a comment - Somehow, yes. I followed instructions at JENKINS-56773 and uninstalled the Blue Ocean Executor Info plugin. This solved the issue immediately.

            People

              jglick Jesse Glick
              jglick Jesse Glick
              Votes:
              2 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: