-
Bug
-
Resolution: Unresolved
-
Critical
ci.jenkins.io froze, and danielbeck found that hundreds of threads were blocked in
"Handling GET /blue/rest/organizations/jenkins/computers/ from ... : ..." #... prio=5 os_prio=0 cpu=9992.49ms elapsed=2517.82s tid=... nid=... waiting for monitor entry [...] java.lang.Thread.State: BLOCKED (on object monitor) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:370) - waiting to lock <...> (a hudson.model.RunMap) at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:228) at org.jenkinsci.plugins.workflow.job.WorkflowJob.getBuildByNumber(WorkflowJob.java:232) at io.jenkins.blueocean.rest.impl.pipeline.PipelineRunImpl.getCommitUrl(PipelineRunImpl.java:197) at java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(java.base@11.0.4/DirectMethodHandle$Holder) at java.lang.invoke.LambdaForm$MH/....invoke(java.base@11.0.4/LambdaForm$MH) at java.lang.invoke.LambdaForm$MH/....invoke_MT(java.base@11.0.4/LambdaForm$MH) at io.jenkins.blueocean.commons.stapler.export.MethodProperty.getValue(MethodProperty.java:72) at io.jenkins.blueocean.commons.stapler.export.ExportInterceptor$1.getValue(ExportInterceptor.java:46) at io.jenkins.blueocean.commons.stapler.Export$BlueOceanExportInterceptor.getValue(Export.java:196) at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:136) at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228) at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:303) at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:169) at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:154) at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228) at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:303) at io.jenkins.blueocean.commons.stapler.export.Property.writeBuffered(Property.java:175) at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:235) at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:169) at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:154) at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228) at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:303) at io.jenkins.blueocean.commons.stapler.export.Property.writeBuffered(Property.java:175) at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:219) at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:169) at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:154) at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228) at io.jenkins.blueocean.commons.stapler.export.Model.writeTo(Model.java:199) at io.jenkins.blueocean.commons.stapler.Export.writeOne(Export.java:177) at io.jenkins.blueocean.commons.stapler.Export.serveExposedBean(Export.java:168) at io.jenkins.blueocean.commons.stapler.Export.doJson(Export.java:108) at io.jenkins.blueocean.commons.stapler.TreeResponse$Processor$1.generateResponse(TreeResponse.java:48) at ...
Looking at the code, this seems like a classic lazy-loading violation: looping over possibly every historical build of a job, each load of which involves expensive disk I/O. Under no circumstances may a plugin do this.
This seems to have been noted in JENKINS-56773 but the response was just to disable the blueocean-executor-info plugin. How are users who already have this plugin installed supposed to know this? Some throttling was added, but evidently not enough.
- is caused by
-
-
- Closed
-
- relates to
-
-
- Closed
-
- links to
[JENKINS-59706] Threads blocked in /blue/rest/organizations/jenkins/computers/ due to loop in PipelineRunImpl.getCommitUrl
Hi,
We're using version 1.21.0 of BlueOcean on Jenkins LTS 2.204.1 and the problem persists 
Calls to /blue/rest/organizations/jenkins/computers/ are so heavy that they freeze the browsers when accessing BlueOcean. We had to disable those calls at proxy level but this makes some features of BlueOcean not working any longer (like triggering a build from the branch view).
We do have an installation with up to 125 transient EC2 nodes and thousands of jobs running every day.
What could I provide which helps fixing the issue?
Thanks,
Damien Coraboeuf
Damien Coraboeuf
added a comment - - edited Hi,
We're using version 1.21.0 of BlueOcean on Jenkins LTS 2.204.1 and the problem persists
Calls to /blue/rest/organizations/jenkins/computers/ are so heavy that they freeze the browsers when accessing BlueOcean. We had to disable those calls at proxy level but this makes some features of BlueOcean not working any longer (like triggering a build from the branch view).
We do have an installation with up to 125 transient EC2 nodes and thousands of jobs running every day.
What could I provide which helps fixing the issue?
Thanks,
Damien Coraboeuf Damien Coraboeuf
added a comment - I've now seen JENKINS-56773 and we will uninstall the "Blue Ocean Executor Info" plugin and check.
Damien Coraboeuf
added a comment - I've now seen JENKINS-56773 and we will uninstall the "Blue Ocean Executor Info" plugin and check. 
Brian hogan
added a comment - damien_coraboeuf Did this resolve your performance issues in blue ocean?
Brian hogan
added a comment - damien_coraboeuf Did this resolve your performance issues in blue ocean? Damien Coraboeuf
added a comment - Somehow, yes. I followed instructions at JENKINS-56773 and uninstalled the Blue Ocean Executor Info plugin. This solved the issue immediately.
Damien Coraboeuf
added a comment - Somehow, yes. I followed instructions at JENKINS-56773 and uninstalled the Blue Ocean Executor Info plugin. This solved the issue immediately.
As far as I can tell, https://github.com/jenkinsci/blueocean-plugin/releases/tag/blueocean-parent-1.21.0 includes this change. Thanks jglick for that pull request! I think this can be marked 'Closed' in release blue ocean 1.21.0