Details
-
Bug
-
Resolution: Unresolved
-
Critical
Description
ci.jenkins.io froze, and danielbeck found that hundreds of threads were blocked in
"Handling GET /blue/rest/organizations/jenkins/computers/ from ... : ..." #... prio=5 os_prio=0 cpu=9992.49ms elapsed=2517.82s tid=... nid=... waiting for monitor entry [...] java.lang.Thread.State: BLOCKED (on object monitor) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:370) - waiting to lock <...> (a hudson.model.RunMap) at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:228) at org.jenkinsci.plugins.workflow.job.WorkflowJob.getBuildByNumber(WorkflowJob.java:232) at io.jenkins.blueocean.rest.impl.pipeline.PipelineRunImpl.getCommitUrl(PipelineRunImpl.java:197) at java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(java.base@11.0.4/DirectMethodHandle$Holder) at java.lang.invoke.LambdaForm$MH/....invoke(java.base@11.0.4/LambdaForm$MH) at java.lang.invoke.LambdaForm$MH/....invoke_MT(java.base@11.0.4/LambdaForm$MH) at io.jenkins.blueocean.commons.stapler.export.MethodProperty.getValue(MethodProperty.java:72) at io.jenkins.blueocean.commons.stapler.export.ExportInterceptor$1.getValue(ExportInterceptor.java:46) at io.jenkins.blueocean.commons.stapler.Export$BlueOceanExportInterceptor.getValue(Export.java:196) at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:136) at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228) at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:303) at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:169) at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:154) at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228) at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:303) at io.jenkins.blueocean.commons.stapler.export.Property.writeBuffered(Property.java:175) at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:235) at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:169) at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:154) at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228) at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:303) at io.jenkins.blueocean.commons.stapler.export.Property.writeBuffered(Property.java:175) at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:219) at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:169) at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:154) at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228) at io.jenkins.blueocean.commons.stapler.export.Model.writeTo(Model.java:199) at io.jenkins.blueocean.commons.stapler.Export.writeOne(Export.java:177) at io.jenkins.blueocean.commons.stapler.Export.serveExposedBean(Export.java:168) at io.jenkins.blueocean.commons.stapler.Export.doJson(Export.java:108) at io.jenkins.blueocean.commons.stapler.TreeResponse$Processor$1.generateResponse(TreeResponse.java:48) at ...
Looking at the code, this seems like a classic lazy-loading violation: looping over possibly every historical build of a job, each load of which involves expensive disk I/O. Under no circumstances may a plugin do this.
This seems to have been noted in JENKINS-56773 but the response was just to disable the blueocean-executor-info plugin. How are users who already have this plugin installed supposed to know this? Some throttling was added, but evidently not enough.
Attachments
Issue Links
- is caused by
-
-
- Closed
-
- relates to
-
-
- Closed
-
- links to