ci.jenkins.io froze, and danielbeck found that hundreds of threads were blocked in
"Handling GET /blue/rest/organizations/jenkins/computers/ from ... : ..." #... prio=5 os_prio=0 cpu=9992.49ms elapsed=2517.82s tid=... nid=... waiting for monitor entry [...]
java.lang.Thread.State: BLOCKED (on object monitor)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:370)
- waiting to lock <...> (a hudson.model.RunMap)
at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:228)
at org.jenkinsci.plugins.workflow.job.WorkflowJob.getBuildByNumber(WorkflowJob.java:232)
at io.jenkins.blueocean.rest.impl.pipeline.PipelineRunImpl.getCommitUrl(PipelineRunImpl.java:197)
at java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(java.base@11.0.4/DirectMethodHandle$Holder)
at java.lang.invoke.LambdaForm$MH/....invoke(java.base@11.0.4/LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/....invoke_MT(java.base@11.0.4/LambdaForm$MH)
at io.jenkins.blueocean.commons.stapler.export.MethodProperty.getValue(MethodProperty.java:72)
at io.jenkins.blueocean.commons.stapler.export.ExportInterceptor$1.getValue(ExportInterceptor.java:46)
at io.jenkins.blueocean.commons.stapler.Export$BlueOceanExportInterceptor.getValue(Export.java:196)
at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:136)
at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228)
at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:303)
at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:169)
at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:154)
at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228)
at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:303)
at io.jenkins.blueocean.commons.stapler.export.Property.writeBuffered(Property.java:175)
at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:235)
at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:169)
at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:154)
at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228)
at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:303)
at io.jenkins.blueocean.commons.stapler.export.Property.writeBuffered(Property.java:175)
at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:219)
at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:169)
at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:154)
at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228)
at io.jenkins.blueocean.commons.stapler.export.Model.writeTo(Model.java:199)
at io.jenkins.blueocean.commons.stapler.Export.writeOne(Export.java:177)
at io.jenkins.blueocean.commons.stapler.Export.serveExposedBean(Export.java:168)
at io.jenkins.blueocean.commons.stapler.Export.doJson(Export.java:108)
at io.jenkins.blueocean.commons.stapler.TreeResponse$Processor$1.generateResponse(TreeResponse.java:48)
at ...
Looking at the code, this seems like a classic lazy-loading violation: looping over possibly every historical build of a job, each load of which involves expensive disk I/O. Under no circumstances may a plugin do this.
This seems to have been noted in JENKINS-56773 but the response was just to disable the blueocean-executor-info plugin. How are users who already have this plugin installed supposed to know this? Some throttling was added, but evidently not enough.
Bug
Critical
