Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59706

Threads blocked in /blue/rest/organizations/jenkins/computers/ due to loop in PipelineRunImpl.getCommitUrl

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      ci.jenkins.io froze, and Daniel Beck found that hundreds of threads were blocked in

      "Handling GET /blue/rest/organizations/jenkins/computers/ from ... : ..." #... prio=5 os_prio=0 cpu=9992.49ms elapsed=2517.82s tid=... nid=... waiting for monitor entry  [...]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:370)
      	- waiting to lock <...> (a hudson.model.RunMap)
      	at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:228)
      	at org.jenkinsci.plugins.workflow.job.WorkflowJob.getBuildByNumber(WorkflowJob.java:232)
      	at io.jenkins.blueocean.rest.impl.pipeline.PipelineRunImpl.getCommitUrl(PipelineRunImpl.java:197)
      	at java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(java.base@11.0.4/DirectMethodHandle$Holder)
      	at java.lang.invoke.LambdaForm$MH/....invoke(java.base@11.0.4/LambdaForm$MH)
      	at java.lang.invoke.LambdaForm$MH/....invoke_MT(java.base@11.0.4/LambdaForm$MH)
      	at io.jenkins.blueocean.commons.stapler.export.MethodProperty.getValue(MethodProperty.java:72)
      	at io.jenkins.blueocean.commons.stapler.export.ExportInterceptor$1.getValue(ExportInterceptor.java:46)
      	at io.jenkins.blueocean.commons.stapler.Export$BlueOceanExportInterceptor.getValue(Export.java:196)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:136)
      	at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:303)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:169)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:154)
      	at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:303)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeBuffered(Property.java:175)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:235)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:169)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:154)
      	at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:303)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeBuffered(Property.java:175)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:219)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeValue(Property.java:169)
      	at io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:154)
      	at io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228)
      	at io.jenkins.blueocean.commons.stapler.export.Model.writeTo(Model.java:199)
      	at io.jenkins.blueocean.commons.stapler.Export.writeOne(Export.java:177)
      	at io.jenkins.blueocean.commons.stapler.Export.serveExposedBean(Export.java:168)
      	at io.jenkins.blueocean.commons.stapler.Export.doJson(Export.java:108)
      	at io.jenkins.blueocean.commons.stapler.TreeResponse$Processor$1.generateResponse(TreeResponse.java:48)
      	at ...
      

      Looking at the code, this seems like a classic lazy-loading violation: looping over possibly every historical build of a job, each load of which involves expensive disk I/O. Under no circumstances may a plugin do this.

      This seems to have been noted in JENKINS-56773 but the response was just to disable the blueocean-executor-info plugin. How are users who already have this plugin installed supposed to know this? Some throttling was added, but evidently not enough.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jglick Jesse Glick
              Reporter:
              jglick Jesse Glick
              Votes:
              2 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated: