Branch indexing blocked by workspace cleanup hung on bad connection

This issue is archived. You can view it, but you can't modify it. Learn more

XMLWordPrintable

      Branch indexing was observed to hang in several threads:

      "Executor #-1 for master : executing BranchIndexing[...] / waiting for ... state=TIMED_WAITING ...
          - waiting on <...> (a hudson.remoting.UserRequest)
          - locked <...> (a hudson.remoting.UserRequest)
          at java.lang.Object.wait(Native Method)
          at hudson.remoting.Request.call(Request.java:147)
          at hudson.remoting.Channel.call(Channel.java:829)
          at hudson.FilePath.act(FilePath.java:985)
          at hudson.FilePath.act(FilePath.java:974)
          at hudson.FilePath.isDirectory(FilePath.java:1507)
          at jenkins.branch.WorkspaceLocatorImpl$Deleter.cleanUp(WorkspaceLocatorImpl.java:133)
          at jenkins.branch.WorkspaceLocatorImpl$Deleter.onDeleted(WorkspaceLocatorImpl.java:125)
          at hudson.model.listeners.ItemListener$4.apply(ItemListener.java:205)
          at hudson.model.listeners.ItemListener$4.apply(ItemListener.java:203)
          at hudson.model.listeners.ItemListener.forAll(ItemListener.java:167)
          at hudson.model.listeners.ItemListener.fireOnDeleted(ItemListener.java:203)
          at com.cloudbees.hudson.plugins.folder.AbstractFolder.onDeleted(AbstractFolder.java:1195)
          at com.cloudbees.hudson.plugins.folder.AbstractFolder.onDeleted(AbstractFolder.java:149)
          at hudson.model.AbstractItem.delete(AbstractItem.java:575)
          at hudson.model.Job.delete(Job.java:683)
          at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
          at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:162)
          at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:967)
          at hudson.model.ResourceController.execute(ResourceController.java:98)
          at hudson.model.Executor.run(Executor.java:405)
      

      Apparently the agent connection had gone sour, which may indicate a bug in Remoting. Regardless, I have observed cases where, say, FilePath.isDirectory just hangs indefinitely for no reason I have been able to diagnose or reproduce. The Timeout utility in Pipeline tries to work around this for specific cases, introduced for JENKINS-32986. The fix of JENKINS-34564 should be amended to perform workspace cleanup asynchronously; there is no reason to block the thread calling Job.delete (usually branch indexing; for dead branches, could potentially be an HTTP request).

            Assignee:
            Jesse Glick
            Reporter:
            Jesse Glick
            Archiver:
            Jenkins Service Account

              Created:
              Updated:
              Resolved:
              Archived: