Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44784

Branch indexing blocked by workspace cleanup hung on bad connection

      Branch indexing was observed to hang in several threads:

      "Executor #-1 for master : executing BranchIndexing[...] / waiting for ... state=TIMED_WAITING ...
          - waiting on <...> (a hudson.remoting.UserRequest)
          - locked <...> (a hudson.remoting.UserRequest)
          at java.lang.Object.wait(Native Method)
          at hudson.remoting.Request.call(Request.java:147)
          at hudson.remoting.Channel.call(Channel.java:829)
          at hudson.FilePath.act(FilePath.java:985)
          at hudson.FilePath.act(FilePath.java:974)
          at hudson.FilePath.isDirectory(FilePath.java:1507)
          at jenkins.branch.WorkspaceLocatorImpl$Deleter.cleanUp(WorkspaceLocatorImpl.java:133)
          at jenkins.branch.WorkspaceLocatorImpl$Deleter.onDeleted(WorkspaceLocatorImpl.java:125)
          at hudson.model.listeners.ItemListener$4.apply(ItemListener.java:205)
          at hudson.model.listeners.ItemListener$4.apply(ItemListener.java:203)
          at hudson.model.listeners.ItemListener.forAll(ItemListener.java:167)
          at hudson.model.listeners.ItemListener.fireOnDeleted(ItemListener.java:203)
          at com.cloudbees.hudson.plugins.folder.AbstractFolder.onDeleted(AbstractFolder.java:1195)
          at com.cloudbees.hudson.plugins.folder.AbstractFolder.onDeleted(AbstractFolder.java:149)
          at hudson.model.AbstractItem.delete(AbstractItem.java:575)
          at hudson.model.Job.delete(Job.java:683)
          at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
          at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:162)
          at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:967)
          at hudson.model.ResourceController.execute(ResourceController.java:98)
          at hudson.model.Executor.run(Executor.java:405)
      

      Apparently the agent connection had gone sour, which may indicate a bug in Remoting. Regardless, I have observed cases where, say, FilePath.isDirectory just hangs indefinitely for no reason I have been able to diagnose or reproduce. The Timeout utility in Pipeline tries to work around this for specific cases, introduced for JENKINS-32986. The fix of JENKINS-34564 should be amended to perform workspace cleanup asynchronously; there is no reason to block the thread calling Job.delete (usually branch indexing; for dead branches, could potentially be an HTTP request).

          [JENKINS-44784] Branch indexing blocked by workspace cleanup hung on bad connection

          Jesse Glick created issue -
          Jesse Glick made changes -
          Link New: This issue blocks JENKINS-34564 [ JENKINS-34564 ]
          Jesse Glick made changes -
          Link New: This issue relates to JENKINS-44785 [ JENKINS-44785 ]
          Jesse Glick made changes -
          Assignee New: Jesse Glick [ jglick ]
          Jesse Glick made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]
          Jesse Glick made changes -
          Remote Link New: This issue links to "PR 101 (Web Link)" [ 17104 ]
          Jesse Glick made changes -
          Status Original: In Progress [ 3 ] New: In Review [ 10005 ]
          Jesse Glick made changes -
          Resolution New: Fixed [ 1 ]
          Status Original: In Review [ 10005 ] New: Resolved [ 5 ]
          James Dumay made changes -
          Remote Link New: This issue links to "CloudBees Internal OSS-2311 (Web Link)" [ 18347 ]

            jglick Jesse Glick
            jglick Jesse Glick
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: