Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44784

Branch indexing blocked by workspace cleanup hung on bad connection

      Branch indexing was observed to hang in several threads:

      "Executor #-1 for master : executing BranchIndexing[...] / waiting for ... state=TIMED_WAITING ...
          - waiting on <...> (a hudson.remoting.UserRequest)
          - locked <...> (a hudson.remoting.UserRequest)
          at java.lang.Object.wait(Native Method)
          at hudson.remoting.Request.call(Request.java:147)
          at hudson.remoting.Channel.call(Channel.java:829)
          at hudson.FilePath.act(FilePath.java:985)
          at hudson.FilePath.act(FilePath.java:974)
          at hudson.FilePath.isDirectory(FilePath.java:1507)
          at jenkins.branch.WorkspaceLocatorImpl$Deleter.cleanUp(WorkspaceLocatorImpl.java:133)
          at jenkins.branch.WorkspaceLocatorImpl$Deleter.onDeleted(WorkspaceLocatorImpl.java:125)
          at hudson.model.listeners.ItemListener$4.apply(ItemListener.java:205)
          at hudson.model.listeners.ItemListener$4.apply(ItemListener.java:203)
          at hudson.model.listeners.ItemListener.forAll(ItemListener.java:167)
          at hudson.model.listeners.ItemListener.fireOnDeleted(ItemListener.java:203)
          at com.cloudbees.hudson.plugins.folder.AbstractFolder.onDeleted(AbstractFolder.java:1195)
          at com.cloudbees.hudson.plugins.folder.AbstractFolder.onDeleted(AbstractFolder.java:149)
          at hudson.model.AbstractItem.delete(AbstractItem.java:575)
          at hudson.model.Job.delete(Job.java:683)
          at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
          at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:162)
          at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:967)
          at hudson.model.ResourceController.execute(ResourceController.java:98)
          at hudson.model.Executor.run(Executor.java:405)
      

      Apparently the agent connection had gone sour, which may indicate a bug in Remoting. Regardless, I have observed cases where, say, FilePath.isDirectory just hangs indefinitely for no reason I have been able to diagnose or reproduce. The Timeout utility in Pipeline tries to work around this for specific cases, introduced for JENKINS-32986. The fix of JENKINS-34564 should be amended to perform workspace cleanup asynchronously; there is no reason to block the thread calling Job.delete (usually branch indexing; for dead branches, could potentially be an HTTP request).

          [JENKINS-44784] Branch indexing blocked by workspace cleanup hung on bad connection

          Code changed in jenkins
          User: Jesse Glick
          Path:
          src/main/java/jenkins/branch/Timeout.java
          src/main/java/jenkins/branch/WorkspaceLocatorImpl.java
          src/test/java/jenkins/branch/WorkspaceLocatorImplTest.java
          http://jenkins-ci.org/commit/branch-api-plugin/bc1fe57c5b955e6aa3b6f9047b43c44d2007165d
          Log:
          [FIXED JENKINS-44784] Ensure workspace cleanup for deleted branch projects proceeds asynchronously, and will time out if the agent is stuck.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/jenkins/branch/Timeout.java src/main/java/jenkins/branch/WorkspaceLocatorImpl.java src/test/java/jenkins/branch/WorkspaceLocatorImplTest.java http://jenkins-ci.org/commit/branch-api-plugin/bc1fe57c5b955e6aa3b6f9047b43c44d2007165d Log: [FIXED JENKINS-44784] Ensure workspace cleanup for deleted branch projects proceeds asynchronously, and will time out if the agent is stuck.

          Jesse Glick added a comment -

          Released as 2.0.10. Due to a wiki outage I am unable to update release notes (CC stephenconnolly).

          Jesse Glick added a comment - Released as 2.0.10. Due to a wiki outage I am unable to update release notes (CC stephenconnolly ).

            jglick Jesse Glick
            jglick Jesse Glick
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: