A user reports many builds stuck waiting for a lock on the project's primary workspace (not @2, etc.), which should only be possible if some code has acquired a "quick" lease on the workspace but then failed to ever release it. There are no threads shown holding the workspace lock. Several threads in AbstractProject.pollWithWorkspace and elsewhere are waiting for it, in some cases for days.

      https://github.com/jenkinsci/jenkins/blob/jenkins-1.580.1/core/src/main/java/hudson/model/AbstractProject.java#L1442-1445 shows a potential window of vulnerability: if getBuiltOn, createLauncher, decorateByEnv, or getEnvironment threw an exception, and it went unreported or was somehow lost in the logs, the lease could be left locked indefinitely.

          [JENKINS-26201] Unreleased workspace locks after SCM polling

          Jesse Glick created issue -
          Jesse Glick made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]

          Jesse Glick added a comment -

          Jesse Glick added a comment - http://repo.jenkins-ci.org/public/org/jenkins-ci/main/jenkins-war/1.580.1.JENKINS-26201-workspace-lock-1/jenkins-war-1.580.1.JENKINS-26201-workspace-lock-1.war may be used to test a backport of this.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          changelog.html
          http://jenkins-ci.org/commit/jenkins/568f906964bb3aaa09d6e7d2fb054256b75262cd
          Log:
          [FIXED JENKINS-26201] Noting 47fa17b, misattributed to an unrelated issue.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html http://jenkins-ci.org/commit/jenkins/568f906964bb3aaa09d6e7d2fb054256b75262cd Log: [FIXED JENKINS-26201] Noting 47fa17b, misattributed to an unrelated issue.
          SCM/JIRA link daemon made changes -
          Resolution New: Fixed [ 1 ]
          Status Original: In Progress [ 3 ] New: Resolved [ 5 ]
          Jesse Glick made changes -
          Labels Original: lock polling robustness workspace New: lock lts-candidate polling robustness workspace
          Jesse Glick made changes -
          Description Original: A user reports many builds stuck waiting for a lock on the project's primary workspace (not {{@2}}, etc.), which should only be possible if some code has acquired a "quick" lease on the workspace but then failed to ever release it. There are no threads shown holding the workspace lock. Several threads in {{AbstractProject.pollWithWorkspace}} and elsewhere are waiting for it, in some cases for days.

          https://github.com/jenkinsci/jenkins/blob/jenkins-1.580.1/core/src/main/java/hudson/model/AbstractProject.java#L1442-1445 shows a potential window of vulnerability: if {{getBuiltOn}}, {{createLauncher}}, {{decorateByEnv}}, or {{getEnvironment}} threw an exception, and it went reported or was somehow lost in the logs, the lease could be left locked indefinitely.
          New: A user reports many builds stuck waiting for a lock on the project's primary workspace (not {{@2}}, etc.), which should only be possible if some code has acquired a "quick" lease on the workspace but then failed to ever release it. There are no threads shown holding the workspace lock. Several threads in {{AbstractProject.pollWithWorkspace}} and elsewhere are waiting for it, in some cases for days.

          https://github.com/jenkinsci/jenkins/blob/jenkins-1.580.1/core/src/main/java/hudson/model/AbstractProject.java#L1442-1445 shows a potential window of vulnerability: if {{getBuiltOn}}, {{createLauncher}}, {{decorateByEnv}}, or {{getEnvironment}} threw an exception, and it went unreported or was somehow lost in the logs, the lease could be left locked indefinitely.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #3899
          [FIXED JENKINS-26201] Noting 47fa17b, misattributed to an unrelated issue. (Revision 568f906964bb3aaa09d6e7d2fb054256b75262cd)

          Result = UNSTABLE
          jesse glick : 568f906964bb3aaa09d6e7d2fb054256b75262cd
          Files :

          • changelog.html

          dogfood added a comment - Integrated in jenkins_main_trunk #3899 [FIXED JENKINS-26201] Noting 47fa17b, misattributed to an unrelated issue. (Revision 568f906964bb3aaa09d6e7d2fb054256b75262cd) Result = UNSTABLE jesse glick : 568f906964bb3aaa09d6e7d2fb054256b75262cd Files : changelog.html
          Oliver Gondža made changes -
          Labels Original: lock lts-candidate polling robustness workspace New: 1.596.1-fixed lock polling robustness workspace
          R. Tyler Croy made changes -
          Workflow Original: JNJira [ 160192 ] New: JNJira + In-Review [ 196359 ]

            jglick Jesse Glick
            jglick Jesse Glick
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: