Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44747

Queue locked indefinitely in ThrottleQueueTaskDispatcher.categoriesForPipeline

XMLWordPrintable

      See this thread dump. This call can block (for example while a Pipeline build is getting ready to resume), whereas this caller ought to return quickly (milliseconds, if at all possible).

      Observed in a Jenkins server after restarting with some running builds. Since all Queue calls hanged, the system seemed to be frozen—no branch indexing, etc. Even taking a fresh support bundle hanged for the same reason (probably support-core ought to interrupt component threads which do not finish within some fixed timeout). Cannot cancel running builds for the same reason. The only way to recover was to disable the TCB plugin.

      Have also heard a field report of a server running TCB 2.0 exhibiting similar symptoms, but have not yet confirmed duplicate, pending existence of a thread dump.

          [JENKINS-44747] Queue locked indefinitely in ThrottleQueueTaskDispatcher.categoriesForPipeline

          Jesse Glick created issue -

          Andrew Bayer added a comment -

          Is there any way to determine whether the call for the FlowNode could block ahead of time?

          Andrew Bayer added a comment - Is there any way to determine whether the call for the FlowNode could block ahead of time?

          Jesse Glick added a comment -

          Filed PR 50. I would consider it more of a hot-patch to avoid the deadlock than a real fix—presumably the throttling would be ineffective if the timeout were in fact hit, which could cause other problems, though surely less severe.

          Jesse Glick added a comment - Filed PR 50. I would consider it more of a hot-patch to avoid the deadlock than a real fix—presumably the throttling would be ineffective if the timeout were in fact hit, which could cause other problems, though surely less severe.
          Jesse Glick made changes -
          Remote Link New: This issue links to "PR 50 (Web Link)" [ 16966 ]
          Jesse Glick made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]
          Jesse Glick made changes -
          Assignee Original: Andrew Bayer [ abayer ] New: Jesse Glick [ jglick ]
          Jesse Glick made changes -
          Status Original: In Progress [ 3 ] New: In Review [ 10005 ]

          SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jesse Glick
          Path:
          pom.xml
          src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java
          http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/a4cb745f7f616d5ffe3a212bb45f5c8612388157
          Log:
          JENKINS-44747 Limit the amount of time categoriesForPipeline can run.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/a4cb745f7f616d5ffe3a212bb45f5c8612388157 Log: JENKINS-44747 Limit the amount of time categoriesForPipeline can run.

          SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          README.md
          http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/c22a7e781ac0e393691f9b03b2221b209ab9abfe
          Log:
          Adding disclaimer for JENKINS-44747 change.

          Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/a4cb745f7f61^...c22a7e781ac0

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: README.md http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/c22a7e781ac0e393691f9b03b2221b209ab9abfe Log: Adding disclaimer for JENKINS-44747 change. Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/a4cb745f7f61 ^...c22a7e781ac0
          Jesse Glick made changes -
          Link New: This issue relates to JENKINS-44756 [ JENKINS-44756 ]

            jglick Jesse Glick
            jglick Jesse Glick
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: