Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34450

Deadlock between Executor and AsynchronousExecution

    XMLWordPrintable

Details

    Description

      A deadlock has been observed sporadically during Pipeline-related tests:

      "jenkins.util.Timer [#3]":
  waiting for ownable synchronizer 0x00000006c25235e0, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "Executing ...SomeTest"
"Executing ...SomeTest":
  waiting to lock monitor 0x00007fd8b8004328 (object 0x00000006c254f860, a org.jenkinsci.plugins.workflow.job.WorkflowRun$2),
  which is held by "jenkins.util.Timer [#3]"
"jenkins.util.Timer [#3]":
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006c25235e0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
	at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
	at hudson.model.Executor.getCurrentWorkUnit(Executor.java:533)
	at hudson.model.queue.WorkUnitContext.synchronizeEnd(WorkUnitContext.java:141)
	at hudson.model.Executor.finish1(Executor.java:460)
	at hudson.model.Executor.completedAsynchronous(Executor.java:484)
	at jenkins.model.queue.AsynchronousExecution.completed(AsynchronousExecution.java:118)
	- locked <0x00000006c254f860> (a org.jenkinsci.plugins.workflow.job.WorkflowRun$2)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$3.run(WorkflowRun.java:249)
	- locked <0x00000006c253fc30> (a java.util.concurrent.atomic.AtomicBoolean)
	at ...
"Executing ...SomeTest":
	at jenkins.model.queue.AsynchronousExecution.getExecutor(AsynchronousExecution.java:98)
	- waiting to lock <0x00000006c254f860> (a org.jenkinsci.plugins.workflow.job.WorkflowRun$2)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$2.interrupt(WorkflowRun.java:234)
	at hudson.model.Executor.interrupt(Executor.java:227)
	at hudson.model.Executor.interrupt(Executor.java:193)
	at hudson.model.Executor.interrupt(Executor.java:187)
	at hudson.model.Executor.interrupt(Executor.java:173)
	at hudson.model.Executor.doStop(Executor.java:867)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.doStop(WorkflowRun.java:629)
	- locked <0x00000006c24f8f10> (a org.jenkinsci.plugins.workflow.job.WorkflowRun)
	at ...SomeTest

      I think the bug is in core, not WorkflowRun. The timer thread is holding an AsynchronousExecution monitor and waiting for the corresponding Executor read lock. This is entirely happening in core code. The test thread is holding the executor write lock (from doStop) and waiting for the AsynchronousExecution monitor. doStop winds up calling back into AsynchronousExecution.interrupt, which in this case is calling getExecutor, as indeed the Javadoc recommends.

      (The test thread is also holding a lock on the WorkflowRun, and the timer thread a lock on WorkflowRun.completed, both of which are potentially concerning, but these do not seem involved here.)

      Attachments

        Issue Links

          is blocking

          Improvement - An improvement or enhancement to an existing feature or task. JENKINS-25938 Lock an Executor without creating a Thread

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-31614 Deadlocks involving Queue

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link workflow-job PR 2

          Activity

            Ascending order - Click to sort in descending order
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java
            http://jenkins-ci.org/commit/workflow-job-plugin/26bc9d79451b2909f96b55bbb6c83827a8c8de3b
            Log:
            JENKINS-34450 Dangerous to acquire the AsynchronousExecution monitor while holding an Executor lock.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java http://jenkins-ci.org/commit/workflow-job-plugin/26bc9d79451b2909f96b55bbb6c83827a8c8de3b Log: JENKINS-34450 Dangerous to acquire the AsynchronousExecution monitor while holding an Executor lock.
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java
            http://jenkins-ci.org/commit/workflow-job-plugin/5e95677577ae786b89245591ae9ea2fa3b20c5f6
            Log:
            Merge pull request #2 from jglick/AsynchronousExecution-deadlock-JENKINS-34450

            JENKINS-34450 Work around deadlock in core

            Compare: https://github.com/jenkinsci/workflow-job-plugin/compare/b1505f8b5c55...5e95677577ae

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java http://jenkins-ci.org/commit/workflow-job-plugin/5e95677577ae786b89245591ae9ea2fa3b20c5f6 Log: Merge pull request #2 from jglick/AsynchronousExecution-deadlock- JENKINS-34450 JENKINS-34450 Work around deadlock in core Compare: https://github.com/jenkinsci/workflow-job-plugin/compare/b1505f8b5c55...5e95677577ae
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            job/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java
            http://jenkins-ci.org/commit/pipeline-plugin/7804b661cf9198044dea1ff352770297d2e90911
            Log:
            JENKINS-34450 Backport https://github.com/jenkinsci/workflow-job-plugin/pull/2 to 1.14.x line.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: job/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java http://jenkins-ci.org/commit/pipeline-plugin/7804b661cf9198044dea1ff352770297d2e90911 Log: JENKINS-34450 Backport https://github.com/jenkinsci/workflow-job-plugin/pull/2 to 1.14.x line.
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            CHANGES.md
            http://jenkins-ci.org/commit/pipeline-plugin/f1d6ce44be97781c98c16641b74ac7c4fdfae01a
            Log:
            JENKINS-34450 Noting backport to 1.14.1.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: CHANGES.md http://jenkins-ci.org/commit/pipeline-plugin/f1d6ce44be97781c98c16641b74ac7c4fdfae01a Log: JENKINS-34450 Noting backport to 1.14.1.

            People

              jglick Jesse Glick
              jglick Jesse Glick
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: