Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46738

Self-deadlock in ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable$1.interrupt

XMLWordPrintable

      What is the business impact?

      Jenkins stop working due the deadlock between support-core plugin and core/org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable$1.interrupt(ExecutorStepExecution.java:570)

      What is the incorrect behaviour?

      The deadlock should not happen

      What is the expected behaviour?

      You should be able to automatically create a bundle and stop a build execution

      Step by step to reproduce

      N/A

      Workaround

      Restart the instance, disable support-core plugin or avoid to stop builds. Issue does not seems to be very frequent.

      Regression

      I don't think this is a regression.

      Environment

      support-core:2.32:not-pinned
      Version: `2.7.21.1 (CloudBees Jenkins Enterprise 2.7.21.1-rolling)`

      Deadlock stacktrace

      Found one Java-level deadlock:
=============================
"SupportPlugin periodic bundle generator: writing support_2017-09-07_20.55.37.zip since Thu Sep 07 16:55:37 EDT 2017":
  waiting for ownable synchronizer 0x00000006f2ccb9e0, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "Handling POST /computer/(master)/executors/0/stop from XXXXXX : RequestHandlerThread[#155]"
"Handling POST /computer/(master)/executors/0/stop from XXXXXX : RequestHandlerThread[#155]":
  waiting for ownable synchronizer 0x00000006f3362550, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "Handling POST /computer/(master)/executors/4/stop from XXXXX : RequestHandlerThread[#161]"
"Handling POST /computer/(master)/executors/4/stop from XXXXXX : RequestHandlerThread[#161]":
  waiting for ownable synchronizer 0x00000006f41de5e0, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "Handling POST /computer/(master)/executors/6/stop from XXXXXXX : RequestHandlerThread[#162]"
"Handling POST /computer/(master)/executors/6/stop from XXXXXXXX : RequestHandlerThread[#162]":
  waiting for ownable synchronizer 0x00000006f2ccb9e0, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "Handling POST /computer/(master)/executors/0/stop from XXXXXXX : RequestHandlerThread[#155]"

Found 1 deadlock.

        1. deadlock-stacktrace.txt
          32 kB

          [JENKINS-46738] Self-deadlock in ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable$1.interrupt

          Jesse Glick added a comment -

          The support-core thread is innocent. We have three threads in a cyclic deadlock:

          "Handling POST /computer/(master)/executors/…/stop …":
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <…> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
	at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
	at hudson.model.Executor.getCurrentExecutable(Executor.java:503)
	at hudson.model.Executor.of(Executor.java:966)
	at hudson.model.Run.getExecutor(Run.java:517)
	at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable$1.interrupt(ExecutorStepExecution.java:570)
	at hudson.model.Executor.interrupt(Executor.java:227)
	at hudson.model.Executor.interrupt(Executor.java:197)
	at hudson.model.Executor.interrupt(Executor.java:187)
	at hudson.model.Executor.interrupt(Executor.java:173)
	at hudson.model.Executor.doStop(Executor.java:864)

          Executor.interrupt acquires a per-Executor lock, then calls into foreign code (AsynchronousExecution.interrupt), which in this case is calling Executor.of, attempting to acquire the locks of other executors.

          Jesse Glick added a comment - The support-core thread is innocent. We have three threads in a cyclic deadlock: "Handling POST /computer/(master)/executors/…/stop …": at sun.misc.Unsafe.park(Native Method) - parking to wait for <…> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283) at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727) at hudson.model.Executor.getCurrentExecutable(Executor.java:503) at hudson.model.Executor.of(Executor.java:966) at hudson.model.Run.getExecutor(Run.java:517) at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable$1.interrupt(ExecutorStepExecution.java:570) at hudson.model.Executor.interrupt(Executor.java:227) at hudson.model.Executor.interrupt(Executor.java:197) at hudson.model.Executor.interrupt(Executor.java:187) at hudson.model.Executor.interrupt(Executor.java:173) at hudson.model.Executor.doStop(Executor.java:864) Executor.interrupt acquires a per- Executor lock, then calls into foreign code ( AsynchronousExecution.interrupt ), which in this case is calling Executor.of , attempting to acquire the locks of other executors.

          Sam Van Oort added a comment -

          Released with v2.15

          Sam Van Oort added a comment - Released with v2.15

            jglick Jesse Glick
            fbelzunc Félix Belzunce Arcos
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: