[JENKINS-37034] Deadlock in hudson.model.Executor

Type: Bug
Resolution: Cannot Reproduce
Priority: Critical
Component/s: core
Labels:
None
Environment:
Jenkins 1.609

Similar Issues:
Powered by SuggestiMate

Show

We caught a deadlock in hudson.model.Executor with this stacktrace (XXX were sensitive data):

"Executor #-1 for XXX : executing XXX #160" daemon prio=10 tid=2249607168 nid=6359
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x5ba959038> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
        at hudson.model.Executor.interrupt(Executor.java:183)
        at hudson.model.Executor.interrupt(Executor.java:164)
        at hudson.model.Executor.interrupt(Executor.java:158)
        at hudson.model.Executor.interrupt(Executor.java:145)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.selfInterrupt(AbstractQueuedSynchronizer.java:802)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:937)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
        at hudson.model.Executor.getCurrentExecutable(Executor.java:475)
        at hudson.model.Executor.of(Executor.java:931)
        at hudson.model.Run.getExecutor(Run.java:517)
        at hudson.matrix.MatrixBuild$MatrixBuildExecution.doRun(MatrixBuild.java:376)
        - locked <0x402e23f78> (a hudson.model.Queue)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
        at hudson.model.Run.execute(Run.java:1738)
        at hudson.matrix.MatrixBuild.run(MatrixBuild.java:301)
        at hudson.model.ResourceController.execute(ResourceController.java:98)
        at hudson.model.Executor.run(Executor.java:381)

Because there is obtained lock for Queue first and than WriteLock waits to releasing ReadLock indefinitely, Jenkins doesn't response to anything (locked Queue)...

I've found a similar issue at ~~JENKINS-28690~~ for Executor.abortResult(). If I understand Stephen's fix correctly it can't be applied here because we don't know from where we have locked via ReadLock. Upgrading ReadLock to WriteLock is impossible way too.

is related to

JENKINS-28690 Deadlock in hudson.model.Executor

Closed

Pavel Janoušek created issue - 2016-07-28 12:15

Pavel Janoušek made changes - 2016-07-28 12:17

Description

Original: We caught a deadlock in {{hudson.model.Executor}} with this stacktrace (XXX were sensitive data):
{code}"Executor #-1 for XXX : executing XXX #160" daemon prio=10 tid=2249607168 nid=6359
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x5ba959038> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
        at hudson.model.Executor.interrupt(Executor.java:183)
        at hudson.model.Executor.interrupt(Executor.java:164)
        at hudson.model.Executor.interrupt(Executor.java:158)
        at hudson.model.Executor.interrupt(Executor.java:145)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.selfInterrupt(AbstractQueuedSynchronizer.java:802)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:937)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
        at hudson.model.Executor.getCurrentExecutable(Executor.java:475)
        at hudson.model.Executor.of(Executor.java:931)
        at hudson.model.Run.getExecutor(Run.java:517)
        at hudson.matrix.MatrixBuild$MatrixBuildExecution.doRun(MatrixBuild.java:376)
        - locked <0x402e23f78> (a hudson.model.Queue)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
        at hudson.model.Run.execute(Run.java:1738)
        at hudson.matrix.MatrixBuild.run(MatrixBuild.java:301)
        at hudson.model.ResourceController.execute(ResourceController.java:98)
        at hudson.model.Executor.run(Executor.java:381)
{code}

I've found a similar issue ~~JENKINS-28690~~ and if I understand Stephen's fix from {{Executor.abortResult()}} correctly, it could help in this situation too.

New: We caught a deadlock in {{hudson.model.Executor}} with this stacktrace (XXX were sensitive data):
{code}"Executor #-1 for XXX : executing XXX #160" daemon prio=10 tid=2249607168 nid=6359
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x5ba959038> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
        at hudson.model.Executor.interrupt(Executor.java:183)
        at hudson.model.Executor.interrupt(Executor.java:164)
        at hudson.model.Executor.interrupt(Executor.java:158)
        at hudson.model.Executor.interrupt(Executor.java:145)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.selfInterrupt(AbstractQueuedSynchronizer.java:802)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:937)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
        at hudson.model.Executor.getCurrentExecutable(Executor.java:475)
        at hudson.model.Executor.of(Executor.java:931)
        at hudson.model.Run.getExecutor(Run.java:517)
        at hudson.matrix.MatrixBuild$MatrixBuildExecution.doRun(MatrixBuild.java:376)
        - locked <0x402e23f78> (a hudson.model.Queue)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
        at hudson.model.Run.execute(Run.java:1738)
        at hudson.matrix.MatrixBuild.run(MatrixBuild.java:301)
        at hudson.model.ResourceController.execute(ResourceController.java:98)
        at hudson.model.Executor.run(Executor.java:381)
{code}

Because there is obtained lock for Queue first and than WriteLock wait to releasing ReadLock indefinitely, Jenkins doesn't response to anything...

I've found a similar issue ~~JENKINS-28690~~ and if I understand Stephen's fix from {{Executor.abortResult()}} correctly, it could help in this situation too.

Pavel Janoušek made changes - 2016-07-28 12:29

Description

Original: We caught a deadlock in {{hudson.model.Executor}} with this stacktrace (XXX were sensitive data):
{code}"Executor #-1 for XXX : executing XXX #160" daemon prio=10 tid=2249607168 nid=6359
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x5ba959038> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
        at hudson.model.Executor.interrupt(Executor.java:183)
        at hudson.model.Executor.interrupt(Executor.java:164)
        at hudson.model.Executor.interrupt(Executor.java:158)
        at hudson.model.Executor.interrupt(Executor.java:145)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.selfInterrupt(AbstractQueuedSynchronizer.java:802)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:937)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
        at hudson.model.Executor.getCurrentExecutable(Executor.java:475)
        at hudson.model.Executor.of(Executor.java:931)
        at hudson.model.Run.getExecutor(Run.java:517)
        at hudson.matrix.MatrixBuild$MatrixBuildExecution.doRun(MatrixBuild.java:376)
        - locked <0x402e23f78> (a hudson.model.Queue)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
        at hudson.model.Run.execute(Run.java:1738)
        at hudson.matrix.MatrixBuild.run(MatrixBuild.java:301)
        at hudson.model.ResourceController.execute(ResourceController.java:98)
        at hudson.model.Executor.run(Executor.java:381)
{code}

Because there is obtained lock for Queue first and than WriteLock wait to releasing ReadLock indefinitely, Jenkins doesn't response to anything...

I've found a similar issue ~~JENKINS-28690~~ and if I understand Stephen's fix from {{Executor.abortResult()}} correctly, it could help in this situation too.

New: We caught a deadlock in {{hudson.model.Executor}} with this stacktrace (XXX were sensitive data):
{code}"Executor #-1 for XXX : executing XXX #160" daemon prio=10 tid=2249607168 nid=6359
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x5ba959038> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
        at hudson.model.Executor.interrupt(Executor.java:183)
        at hudson.model.Executor.interrupt(Executor.java:164)
        at hudson.model.Executor.interrupt(Executor.java:158)
        at hudson.model.Executor.interrupt(Executor.java:145)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.selfInterrupt(AbstractQueuedSynchronizer.java:802)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:937)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
        at hudson.model.Executor.getCurrentExecutable(Executor.java:475)
        at hudson.model.Executor.of(Executor.java:931)
        at hudson.model.Run.getExecutor(Run.java:517)
        at hudson.matrix.MatrixBuild$MatrixBuildExecution.doRun(MatrixBuild.java:376)
        - locked <0x402e23f78> (a hudson.model.Queue)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
        at hudson.model.Run.execute(Run.java:1738)
        at hudson.matrix.MatrixBuild.run(MatrixBuild.java:301)
        at hudson.model.ResourceController.execute(ResourceController.java:98)
        at hudson.model.Executor.run(Executor.java:381)
{code}

Because there is obtained lock for Queue first and than WriteLock waits to releasing ReadLock indefinitely, Jenkins doesn't response to anything (locked Queue)...

I've found a similar issue at ~~JENKINS-28690~~ for {{Executor.abortResult()}}. If I understand Stephen's fix correctly it can't be applied here because we don't know from where we have locked via ReadLock. Upgrading ReadLock to WriteLock is impossible way too.

Pavel Janoušek made changes - 2016-07-28 12:29

Assignee

Original: Pavel Janoušek [ pajasoft ]

Pavel Janoušek made changes - 2016-08-02 14:12

Link

New: This issue is related to ~~JENKINS-28690~~ [ ~~JENKINS-28690~~ ]

Oleg Nenashev made changes - 2017-03-12 12:05

Resolution		New: Cannot Reproduce [ 5 ]
Status	Original: Open [ 1 ]	New: Resolved [ 5 ]

Oleg Nenashev made changes - 2017-03-12 12:05

Environment

New: Jenkins 1.609

Assignee:: Unassigned

Reporter:: Pavel Janoušek

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Created:: 2016-07-28 12:15

Updated:: 2017-03-12 12:05

Resolved:: 2017-03-12 12:05

Jenkins

Details

Description

Attachments

Issue Links

Activity

People

Dates