[JENKINS-30057] Slaves NodeProperties are owned by Jenkins not Nodes.

Type: Bug
Resolution: Fixed
Priority: Minor
Component/s: core
Labels:
- 1.625.1-fixed
- deadlock

Similar Issues:
Powered by SuggestiMate

Show

A Slaves nodeProperties are owned by Jenkins.

So whenever a NodeProperty is changed this causes persistance of the entire jenkins configuration rather than just Nodes which is what actually persists the NodeProperties.

Whilst this is just inefficient - if anyone wants to add a node property to new nodes created (e.g. by implementing a cloud listener) then is it possible to get a deadlock if someone also saves the Jenkins configuration at the same time. (as Jenkins.save() will lock Jenkins and then the queue, but callback for Nodes can happen with the queue locked).

"jenkins.util.Timer [#3]" id=34 (0x22) state=BLOCKED cpu=91%
    - waiting to lock <0x107c53da> (a hudson.model.Hudson)
     owned by Handling POST /configSubmit : http-8080-5 id=42
    at jenkins.model.Jenkins.save(Jenkins.java:2734)
    at hudson.util.PersistedList.onModified(PersistedList.java:173)
    at hudson.util.PersistedList._onModified(PersistedList.java:181)
    at hudson.util.PersistedList.add(PersistedList.java:72)
    at com.cloudbees.librato.CloudListener.onComplete(CloudListener.java:83)
    at hudson.slaves.NodeProvisioner$2.run(NodeProvisioner.java:223)
    at hudson.model.Queue._withLock(Queue.java:1235)
    at hudson.model.Queue.withLock(Queue.java:1171)
    at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:208)
    at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:57)
    at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:777)
    at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

    Locked synchronizers: count = 3
      - java.util.concurrent.locks.ReentrantLock$NonfairSync@2e7d6057
      - java.util.concurrent.ThreadPoolExecutor$Worker@6d40c3f9
      - java.util.concurrent.locks.ReentrantLock$NonfairSync@640c882b



"Handling POST /configSubmit : http-8080-5" id=42 (0x2a) state=WAITING cpu=96%
    - waiting on <0x2e7d6057> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    - locked <0x2e7d6057> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
     owned by jenkins.util.Timer [#3] id=34
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
    at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
    at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
    at hudson.model.Queue._withLock(Queue.java:1233)
    at hudson.model.Queue.withLock(Queue.java:1171)
    at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:168)
    at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
    at jenkins.model.Jenkins.doConfigSubmit(Jenkins.java:2858)
...

is related to

JENKINS-30060 Listeners are called back whilst holding locks which is prone to deadlocks

Open

links to

PR#1807

James Nord created issue - 2015-08-20 13:52

James Nord made changes - 2015-08-20 13:53

Description

Original: A [Slaves nodeProperties](https://github.com/jenkinsci/jenkins/blob/9c443c8d5bafd63fce574f6d0cf400cd8fe1f124/core/src/main/java/hudson/model/Slave.java#L130) are owned by Jenkins.

So whenever a NodeProperty is changed this causes persistance of the entire jenkins configuration rather than just Nodes which is what actually persists the NodeProperties.

Whilst this is just inefficient - if anyone wants to add a node property to new nodes created (e.g. by implementing a cloud listener) then is it possible to get a deadlock if someone also saves the Jenkins configuration at the same time. (as Jenkins.save() will lock Jenkins and then the queue, but callback for Nodes can happen with the queue locked).

{noformat}
"jenkins.util.Timer [#3]" id=34 (0x22) state=BLOCKED cpu=91%
    - waiting to lock <0x107c53da> (a hudson.model.Hudson)
     owned by Handling POST /configSubmit : http-8080-5 id=42
    at jenkins.model.Jenkins.save(Jenkins.java:2734)
    at hudson.util.PersistedList.onModified(PersistedList.java:173)
    at hudson.util.PersistedList._onModified(PersistedList.java:181)
    at hudson.util.PersistedList.add(PersistedList.java:72)
    at com.cloudbees.librato.CloudListener.onComplete(CloudListener.java:83)
    at hudson.slaves.NodeProvisioner$2.run(NodeProvisioner.java:223)
    at hudson.model.Queue._withLock(Queue.java:1235)
    at hudson.model.Queue.withLock(Queue.java:1171)
    at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:208)
    at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:57)
    at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:777)
    at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

    Locked synchronizers: count = 3
      - java.util.concurrent.locks.ReentrantLock$NonfairSync@2e7d6057
      - java.util.concurrent.ThreadPoolExecutor$Worker@6d40c3f9
      - java.util.concurrent.locks.ReentrantLock$NonfairSync@640c882b

"Handling POST /configSubmit : http-8080-5" id=42 (0x2a) state=WAITING cpu=96%
    - waiting on <0x2e7d6057> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    - locked <0x2e7d6057> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
     owned by jenkins.util.Timer [#3] id=34
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
    at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
    at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
    at hudson.model.Queue._withLock(Queue.java:1233)
    at hudson.model.Queue.withLock(Queue.java:1171)
    at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:168)
    at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
    at jenkins.model.Jenkins.doConfigSubmit(Jenkins.java:2858)
...
{noformat}

New: A [Slaves nodeProperties | (https://github.com/jenkinsci/jenkins/blob/9c443c8d5bafd63fce574f6d0cf400cd8fe1f124/core/src/main/java/hudson/model/Slave.java#L130] are owned by Jenkins.

So whenever a NodeProperty is changed this causes persistance of the entire jenkins configuration rather than just Nodes which is what actually persists the NodeProperties.

Whilst this is just inefficient - if anyone wants to add a node property to new nodes created (e.g. by implementing a cloud listener) then is it possible to get a deadlock if someone also saves the Jenkins configuration at the same time. (as Jenkins.save() will lock Jenkins and then the queue, but callback for Nodes can happen with the queue locked).

{noformat}
"jenkins.util.Timer [#3]" id=34 (0x22) state=BLOCKED cpu=91%
    - waiting to lock <0x107c53da> (a hudson.model.Hudson)
     owned by Handling POST /configSubmit : http-8080-5 id=42
    at jenkins.model.Jenkins.save(Jenkins.java:2734)
    at hudson.util.PersistedList.onModified(PersistedList.java:173)
    at hudson.util.PersistedList._onModified(PersistedList.java:181)
    at hudson.util.PersistedList.add(PersistedList.java:72)
    at com.cloudbees.librato.CloudListener.onComplete(CloudListener.java:83)
    at hudson.slaves.NodeProvisioner$2.run(NodeProvisioner.java:223)
    at hudson.model.Queue._withLock(Queue.java:1235)
    at hudson.model.Queue.withLock(Queue.java:1171)
    at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:208)
    at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:57)
    at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:777)
    at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

    Locked synchronizers: count = 3
      - java.util.concurrent.locks.ReentrantLock$NonfairSync@2e7d6057
      - java.util.concurrent.ThreadPoolExecutor$Worker@6d40c3f9
      - java.util.concurrent.locks.ReentrantLock$NonfairSync@640c882b

"Handling POST /configSubmit : http-8080-5" id=42 (0x2a) state=WAITING cpu=96%
    - waiting on <0x2e7d6057> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    - locked <0x2e7d6057> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
     owned by jenkins.util.Timer [#3] id=34
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
    at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
    at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
    at hudson.model.Queue._withLock(Queue.java:1233)
    at hudson.model.Queue.withLock(Queue.java:1171)
    at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:168)
    at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
    at jenkins.model.Jenkins.doConfigSubmit(Jenkins.java:2858)
...
{noformat}

James Nord made changes - 2015-08-20 14:28

Summary

Original: Slaves NodeProperties are owneb by Jenkins not Nodes.

New: Slaves NodeProperties are owned by Jenkins not Nodes.

James Nord made changes - 2015-08-20 14:28

Assignee

New: James Nord [ teilo ]

James Nord made changes - 2015-08-20 14:28

Status

Original: Open [ 1 ]

New: In Progress [ 3 ]

James Nord made changes - 2015-08-20 14:34

Remote Link

New: This issue links to "PR#1807 (Web Link)" [ 13121 ]

James Nord made changes - 2015-08-20 14:47

Link

New: This issue is related to JENKINS-30060 [ JENKINS-30060 ]

SCM/JIRA link daemon made changes - 2015-08-24 11:15

Resolution		New: Fixed [ 1 ]
Status	Original: In Progress [ 3 ]	New: Resolved [ 5 ]

James Nord made changes - 2015-08-25 11:23

Labels

Original: deadlock

New: deadlock lts-candidate

Oliver Gondža made changes - 2015-09-10 14:20

Labels

Original: deadlock lts-candidate

New: 1.625-fixed deadlock

Oliver Gondža made changes - 2015-09-10 14:21

Labels

Original: 1.625-fixed deadlock

New: 1.625.1-fixed deadlock

Assignee:: Unassigned

Reporter:: James Nord

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Created:: 2015-08-20 13:52

Updated:: 2020-12-10 13:45

Resolved:: 2015-08-24 11:15

Jenkins

Details

Description

Attachments

Issue Links

Activity

People

Dates