Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-30057

Slaves NodeProperties are owned by Jenkins not Nodes.

      A Slaves nodeProperties are owned by Jenkins.

      So whenever a NodeProperty is changed this causes persistance of the entire jenkins configuration rather than just Nodes which is what actually persists the NodeProperties.

      Whilst this is just inefficient - if anyone wants to add a node property to new nodes created (e.g. by implementing a cloud listener) then is it possible to get a deadlock if someone also saves the Jenkins configuration at the same time. (as Jenkins.save() will lock Jenkins and then the queue, but callback for Nodes can happen with the queue locked).

      "jenkins.util.Timer [#3]" id=34 (0x22) state=BLOCKED cpu=91%
          - waiting to lock <0x107c53da> (a hudson.model.Hudson)
           owned by Handling POST /configSubmit : http-8080-5 id=42
          at jenkins.model.Jenkins.save(Jenkins.java:2734)
          at hudson.util.PersistedList.onModified(PersistedList.java:173)
          at hudson.util.PersistedList._onModified(PersistedList.java:181)
          at hudson.util.PersistedList.add(PersistedList.java:72)
          at com.cloudbees.librato.CloudListener.onComplete(CloudListener.java:83)
          at hudson.slaves.NodeProvisioner$2.run(NodeProvisioner.java:223)
          at hudson.model.Queue._withLock(Queue.java:1235)
          at hudson.model.Queue.withLock(Queue.java:1171)
          at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:208)
          at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:57)
          at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:777)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)
      
          Locked synchronizers: count = 3
            - java.util.concurrent.locks.ReentrantLock$NonfairSync@2e7d6057
            - java.util.concurrent.ThreadPoolExecutor$Worker@6d40c3f9
            - java.util.concurrent.locks.ReentrantLock$NonfairSync@640c882b
      
      
      
      "Handling POST /configSubmit : http-8080-5" id=42 (0x2a) state=WAITING cpu=96%
          - waiting on <0x2e7d6057> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
          - locked <0x2e7d6057> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
           owned by jenkins.util.Timer [#3] id=34
          at sun.misc.Unsafe.park(Native Method)
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
          at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
          at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
          at hudson.model.Queue._withLock(Queue.java:1233)
          at hudson.model.Queue.withLock(Queue.java:1171)
          at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:168)
          at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
          at jenkins.model.Jenkins.doConfigSubmit(Jenkins.java:2858)
      ...
      

          [JENKINS-30057] Slaves NodeProperties are owned by Jenkins not Nodes.

          James Nord created issue -
          James Nord made changes -
          Description Original: A [Slaves nodeProperties](https://github.com/jenkinsci/jenkins/blob/9c443c8d5bafd63fce574f6d0cf400cd8fe1f124/core/src/main/java/hudson/model/Slave.java#L130) are owned by Jenkins.

          So whenever a NodeProperty is changed this causes persistance of the entire jenkins configuration rather than just Nodes which is what actually persists the NodeProperties.

          Whilst this is just inefficient - if anyone wants to add a node property to new nodes created (e.g. by implementing a cloud listener) then is it possible to get a deadlock if someone also saves the Jenkins configuration at the same time. (as Jenkins.save() will lock Jenkins and then the queue, but callback for Nodes can happen with the queue locked).

          {noformat}
          "jenkins.util.Timer [#3]" id=34 (0x22) state=BLOCKED cpu=91%
              - waiting to lock <0x107c53da> (a hudson.model.Hudson)
               owned by Handling POST /configSubmit : http-8080-5 id=42
              at jenkins.model.Jenkins.save(Jenkins.java:2734)
              at hudson.util.PersistedList.onModified(PersistedList.java:173)
              at hudson.util.PersistedList._onModified(PersistedList.java:181)
              at hudson.util.PersistedList.add(PersistedList.java:72)
              at com.cloudbees.librato.CloudListener.onComplete(CloudListener.java:83)
              at hudson.slaves.NodeProvisioner$2.run(NodeProvisioner.java:223)
              at hudson.model.Queue._withLock(Queue.java:1235)
              at hudson.model.Queue.withLock(Queue.java:1171)
              at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:208)
              at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:57)
              at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:777)
              at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)

              Locked synchronizers: count = 3
                - java.util.concurrent.locks.ReentrantLock$NonfairSync@2e7d6057
                - java.util.concurrent.ThreadPoolExecutor$Worker@6d40c3f9
                - java.util.concurrent.locks.ReentrantLock$NonfairSync@640c882b



          "Handling POST /configSubmit : http-8080-5" id=42 (0x2a) state=WAITING cpu=96%
              - waiting on <0x2e7d6057> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
              - locked <0x2e7d6057> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
               owned by jenkins.util.Timer [#3] id=34
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
              at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
              at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
              at hudson.model.Queue._withLock(Queue.java:1233)
              at hudson.model.Queue.withLock(Queue.java:1171)
              at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:168)
              at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
              at jenkins.model.Jenkins.doConfigSubmit(Jenkins.java:2858)
          ...
          {noformat}
          New: A [Slaves nodeProperties | (https://github.com/jenkinsci/jenkins/blob/9c443c8d5bafd63fce574f6d0cf400cd8fe1f124/core/src/main/java/hudson/model/Slave.java#L130] are owned by Jenkins.

          So whenever a NodeProperty is changed this causes persistance of the entire jenkins configuration rather than just Nodes which is what actually persists the NodeProperties.

          Whilst this is just inefficient - if anyone wants to add a node property to new nodes created (e.g. by implementing a cloud listener) then is it possible to get a deadlock if someone also saves the Jenkins configuration at the same time. (as Jenkins.save() will lock Jenkins and then the queue, but callback for Nodes can happen with the queue locked).

          {noformat}
          "jenkins.util.Timer [#3]" id=34 (0x22) state=BLOCKED cpu=91%
              - waiting to lock <0x107c53da> (a hudson.model.Hudson)
               owned by Handling POST /configSubmit : http-8080-5 id=42
              at jenkins.model.Jenkins.save(Jenkins.java:2734)
              at hudson.util.PersistedList.onModified(PersistedList.java:173)
              at hudson.util.PersistedList._onModified(PersistedList.java:181)
              at hudson.util.PersistedList.add(PersistedList.java:72)
              at com.cloudbees.librato.CloudListener.onComplete(CloudListener.java:83)
              at hudson.slaves.NodeProvisioner$2.run(NodeProvisioner.java:223)
              at hudson.model.Queue._withLock(Queue.java:1235)
              at hudson.model.Queue.withLock(Queue.java:1171)
              at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:208)
              at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:57)
              at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:777)
              at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)

              Locked synchronizers: count = 3
                - java.util.concurrent.locks.ReentrantLock$NonfairSync@2e7d6057
                - java.util.concurrent.ThreadPoolExecutor$Worker@6d40c3f9
                - java.util.concurrent.locks.ReentrantLock$NonfairSync@640c882b



          "Handling POST /configSubmit : http-8080-5" id=42 (0x2a) state=WAITING cpu=96%
              - waiting on <0x2e7d6057> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
              - locked <0x2e7d6057> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
               owned by jenkins.util.Timer [#3] id=34
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
              at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
              at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
              at hudson.model.Queue._withLock(Queue.java:1233)
              at hudson.model.Queue.withLock(Queue.java:1171)
              at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:168)
              at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1219)
              at jenkins.model.Jenkins.doConfigSubmit(Jenkins.java:2858)
          ...
          {noformat}
          James Nord made changes -
          Summary Original: Slaves NodeProperties are owneb by Jenkins not Nodes. New: Slaves NodeProperties are owned by Jenkins not Nodes.
          James Nord made changes -
          Assignee New: James Nord [ teilo ]
          James Nord made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]
          James Nord made changes -
          Remote Link New: This issue links to "PR#1807 (Web Link)" [ 13121 ]
          James Nord made changes -
          Link New: This issue is related to JENKINS-30060 [ JENKINS-30060 ]

          Code changed in jenkins
          User: James Nord
          Path:
          core/src/main/java/hudson/model/Computer.java
          core/src/main/java/hudson/model/Slave.java
          core/src/main/java/jenkins/model/Jenkins.java
          http://jenkins-ci.org/commit/jenkins/06f690dbb9f491d592d42f5b911ae9d49f6e1290
          Log:
          JENKINS-30057 NodeProperties should be owned by the corresponding Saveable.

          Node objects are owned by the Nodes class not by Jenkins so NodeProperties
          and anything modifiying them needs to be updated. Also modifying Nodes
          requires a lock on the Queue so use this lock not Jenkins to prevent the
          deadlock observed in JENKINS-30057

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: James Nord Path: core/src/main/java/hudson/model/Computer.java core/src/main/java/hudson/model/Slave.java core/src/main/java/jenkins/model/Jenkins.java http://jenkins-ci.org/commit/jenkins/06f690dbb9f491d592d42f5b911ae9d49f6e1290 Log: JENKINS-30057 NodeProperties should be owned by the corresponding Saveable. Node objects are owned by the Nodes class not by Jenkins so NodeProperties and anything modifiying them needs to be updated. Also modifying Nodes requires a lock on the Queue so use this lock not Jenkins to prevent the deadlock observed in JENKINS-30057

          Code changed in jenkins
          User: James Nord
          Path:
          core/src/main/java/hudson/model/Computer.java
          core/src/main/java/hudson/model/Slave.java
          core/src/main/java/jenkins/model/Jenkins.java
          http://jenkins-ci.org/commit/jenkins/5e9492d1229921d0c1701d880650671fddd6cbfd
          Log:
          Merge pull request #1807 from jtnord/JENKINS-30057

          [FIXED JENKINS-30057] NodeProperties should be owned by the corresponding Saveable

          Compare: https://github.com/jenkinsci/jenkins/compare/05a0b6ca0b4e...5e9492d12299

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: James Nord Path: core/src/main/java/hudson/model/Computer.java core/src/main/java/hudson/model/Slave.java core/src/main/java/jenkins/model/Jenkins.java http://jenkins-ci.org/commit/jenkins/5e9492d1229921d0c1701d880650671fddd6cbfd Log: Merge pull request #1807 from jtnord/ JENKINS-30057 [FIXED JENKINS-30057] NodeProperties should be owned by the corresponding Saveable Compare: https://github.com/jenkinsci/jenkins/compare/05a0b6ca0b4e...5e9492d12299
          SCM/JIRA link daemon made changes -
          Resolution New: Fixed [ 1 ]
          Status Original: In Progress [ 3 ] New: Resolved [ 5 ]

            Unassigned Unassigned
            teilo James Nord
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: