Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53569

Remoting deadlock observed after upgrading to 3.26

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • remoting, swarm-plugin
    • None
    • Server: Jenkins 2.138.1 LTS (Remoting 3.25)
      Client: Swarm Client 3.14 (Remoting 3.26)
    • Remoting 3.27, Jenkins 2.144

      After upgrading my Jenkins master and the Swarm Client to the latest stable versions, I am seeing a new deadlock on the Swarm Client side when trying to connect to the master.

      The relevant output from jstack:

      Found one Java-level deadlock:
      =============================
      "pool-1-thread-3":
        waiting to lock monitor 0x0000000000d12970 (object 0x0000000784a00fc0, a org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer),
        which is held by "Thread-2"
      "Thread-2":
        waiting for ownable synchronizer 0x0000000784a4ac68, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
        which is held by "pool-1-thread-3"
      
      Java stack information for the threads listed above:
      ===================================================
      "pool-1-thread-3":
              at org.jenkinsci.remoting.protocol.FilterLayer.onRecvRemoved(FilterLayer.java:134)
              - waiting to lock <0x0000000784a00fc0> (a org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.getNextRecv(ProtocolStack.java:929)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:663)
              at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:369)
              at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669)
              at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136)
              at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$2200(BIONetworkLayer.java:48)
              at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:283)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
              at hudson.remoting.Engine$1$$Lambda$5/613009671.run(Unknown Source)
              at java.lang.Thread.run(Thread.java:748)
      "Thread-2":
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x0000000784a4ac68> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
              at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.remove(ProtocolStack.java:755)
              at org.jenkinsci.remoting.protocol.FilterLayer.completed(FilterLayer.java:108)
              - locked <0x0000000784a00fc0> (a org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer)
              at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.complete(ConnectionHeadersFilterLayer.java:363)
              at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.doSend(ConnectionHeadersFilterLayer.java:499)
              - locked <0x0000000784a00fc0> (a org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:692)
              at org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:157)
              at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.start(ChannelApplicationLayer.java:230)
              at org.jenkinsci.remoting.protocol.ProtocolStack.init(ProtocolStack.java:201)
              at org.jenkinsci.remoting.protocol.ProtocolStack.access$700(ProtocolStack.java:106)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Builder.build(ProtocolStack.java:554)
              at org.jenkinsci.remoting.engine.JnlpProtocol4Handler.connect(JnlpProtocol4Handler.java:179)
              at org.jenkinsci.remoting.engine.JnlpProtocolHandler.connect(JnlpProtocolHandler.java:157)
              at hudson.remoting.Engine.innerRun(Engine.java:573)
              at hudson.remoting.Engine.run(Engine.java:474)
      
      Found 1 deadlock.
      

      After encountering this deadlock, the Swarm Client never finishes connecting to the master. The master is unable to use the Swarm Client as a node when it reaches this hung state.

            jthompson Jeff Thompson
            basil Basil Crow
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: