Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55700

deadlock remoting TCP agent connection handler

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Minor Minor
    • remoting
    • Server: Jenkins 2.138.2
      Remoting: 3.25
      Java: 1.8.181
    • Remoting 3.27

      Found one Java-level deadlock:
      =============================
      "TCP agent connection handler #159021 with /10.116.113.11:54776":
        waiting for ownable synchronizer 0x00000002ce4a6770, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
        which is held by "IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/10.116.241.122:8181 remote=10.116.113.11/10.116.113.11:54776]] / Computer.threadPoolForRemoting [#12366]"
      "IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/10.116.241.122:8181 remote=10.116.113.11/10.116.113.11:54776]] / Computer.threadPoolForRemoting [#12366]":
        waiting to lock monitor 0x00007f75cc5900f8 (object 0x00000002ce4de778, a org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer),
        which is held by "TCP agent connection handler #159021 with /10.116.113.11:54776"
      
      Java stack information for the threads listed above:
      ===================================================
      "TCP agent connection handler #159021 with /10.116.113.11:54776":
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000002ce4a6770> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
      	at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.getNextSend(ProtocolStack.java:857)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:686)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:518)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
      	- locked <0x00000002ce4a6800> (a java.lang.Object)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:692)
      	at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.doSend(ConnectionHeadersFilterLayer.java:497)
      	- locked <0x00000002ce4de778> (a org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:692)
      	at org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:157)
      	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.start(ChannelApplicationLayer.java:230)
      	at org.jenkinsci.remoting.protocol.ProtocolStack.init(ProtocolStack.java:201)
      	at org.jenkinsci.remoting.protocol.ProtocolStack.access$700(ProtocolStack.java:106)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Builder.build(ProtocolStack.java:554)
      	at org.jenkinsci.remoting.engine.JnlpProtocol4Handler.handle(JnlpProtocol4Handler.java:153)
      	at jenkins.slaves.JnlpSlaveAgentProtocol4.handle(JnlpSlaveAgentProtocol4.java:205)
      	at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:260)
      "IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/10.116.241.122:8181 remote=10.116.113.11/10.116.113.11:54776]] / Computer.threadPoolForRemoting [#12366]":
      	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvRemoved(FilterLayer.java:133)
      	- waiting to lock <0x00000002ce4de778> (a org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.getNextRecv(ProtocolStack.java:929)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:663)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:369)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669)
      	at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136)
      	at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:160)
      	at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      
      Found 1 deadlock.
       

        1. jenkins.tdump
          594 kB
          Alexander Moiseenko

            jthompson Jeff Thompson
            brainsam Alexander Moiseenko
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: