Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48611

SSH threads become blocked when trying to close the connection

XMLWordPrintable

      SSH remoting threads become blocked on the following stacktrace:

      "Computer.threadPoolForRemoting [#104]" #1768 daemon prio=5 os_prio=0 tid=0x00007f07e02db800 nid=0x7d46 waiting for monitor entry [0x00007f07c24f5000]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at com.trilead.ssh2.Connection.close(Connection.java:573)
      - waiting to lock <0x0000000594003de0> (a com.trilead.ssh2.Connection)
      at hudson.plugins.sshslaves.SSHLauncher.cleanupConnection(SSHLauncher.java:897)
      at hudson.plugins.sshslaves.SSHLauncher.tearDownConnection(SSHLauncher.java:1445)
      - locked <0x0000000608aa1468> (a hudson.plugins.sshslaves.SSHLauncher)
      at hudson.plugins.sshslaves.SSHLauncher.afterDisconnect(SSHLauncher.java:1371)
      at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:633)
      at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:748)
      
      Locked ownable synchronizers:
      - <0x000000058bc977c8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
      
      "Computer.threadPoolForRemoting [#98]" #1714 daemon prio=5 os_prio=0 tid=0x00007f08002df800 nid=0x7ce4 waiting for monitor entry [0x00007f07c0546000]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at hudson.plugins.sshslaves.SSHLauncher.launch(SSHLauncher.java:799)
      - waiting to lock <0x0000000608aa1468> (a hudson.plugins.sshslaves.SSHLauncher)
      at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:262)
      at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:748)
      

      It appears to be trying to close a SSH connection and that is blocking other threads from closing and new threads from opening a new connection. This ends up creating a large volume of SSH threads which could bring a master down.

            Unassigned Unassigned
            ataylor Alex Taylor
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: