Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50984

SSHLauncher/Fingerprint Thread Locking Stopping Dynamic Slave Launch

XMLWordPrintable

      Curious if someone can help me unpack this.  We recently upgraded Jenkins.  We use the Docker-plugin to dynamically provision slaves and we're now running into a situation where the slaves do not properly finish provisioning (The SSH connection is never established). When taking a thread dump there is a very large number of Blocked threads on the SSHLauncher teardown and Fingerprinting for some reason, here's the dumps:

      "Computer.threadPoolForRemoting [#1020]" daemon prio=5 BLOCKED
	hudson.plugins.sshslaves.SSHLauncher.tearDownConnection(SSHLauncher.java:1407)
	hudson.plugins.sshslaves.SSHLauncher.afterDisconnect(SSHLauncher.java:1403)
	com.nirima.jenkins.plugins.docker.launcher.DockerComputerLauncher.afterDisconnect(DockerComputerLauncher.java:71)
	hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:665)
	jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	java.lang.Thread.run(Thread.java:748)

"Computer.threadPoolForRemoting [#1019]" daemon prio=5 BLOCKED
	hudson.model.Fingerprint.save(Fingerprint.java:1238)
	hudson.BulkChange.commit(BulkChange.java:98)
	com.cloudbees.plugins.credentials.CredentialsProvider.trackAll(CredentialsProvider.java:1533)
	com.cloudbees.plugins.credentials.CredentialsProvider.track(CredentialsProvider.java:1478)
	hudson.plugins.sshslaves.SSHLauncher.launch(SSHLauncher.java:866)
	com.nirima.jenkins.plugins.docker.launcher.DockerComputerLauncher.launch(DockerComputerLauncher.java:66)
	hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:288)
	jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	java.lang.Thread.run(Thread.java:748)

      The thread dump has many of these (when things get bad it gets to 100's).  We're currently planning a Docker-plugin upgrade and moving away from the SSH Launcher but I'm looking for ideas as to why this may be happening.

          [JENKINS-50984] SSHLauncher/Fingerprint Thread Locking Stopping Dynamic Slave Launch

          Maxfield Stewart created issue -
          Maxfield Stewart made changes -
          Component/s New: credentials-plugin [ 16523 ]
          Environment Original: Jenkins v2.1112
          Docker Plugin 0.15
          Latest SSH Slaves Plugin           		New: Jenkins v2.1112
          Docker Plugin 0.15
          SSH Slaves Plugin 1.26
          Ivan Fernandez Calvo made changes -
          Link New: This issue duplicates JENKINS-49235 [ JENKINS-49235 ]
          Ivan Fernandez Calvo made changes -
          Assignee Original: Nicolas De Loof [ ndeloof ] New: Ivan Fernandez Calvo [ ifernandezcalvo ]
          Resolution New: Duplicate [ 3 ]
          Status Original: Open [ 1 ] New: Closed [ 6 ]

            ifernandezcalvo Ivan Fernandez Calvo
            maxfields2000 Maxfield Stewart
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: