VMware here too.

My agents are baremetal:

<slave>

<name>####</name>
<description>####</description>
<remoteFS>####</remoteFS>
<numExecutors>2</numExecutors>
<mode>NORMAL</mode>
<retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
<launcher class="hudson.plugins.sshslaves.SSHLauncher" plugin="ssh-slaves@1.31.2">
<host>####</host>
<port>###</port>
<credentialsId>####</credentialsId>
<launchTimeoutSeconds>60</launchTimeoutSeconds>
<maxNumRetries>10</maxNumRetries>
<retryWaitTime>15</retryWaitTime>
<sshHostKeyVerificationStrategy class="hudson.plugins.sshslaves.verifiers.KnownHostsFileKeyVerificationStrategy"/>
<tcpNoDelay>true</tcpNoDelay>
</launcher>
<label>CentOS7</label>
<nodeProperties/>
</slave>

I am also running into this issue this morning after updating everything yesterday.

Jenkins version: 2.249.1

ssh build agents plugin: 1.31.2

Jenkins Master is Windows Server 2016 all slaves are ubuntu 18.04. All slaves are running as VMs in Hyper-V and failing to connect. 

ssh versions:

node1: OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017
node2: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016
node3: OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017
node4: OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017

All are running the same version of Java:

openjdk version "1.8.0_265"
OpenJDK Runtime Environment (build 1.8.0_265-8u265-b01-0ubuntu2~18.04-b01)
OpenJDK 64-Bit Server VM (build 25.265-b01, mixed mode)

I start thinking it is related to a change in how the timeout is managed in the trilead-ssh2 library, it fixes an issue to avoid an infinite wait but I think it breaks something. I will try to replicate it with VMs on the cloud and low timeouts, If I replicate it we have a winner.

I'd like to "me too" this ticket!  My careless clicking "upgrade" was going fine until ssh-slaves pulled in and we ran into this.

My master is an old linux box with 1.8 and the build agents are newer linux with java 1.8

I followed the advice here and downloaded 1.0.10 from https://updates.jenkins.io/download/plugins/trilead-api/ into my plugins/ directory (rename the hpi to jpi) and restarted jenkins master to get connected again

Same here, downgraded fixed it.

I've added a note to the release notes, to warn people that the update can cause this issue on some systems.

I've experienced this too. Again master is on Windows and most nodes are Linux VMs in Azure. Some nodes are also Windows (SSH), others AIX and IBM i. All exhibit this behaviour.

It also knocked out connections to Git hosted in Azure DevOps (cloud).

Reverting to the previous version and restarting got things back up and running again.

I did not replicate the exact issue because finally it connects, but I see a weird timeout. In this case, I have used an e2.micro Ubuntu 16.04 VM in GCP, I will continue from this point to test a trilead-ssh2 library without the timeout change.

Sep 30, 2020 6:20:54 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to /home/inifc/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
connect timed out
SSH Connection failed with IOException: "connect timed out", retrying in 15 seconds. There are 7 more retries left.
Evacuated stdout
connect timed out
SSH Connection failed with IOException: "connect timed out", retrying in 15 seconds. There are 4 more retries left.
Agent successfully connected and online

ok, after reverting the change the GCP agent works as expected, what I wonder is why because the change seems fair it only add a 120s timeout to the Object.wait methods to avoid an infinite wait https://github.com/jenkinsci/trilead-ssh2/pull/50

Lost a day to this (downgrade resolves it):

Jenkins Server: Windows Server 2016 (VM)

2 MacOSX Agents (1 Catalina, 1 High Sierra, both Mac Mini bare metal)

Jenkins sat in an endless loop never connecting and on the MacOSX side the logs were filled with something to the effect of "sshd service exited with abnormal code 255" for each attempt.

Additionally, the naming of this plugin is unfortunate so I had no idea it was related to SSH until I found this ticket as a matter of lucky googling.

The release notes has a section known issues this ticket is linked there for a few days

No one is going to find release notes. Why not just release a .12 reverting the changes in .11 until things can be sorted out?

jglick because I cannot replicate the issue consistently and on my tests everything works. I will release an incremental this weekend with the possible fix, I’ll need someone that has the issue to check if the issue is resolved or not.

I have a test environment I could try it on if you can let me know how and how to revert.

ifernandezcalvo I'm seeing an issue with 1.0.11 in my Docker environment that uses JDK 11 on Alpine and a combination of GCP, other cloud, and local agents. I'm happy to try the 1.0.12 release as well. For the moment, I've reverted my installation to 1.0.10 so that I can continue testing Jenkins 2.249.2-rc.

finally, I did not get the incremental configured in time for the trilead-ssh2 lib, but it does not matter, I have uploaded the snapshot from revert-44-patch-2 Artifacts to the Artifactory(build-217-jenkins-25-SNAPSHOT), then I've bumped the version locally and generate a binary, you can install trilead-api.hpi manually from the plugins management page in the advanced tab, from there you submit the plugin and it will be installed, after restarting the instance the new version should be installed. If the change reverted is the cause of the issue everything would work, if not, to revert the change you have to go to the plugins management page in the installed tab, search for the trilead-api plugin, and downgrade to the previous version.

I'll start testing now.

That pre-release allowed my 30 agents in various configurations to connect reliably with both JDK 8 and JDK 11 tests. The JDK 8 testing is running with Jenkins 2.249.1. The JDK 11 testing is running with Jenkins 2.249.2-rc.

SSH agents were connected from a Docker image of 2.249.1 and 2.249.2-rc including:

• CentOS 7 on Google Cloud
• CentOS 8 on Google Cloud
• Debian 9 on Google Cloud
• Debian 10 on Google Cloud
• Debian 10 on local network
• Debian testing on local network
• FreeBSD 12 on local network
• IBM PowerPC 64le on an IBM server
• IBM SystemZ on an IBM server
• OpenBSD 6.7 on local network
• Raspbian 10 on local network
• Ubuntu 18 on Google Cloud
• Ubuntu 20 on Google Cloud
• Windows 10 using Windows OpenSSH on local network

thanks a lot Mark

1.0.12-SNAPSHOT (private-4f699fb0-inifc) didn't work for me connecting to one Win2019 server.

nsleigh can you provide more details about the failure on your Windows 2019 server? Were you connecting through Windows OpenSSH or another SSH server? Does it work with 1.0.10? Does it fail with 1.0.11?

markewaite it is exactly the same as my original report (I reported this initially). It is Windows OpenSSH to Windows OpenSSH. v1.0.10 works and 1.0.11/1.0.12 fail in the same way.

I have reverted to v1.0.10 now and it is working again.

[10/05/20 13:13:06] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
Key exchange was not finished, connection is closed.
SSH Connection failed with IOException: "Key exchange was not finished, connection is closed.", retrying in 15 seconds. There are 1 more retries left.
[10/05/20 13:13:07] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
Key exchange was not finished, connection is closed.
[10/05/20 13:13:21] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
Key exchange was not finished, connection is closed.
ERROR: Connection is not established!
java.lang.IllegalStateException: Connection is not established!
at com.trilead.ssh2.Connection.getRemainingAuthMethods(Connection.java:988)
at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.getRemainingAuthMethods(TrileadSSHPublicKeyAuthenticator.java:88)
at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.canAuthenticate(TrileadSSHPublicKeyAuthenticator.java:80)
at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.newInstance(SSHAuthenticator.java:218)
at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.newInstance(SSHAuthenticator.java:171)
at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:863)
at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:435)
at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:422)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

We are facing issues connecting EC2( amazonlinux) Jenkins executors where we can able successfully ssh from master to executors but not able to connect in Jenkinns UI after we updated to LTS 2.249.1 and latest version of trilead api plugin.

Do we know that it affects linux too? as they only mentioned about windows so far

markewaite sounds like you are able to reproduce a regression; have you tried bisecting https://github.com/jenkinsci/trilead-ssh2/compare/trilead-ssh2-build-217-jenkins-21...trilead-ssh2-build-217-jenkins-25 ? Looks like there were a bunch of significant changes.

yrsuryahttps://issues.jenkins-ci.org/browse/JENKINS-63790?focusedCommentId=398405&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-398405  multiple people have reported the same issue on linux

yrsurya my testing with trilead-api-plugin 1.0.11 with Docker images running Jenkins 2.249.1 and 2.249.2-rc on Linux showed that I was not reliably getting connections to all of my agents. The problem affects Linux as well as Windows as far as I can tell.

jglick I have not attempted to bisect the changes from trilead-api-plugin 1.0.10 to 1.0.11.

we are not seeing this in dev-jenkins were we have replica of prod but only difference dev running in EKS( kubenetes) agents able to connect using SSH Keys. Issue with prod Jenkins(running in EC2) 

yrsurya some of my agents using trilead-api-plugin 1.0.11 connect successfully while others do not. I didn't see any pattern that I recognized.

All my agents connect successfully using the trilead-api-plugin 1.0.12 pre-release that is referenced by ifernandezcalvo. nsleigh reports that his Windows Server 2019 agents do not connect reliably with either trilead-api-plugin 1.0.11 or trilead-api-plugin 1.0.12 pre-release. I don't know what's different between his configuration and mine, since my Windows 10.0.1909 agents connect reliably with trilead-api-plugin 1.0.12 pre-release and do not all connect reliably with trilead-api-plugin 1.0.11.

We just upgraded to ver 1.0.11 and got the same error.  Had to rollback to 1.0.10 and it connects to agent again.

Same result as Larry Charbonneau (and others. Updated to v1.0.11 and could not connect to agents via ssh. Ours are Linux clients. Downgrading to v1.0.8 resolves the issues.

INFO: Waiting for SSH to come up. Sleeping 5.
Oct 07, 2020 1:34:05 PM hudson.plugins.ec2.EC2Cloud
INFO: No SSH key verification (ssh-ed25519 76:0e:b5:a3:f9:04:g3:a6:d6:61:70:1b:df:bf:05:5c) for connections to EC2 (ec2-slave) - deploy-slave (...)
Oct 07, 2020 1:34:05 PM hudson.plugins.ec2.EC2Cloud
INFO: Failed to connect via ssh: There was a problem while connecting to ...
    

    
  

  

If someone else with a test environment could test the pre-release attached to this Jira, we can confirm if the fix works and we would release a version with the fix.

I tried the pre-release in attachment and it did not solve the agent-connect issues for us:

SSHLauncher{host='s204.ourcompany.nl', port=22, credentialsId='c48df730-9351-4574-9895-4ab8f483eca7', jvmOptions='-Djava.io.tmpdir=/jenkins/tmp', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=60, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.KnownHostsFileKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
[10/08/20 10:31:14] [SSH] Opening SSH connection to s204.ourcompany.nl:22.
Searching for s204.ourcompany.nl in /opt/jenkins/.ssh/known_hosts
Searching for s204.ourcompany.nl:22 in /opt/jenkins/.ssh/known_hosts
[10/08/20 10:31:14] [SSH] SSH host key matches key in Known Hosts file. Connection will be allowed.
Key exchange was not finished, connection is closed.
SSH Connection failed with IOException: "Key exchange was not finished, connection is closed.", retrying in 15 seconds. There are 10 more retries left.
Searching for s204.ourcompany.nl in /opt/jenkins/.ssh/known_hosts
Searching for s204.ourcompany.nl:22 in /opt/jenkins/.ssh/known_hosts
[10/08/20 10:31:30] [SSH] SSH host key matches key in Known Hosts file. Connection will be allowed.
Key exchange was not finished, connection is closed.
SSH Connection failed with IOException: "Key exchange was not finished, connection is closed.", retrying in 15 seconds. There are 9 more retries left.
Searching for s204.ourcompany.nl in /opt/jenkins/.ssh/known_hosts
Searching for s204.ourcompany.nl:22 in /opt/jenkins/.ssh/known_hosts
[10/08/20 10:31:45] [SSH] SSH host key matches key in Known Hosts file. Connection will be allowed.
Key exchange was not finished, connection is closed.
SSH Connection failed with IOException: "Key exchange was not finished, connection is closed.", retrying in 15 seconds. There are 8 more retries left.
Searching for s204.ourcompany.nl in /opt/jenkins/.ssh/known_hosts
Searching for s204.ourcompany.nl:22 in /opt/jenkins/.ssh/known_hosts
[10/08/20 10:32:01] [SSH] SSH host key matches key in Known Hosts file. Connection will be allowed.
Key exchange was not finished, connection is closed.
SSH Connection failed with IOException: "Key exchange was not finished, connection is closed.", retrying in 15 seconds. There are 7 more retries left. 

Running with v1.0.10 works fine.

Private key is a 2048 bit RSA key, unencrypted

Both master and agent are on-prem CentOS servers:

• master is running CentOS Linux release 7.6.1810 (Core) / OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
• agent is running CentOS Linux release 7.8.2003 (Core) / OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017

Jenkins v2.258

SSH Build Agents plugin v1.31.2

Same issue with OSX slave... master is a Windows 10 x64.  Rolling back solves the issue.

Good news, I have an environment that replicates the issue, I've configured the EC2 plugin to provision t2.medium instances of Ubuntu 20.04 with java 8 installed

With trilead-api 1.0.10 it works

 Connection from <IP> port 64966 on 172.20.1.252 port 22 rdomain ""
 debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
 debug1: Remote protocol version 2.0, remote software version TrileadSSH2Java_213
 debug1: no match: TrileadSSH2Java_213
 debug1: permanently_set_uid: 109/65534 [preauth
 debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth
 debug1: SSH2_MSG_KEXINIT sent [preauth
 User child is on pid 1419
 debug1: do_cleanup
 debug1: PAM: cleanup
 debug1: PAM: closing session
 pam_unix(sshd:session): session closed for user ubuntu
 debug1: PAM: deleting credentials
 debug1: temporarily_use_uid: 1000/1000 (e=0/0)
 debug1: restore_uid: 0/0
 debug1: audit_event: unhandled event 12
 debug1: main_sigchld_handler: Child exited
 debug1: SSH2_MSG_KEXINIT received [preauth
 debug1: kex: algorithm: diffie-hellman-group-exchange-sha256 [preauth
 debug1: kex: host key algorithm: ssh-ed25519 [preauth
 debug1: kex: client->server cipher: aes256-ctr MAC: hmac-sha2-512 compression: none [preauth
 debug1: kex: server->client cipher: aes256-ctr MAC: hmac-sha2-512 compression: none [preauth
 debug1: expecting SSH2_MSG_KEX_DH_GEX_REQUEST [preauth
 debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received [preauth
 debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent [preauth
 debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT [preauth
 debug1: rekey out after 4294967296 blocks [preauth
 debug1: SSH2_MSG_NEWKEYS sent [preauth
 debug1: expecting SSH2_MSG_NEWKEYS [preauth
 debug1: SSH2_MSG_NEWKEYS received [preauth
 debug1: rekey in after 4294967296 blocks [preauth
 debug1: KEX done [preauth
 debug1: userauth-request for user ubuntu service ssh-connection method none [preauth
 debug1: attempt 0 failures 0 [preauth
 debug1: PAM: initializing for "ubuntu"
 debug1: PAM: setting PAM_RHOST to "<IP>"
 debug1: PAM: setting PAM_TTY to "ssh"
 debug1: userauth-request for user ubuntu service ssh-connection method publickey [preauth
 debug1: attempt 1 failures 0 [preauth
 debug1: temporarily_use_uid: 1000/1000 (e=0/0)
 debug1: trying public key file /home/ubuntu/.ssh/authorized_keys
 debug1: fd 5 clearing O_NONBLOCK
 debug1: /home/ubuntu/.ssh/authorized_keys:1: matching key found: RSA SHA256:XXX
 debug1: /home/ubuntu/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
 Accepted key RSA SHA256:XXX found at /home/ubuntu/.ssh/authorized_keys:1
 debug1: restore_uid: 0/0
 debug1: auth_activate_options: setting new authentication options
 debug1: do_pam_account: called
 Accepted publickey for ubuntu from <IP> port 64966 ssh2: RSA SHA256:XXX
 debug1: monitor_child_preauth: ubuntu has been authenticated by privileged process
 debug1: auth_activate_options: setting new authentication options [preauth
 debug1: monitor_read_log: child log fd closed
 debug1: PAM: establishing credentials
 pam_unix(sshd:session): session opened for user ubuntu by (uid=0)

And it fails with trilead-api 1.0.11

 Connection from <IP> port 64888 on 172.20.1.252 port 22 rdomain ""
 debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
 debug1: Remote protocol version 2.0, remote software version TrileadSSH2Java_213
 debug1: no match: TrileadSSH2Java_213
 debug1: permanently_set_uid: 109/65534 [preauth
 debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth
 debug1: SSH2_MSG_KEXINIT sent [preauth
 debug1: SSH2_MSG_KEXINIT received [preauth
 debug1: kex: algorithm: diffie-hellman-group-exchange-sha256 [preauth
 debug1: kex: host key algorithm: ssh-ed25519 [preauth
 debug1: kex: client->server cipher: aes256-ctr MAC: hmac-sha2-512 compression: none [preauth
 debug1: kex: server->client cipher: aes256-ctr MAC: hmac-sha2-512 compression: none [preauth
 debug1: expecting SSH2_MSG_KEX_DH_GEX_REQUEST [preauth
 debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received [preauth
 debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent [preauth
 debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT [preauth
 debug1: rekey out after 4294967296 blocks [preauth
 debug1: SSH2_MSG_NEWKEYS sent [preauth
 debug1: expecting SSH2_MSG_NEWKEYS [preauth
 Connection closed by <IP> port 64888 [preauth
 debug1: do_cleanup [preauth
 debug1: monitor_read_log: child log fd closed
 debug1: do_cleanup
 debug1: Killing privsep child 1109
 debug1: audit_event: unhandled event 12
 debug1: main_sigchld_handler: Child exited
 debug1: Forked child 1110.
 debug1: Set /proc/self/oom_score_adj to 0
 debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
 debug1: inetd sockets after dupping: 4, 4

after taking a look at the logs for some reason in some environments the SSH2_MSG_NEWKEYS message is not sent from Jenkins, so the problem is in the Key negotiation. The pre-release attached revert a change related to the timeouts that are not related at all, the failure is in one of the PRs related to the new support for new algorithms.
My plan this weekend is to start with the version 1.0.10, and add the changes one by one testing the result with this environment, once I found the PR that causes the issue I will take a look at what can be the cause.

Facing the same issue for Master on Window Server NT and slave on the Window Server 2016. In my case also it works with 1.0.10 version

""" 
[10/09/20 04:28:19] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. Key exchange was not finished, connection is closed. SSH Connection failed with IOException: "Key exchange was not finished, connection is closed.", retrying in 15 seconds. There are 3 more retries left. Searching for xxx.xx.xx.xx in C:\Users\INF_Jenkins_Service\.ssh\known_hosts Searching for xxx.xx.xx.xx:22 in C:\Users\xxxxxxxxx\.ssh\known_hosts [10/09/20 04:28:22] [SSH] SSH host key matches key in Known Hosts file. Connection will be allowed. Key exchange was not finished, connection is closed. SSH Connection failed with IOException: "Key exchange was not finished, connection is closed.", retrying in 15 seconds. There are 1 more retries left. ERROR: Connection is not established! java.lang.IllegalStateException: Connection is not established! at com.trilead.ssh2.Connection.getRemainingAuthMethods(Connection.java:988) at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPasswordAuthenticator.canAuthenticate(TrileadSSHPasswordAuthenticator.java:82) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.newInstance(SSHAuthenticator.java:218) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.newInstance(SSHAuthenticator.java:171) at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:863) at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:435) at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:422) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) [10/09/20 04:28:24] Launch failed - cleaning up connection [10/09/20 04:28:24] [SSH] Connection closed. [10/09/20 04:28:35] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. Key exchange was not finished, connection is closed. SSH Connection failed with IOException: "Key exchange was not finished, connection is closed.", retrying in 15 seconds. There are 2 more retries left. Searching for xxx.xx.xx.xx in C:\Users\xxxxxxxxxxxxxxxx\.ssh\known_hosts Searching for xxx.xx.xx.xx:22 in C:\Users\xxxxxxxxxxxxxxxx\.ssh\known_hosts [10/09/20 04:28:38] [SSH] SSH host key matches key in Known Hosts file. Connection will be allowed. Key exchange was not finished, connection is closed. [10/09/20 04:28:50] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. Key exchange was not finished, connection is closed. SSH Connection failed with IOException: "Key exchange was not finished, connection is closed.", retrying in 15 seconds. There are 1 more retries left. ERROR: Connection is not established! java.lang.IllegalStateException: Connection is not established! at com.trilead.ssh2.Connection.getRemainingAuthMethods(Connection.java:988) at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPasswordAuthenticator.canAuthenticate(TrileadSSHPasswordAuthenticator.java:82) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.newInstance(SSHAuthenticator.java:218) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.newInstance(SSHAuthenticator.java:171) at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:863) at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:435) at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:422) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) [10/09/20 04:28:53] Launch failed - cleaning up connection [10/09/20 04:28:53] [SSH] Connection closed. [10/09/20 04:29:06] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. Key exchange was not finished, connection is closed. ERROR: Connection is not established! java.lang.IllegalStateException: Connection is not established! at com.trilead.ssh2.Connection.getRemainingAuthMethods(Connection.java:988) at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPasswordAuthenticator.canAuthenticate(TrileadSSHPasswordAuthenticator.java:82) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.newInstance(SSHAuthenticator.java:218) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.newInstance(SSHAuthenticator.java:171) at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:863) at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:435) at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:422) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) [10/09/20 04:29:21] Launch failed - cleaning up connection [10/09/20 04:29:21] [SSH] Connection closed.

"""

I was hitting an intermittent issue on the EC2 plugin, for some reason, the root fs was not set sometimes. So I am not able to replicate the issue on the following systems (https://github.com/kuisathaverat/jenkins-issues/tree/master/JENKINS-63790):

Jenkins 2.249.2, SSH build agents 1.31.2, Trilead API 1.10.11
Jenkins 2.257, SSH build agents 1.31.2, Trilead API 1.10.11
Jenkins 2.258, SSH build agents 1.31.2, Trilead API 1.10.11

Agents
GCP Ubuntu 20.04 TLS - e2-micro, e2-small, and e2-medium
AWS Ubuntu 20.04 TLS - t2.micro, t2.small, and t2.medium
Docker Ubuntu 20.04 TLS
Docker Ubuntu 18.04 TLS
Docker Ubuntu 16.04 TLS
Docker Ubuntu 14.04 TLS
Docker Debian 9
Docker Debian 10

Authentication method
user+password
key DSA
key RSA
key RSA-256
key RSA-512
key ED25519
key EC
encrypted key DES-EDE3-CBC
encrypted key DES-CBC
encrypted key AES-128-CBC
encrypted key AES-192-CBC
encrypted key AES-256-CBC

JDK
8 and 11

OpenSSH versions
OpenSSH_6.6.1p1 , OpenSSL 1.0.1f 6 Jan 2014
OpenSSH_7.2p2 , OpenSSL 1.0.2g 1 Mar 2016
OpenSSH_7.6p1 , OpenSSL 1.0.2n 7 Dec 2017
OpenSSH_8.2p1 , OpenSSL 1.1.1f 31 Mar 2020

It is something related to the key exchange, Could someone attach the /etc/ssh/sshd_config file of one of those agents?

I have obtained one sshd configuration that has the issue, I've tested it on my test environment without luck so I do not think is related to the sshd configuration.

ChallengeResponseAuthentication no
LogLevel DEBUG2
UsePAM yes
X11Forwarding yes
PrintMotd yes
AcceptEnv LANG LC_*
PermitRootLogin no
PasswordAuthentication yes
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
ClientAliveInterval 600
ClientAliveCountMax 4
Subsystem       sftp    /usr/lib/openssh/sftp-server

After that, I've tested different JDK versions on the Jenkins instances I have shown in the comments here (1.8.0_242, 1.8.0_261, 1.8.0_265) it does not seem related too.

This stacktrace is from https://issues.jenkins-ci.org/browse/JENKINS-63829 that is the same issue but in the git plugin

using credential my_git_key
Cloning the remote Git repository
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: org.eclipse.jgit.api.errors.TransportException: ssh://git@git.intra.home:7999/~myuser/pipelinetest.git: Failed to connect
	at org.jenkinsci.plugins.gitclient.JGitAPIImpl$4.execute(JGitAPIImpl.java:1490)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:375)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to Ubuntu 16.04 64Bit
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:1001)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
		at com.sun.proxy.$Proxy84.execute(Unknown Source)
		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1219)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1297)
		at hudson.scm.SCM.checkout(SCM.java:505)
		at hudson.model.AbstractProject.checkout(AbstractProject.java:1206)
		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
		at hudson.model.Run.execute(Run.java:1894)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:428)
Caused by: org.eclipse.jgit.api.errors.TransportException: ssh://git@git.intra.home:7999/~myuser/pipelinetest.git: Failed to connect
	at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:222)
	at org.jenkinsci.plugins.gitclient.JGitAPIImpl$4.execute(JGitAPIImpl.java:1482)
	... 10 more
Caused by: org.eclipse.jgit.errors.TransportException: ssh://git@git.intra.home:7999/~myuser/pipelinetest.git: Failed to connect
	at org.jenkinsci.plugins.gitclient.trilead.TrileadSessionFactory.getSession(TrileadSessionFactory.java:52)
	at org.eclipse.jgit.transport.SshTransport.getSession(SshTransport.java:107)
	at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.<init>(TransportGitSsh.java:254)
	at org.eclipse.jgit.transport.TransportGitSsh.openFetch(TransportGitSsh.java:144)
	at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:105)
	at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:91)
	at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1260)
	at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:211)
	... 11 more
Caused by: java.io.IOException: There was a problem while connecting to git.intra.home:7999
	at com.trilead.ssh2.Connection.connect(Connection.java:781)
	at com.trilead.ssh2.Connection.connect(Connection.java:638)
	at com.trilead.ssh2.Connection.connect(Connection.java:578)
	at org.jenkinsci.plugins.gitclient.trilead.TrileadSessionFactory.getSession(TrileadSessionFactory.java:29)
	... 18 more
Caused by: java.io.IOException: Key exchange was not finished, connection is closed.
	at com.trilead.ssh2.transport.KexManager.getOrWaitForConnectionInfo(KexManager.java:95)
	at com.trilead.ssh2.transport.TransportManager.getConnectionInfo(TransportManager.java:237)
	at com.trilead.ssh2.Connection.connect(Connection.java:732)
	... 21 more
Caused by: java.io.IOException: Fatal error during MAC startup!
	at com.trilead.ssh2.transport.KexManager.finishKex(KexManager.java:298)
	at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:571)
	at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:790)
	at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:502)
	... 1 more
ERROR: Error cloning remote repo 'origin'

this point me to this code

		try
		{
			cbc = BlockCipherFactory.createCipher(kxs.np.enc_algo_client_to_server, true, km.enc_key_client_to_server,
					km.initial_iv_client_to_server);

			mac = new MessageMac(kxs.np.mac_algo_client_to_server, km.integrity_key_client_to_server);

		}
		catch (IllegalArgumentException e1)
		{
			throw new IOException("Fatal error during MAC startup!");
		}

finally this one, I guess JreCipherWrapper.getInstance is throwing an exception because the algorithm is not found, what remembers me https://issues.jenkins-ci.org/browse/JENKINS-63601 that was a regression caused by https://github.com/jenkinsci/trilead-ssh2/pull/45, I am going to make tests for all the algorithm to find the culprit.

	public static BlockCipher createCipher(String type, boolean encrypt, byte[] key, byte[] iv)
	{
		CipherEntry ce = getEntry(type);
		BlockCipher bc = JreCipherWrapper.getInstance(ce.algorithm, new IvParameterSpec(iv));
		bc.init(encrypt, key);
		return bc;
	}

more tests added https://github.com/jenkinsci/trilead-ssh2/pull/55 all ciphers pass the test, on the test it compares the old trilead implementations with the new JDK implementation.

Can someone add a logger (Manage Jenkins/system log) for the package com.trilead.ssh2.transport on level FINER?

then disconnect and connect an agent with the issue and make a screen capture fo the logger output, something like this

I want to know the exact combination that fails.

Working scenario (1.0.10):

Oct 12, 2020 7:02:54 PM FINER com.trilead.ssh2.transport.KexManagerkex_algo=diffie-hellman-group-exchange-sha256
Oct 12, 2020 7:02:54 PM FINER com.trilead.ssh2.transport.KexManagerserver_host_key_algo=ecdsa-sha2-nistp256
Oct 12, 2020 7:02:54 PM FINER com.trilead.ssh2.transport.KexManagerenc_algo_client_to_server=aes256-ctr
Oct 12, 2020 7:02:54 PM FINER com.trilead.ssh2.transport.KexManagerenc_algo_server_to_client=aes256-ctr
Oct 12, 2020 7:02:54 PM FINER com.trilead.ssh2.transport.KexManagermac_algo_client_to_server=hmac-sha2-512
Oct 12, 2020 7:02:54 PM FINER com.trilead.ssh2.transport.KexManagermac_algo_server_to_client=hmac-sha2-512
Oct 12, 2020 7:02:54 PM FINER com.trilead.ssh2.transport.KexManagercomp_algo_client_to_server=none
Oct 12, 2020 7:02:54 PM FINER com.trilead.ssh2.transport.KexManagercomp_algo_server_to_client=none 

Failing scenario: (1.0.11):

Oct 12, 2020 6:59:09 PM FINER com.trilead.ssh2.transport.KexManagerkex_algo=diffie-hellman-group-exchange-sha256
Oct 12, 2020 6:59:09 PM FINER com.trilead.ssh2.transport.KexManagerserver_host_key_algo=ecdsa-sha2-nistp256
Oct 12, 2020 6:59:09 PM FINER com.trilead.ssh2.transport.KexManagerenc_algo_client_to_server=aes256-ctr
Oct 12, 2020 6:59:09 PM FINER com.trilead.ssh2.transport.KexManagerenc_algo_server_to_client=aes256-ctr
Oct 12, 2020 6:59:09 PM FINER com.trilead.ssh2.transport.KexManagermac_algo_client_to_server=hmac-sha2-512
Oct 12, 2020 6:59:09 PM FINER com.trilead.ssh2.transport.KexManagermac_algo_server_to_client=hmac-sha2-512
Oct 12, 2020 6:59:09 PM FINER com.trilead.ssh2.transport.KexManagercomp_algo_client_to_server=none
Oct 12, 2020 6:59:09 PM FINER com.trilead.ssh2.transport.KexManagercomp_algo_server_to_client=none
Oct 12, 2020 6:59:10 PM FINE com.trilead.ssh2.transport.TransportManagerReceive thread: error in receiveLoop
java.io.IOException: Fatal error during MAC startup!
	at com.trilead.ssh2.transport.KexManager.finishKex(KexManager.java:298)
	at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:571)
	at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:790)
	at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:502)
	at java.lang.Thread.run(Thread.java:748)

Oct 12, 2020 6:59:10 PM FINER com.trilead.ssh2.transport.TransportManagerReceive thread: back from receiveLoop

I configured the same settings, in my case the agent connects (BRRRRRR!!!), Could I ask you to do one more thing? In the PRs https://github.com/jenkinsci/trilead-ssh2/pull/56 and https://github.com/jenkinsci/trilead-api-plugin/pull/18, I have made a change to add the exception that causes the issue to the exception that is launched at that point, this should show more info about the real issue, Could you install the incremental at https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/trilead-api/1.0.12-rc44.41a409e0b8c2/ restart the Jenkins, disconnect and connect the agent, and finally check the logger?

Oct 12, 2020 5:25:53 PM FINER com.trilead.ssh2.transport.TransportManager
Receive thread: back from receiveLoop
Oct 12, 2020 5:25:55 PM FINER com.trilead.ssh2.transport.KexManager
kex_algo=diffie-hellman-group-exchange-sha256
Oct 12, 2020 5:25:55 PM FINER com.trilead.ssh2.transport.KexManager
server_host_key_algo=ecdsa-sha2-nistp256
Oct 12, 2020 5:25:55 PM FINER com.trilead.ssh2.transport.KexManager
enc_algo_client_to_server=aes256-ctr
Oct 12, 2020 5:25:55 PM FINER com.trilead.ssh2.transport.KexManager
enc_algo_server_to_client=aes256-ctr
Oct 12, 2020 5:25:55 PM FINER com.trilead.ssh2.transport.KexManager
mac_algo_client_to_server=hmac-sha2-512
Oct 12, 2020 5:25:55 PM FINER com.trilead.ssh2.transport.KexManager
mac_algo_server_to_client=hmac-sha2-512
Oct 12, 2020 5:25:55 PM FINER com.trilead.ssh2.transport.KexManager
comp_algo_client_to_server=none
Oct 12, 2020 5:25:55 PM FINER com.trilead.ssh2.transport.KexManager
comp_algo_server_to_client=none
Oct 12, 2020 5:25:56 PM FINER com.trilead.ssh2.transport.TransportManager

Illegal key size; what size keys are you using? I wonder if the old ciphers were more flexible in which key sizes they allow.

Private key for our jenkins user used to connect is a 2048 bit RSA key, unencrypted.

I could generate another type of key to see if it changes anything...

I have generated a new incremental with the ciphers changes reverted, it should fix the issue
https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/trilead-api/1.0.12-rc45.30c196a4f01a/
https://github.com/jenkinsci/trilead-ssh2/pull/57
https://github.com/jenkinsci/trilead-api-plugin/pull/18

The issue I think is not related to the key used by the client because it fails when trying to initialize the `aes256-ctr` cipher used for encrypting the SSH channel, the key used there is negotiated between client and server and should have a 256bits, so does not make sense that has another length, should be an error in the migrated code but I did not find it.

A 2048-bit RSA key is super common and shouldn't be the problem. If you're using RSA keys, though, I wonder if the other RSA/SHA-2 patch is the actual cause behind this issue and not the cipher class normalization.

I can confirm that installing https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/trilead-api/1.0.12-rc45.30c196a4f01a/ solves the issue.

trilead-api-1.0.12-rc45.30c196a4f01a worked for me - Win2019 server and node.

ifernandezcalvo Can you please publish a new release reverting the regression while you investigate what happened?

trilead-api-1.0.12-rc45.30c196a4f01a worked for me too
Windows Server 2012 R2
jre 1.8.0

I can confirm that https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/trilead-api/1.0.12-rc45.30c196a4f01a fixes issue on Debian 7/9 (native)

I just released trilead-api-1.0.12 with the fix, I will keep this issue open to try to understand what causes the issue in your environments. So I have some questions :

• Do you pass any Java property to the Jenkins command line? Which ones?
• Which locale do you have on those agents? you can see it by running `locale` with the user you connect.

I confirm that trilead-api-1.0.12 does work for me.

Java properties etc.:

-Xrs -Xmx1g -Xms1g -Dhudson.model.Run.ArtifactList.listCutoff=40 -Dhudson.model.Run.ArtifactList.treeCutoff=100 -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -Djavax.net.ssl.trustStore=%JENKINS_HOME%\.keystore\cacerts -Djavax.net.ssl.trustStorePassword=hugesecret -Dhudson.model.DirectoryBrowserSupport.CSP= -Dhudson.tasks.MailSender.SEND_TO_USERS_WITHOUT_READ=true -Dhudson.tasks.MailSender.SEND_TO_UNKNOWN_USERS=true -jar "%BASE%\jenkins.war" --sessionTimeout=1440 --sessionEviction=43200 --httpPort=-1 --httpsPort=443 --httpsKeyStore="%JENKINS_HOME%\mykeystore.jks" --httpsKeyStorePassword="bigsecret" --webroot="%BASE%\war"

Locales: per node type (uname)

Some nodes are running Windows so I've not got the equivalent information for those.

Linux (Linux hostname 4.18.0-193.19.1.el8_2.x86_64 #1 SMP Mon Sep 14 14:37:00 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux):

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

IBM i (OS400 hostname 2 7):

LANG=en_GB
LC_COLLATE="en_GB"
LC_CTYPE="en_GB"
LC_MONETARY="en_GB"
LC_NUMERIC="en_GB"
LC_TIME="en_GB"
LC_MESSAGES="en_GB"
LC_ALL=

IBM AIX (AIX hostname 1 7):

LANG=en_US
LC_COLLATE="en_US"
LC_CTYPE="en_US"
LC_MONETARY="en_US"
LC_NUMERIC="en_US"
LC_TIME="en_US"
LC_MESSAGES="en_US"
LC_ALL=

Solaris (SunOS hostname 5.11 11.1 sun4v sparc sun4v):

LANG=C
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_ALL=

Jenkins commandline:

java -Dcom.sun.akuma.Daemon=daemonized -Djava.awt.headless=true -Djava.io.tmpdir=/opt/jenkins/tmp -DJENKINS_HOME=/opt/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=8080 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20 

Locale CentOS master:

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL= 

Locale CentOS agent:

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
 

After evaluating all the work we have to make to maintain a dead SSHD library, we are thinking that it is better to migrate everything to a well-maintained SSHD library. The Jenkins core uses an old version of Apache Mina sshd, this library is well maintained, so we have opened an Epic issue to migrate to that library everything https://issues.jenkins-ci.org/browse/JENKINS-64104