It is likely caused by TrileadSSH upgrade in 2.58. CC paladox and mc1arke

Here is the log I've got with 2.58 on a Linux slave startup:

executing pre-launch scripts ... [05/02/17 09:01:53] [SSH] Opening SSH connection to kaiku.artelys.lan:22. Key exchange was not finished, connection is closed. java.io.IOException: There was a problem while connecting to kaiku.artelys.lan:22 at com.trilead.ssh2.Connection.connect(Connection.java:834) at com.trilead.ssh2.Connection.connect(Connection.java:703) at com.trilead.ssh2.Connection.connect(Connection.java:617) at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1265) at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:790) at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:785) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Key exchange was not finished, connection is closed. at com.trilead.ssh2.transport.KexManager.getOrWaitForConnectionInfo(KexManager.java:95) at com.trilead.ssh2.transport.TransportManager.getConnectionInfo(TransportManager.java:237) at com.trilead.ssh2.Connection.connect(Connection.java:786) ... 9 more Caused by: java.io.IOException: The server hostkey was not accepted by the verifier callback. at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:543) at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:790) at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:502) ... 1 more Caused by: java.lang.IllegalArgumentException: Unknown hostkey type ssh-rsa at com.trilead.ssh2.KnownHosts.decodeHostKey(KnownHosts.java:607) at com.trilead.ssh2.KnownHosts.verifyHostkey(KnownHosts.java:569) at hudson.plugins.sshslaves.verifiers.KnownHostsFileKeyVerificationStrategy.verify(KnownHostsFileKeyVerificationStrategy.java:62) at hudson.plugins.sshslaves.SSHLauncher$3.verifyServerHostKey(SSHLauncher.java:1279) at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:539) ... 3 more [05/02/17 09:01:53] Launch failed - cleaning up connection [05/02/17 09:01:53] [SSH] Connection closed.

ycollette I'm not sure how you're able to get "Unknown hostkey type ssh-rsa", as ssh-rsa keys are supported by Trilead. Are you able to confirm there are no other spaces or special characters around 'ssh-rsa'? I should really have added speech marks around the field in the error message....

maksonlee Is it OpenSSH you're running on Ubuntu, and have you performed any specific configuration or hardening on it? Your error indicates that there wasn't enough data in the stream to allow proper decryption, so I'd need to try and replicate it in a similar environment (or be given more information about what the SSH host has logged and what's travelling across the wire) to investigate this properly.

It has nothing to do with the openssh configuration.

I used 2.57 without problems.

And now, I switched back to 2.57 and everything works fine now.

yes, we are using openssh, and no, we didn't do any specific configuration.

btw, our ubuntu 14.04 and 16.04 agents are working fine with jenkins 2.58. 

ycollette Trilead was updated in the latest Jenkins release. The version of Trilead being used worked against various test machines (RHEL6, RHEL7, Debian 8), so the failures are likely to be due to a difference on OpenSSH versions or configurations or associated network or security components (firewalls, routers, SELinux etc). The more people that have the issue that can confirm software versions and any specific configurations that are applied, the easier it will be to narrow down a specific set of cases to replicate this with and create a working fix or suggested workaround.

The debian version of the failing slave is 7.11

The sshd_config file is nearly a default one. I only switched on PermitRootLogin

Port 22
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
UsePrivilegeSeparation yes
KeyRegenerationInterval 3600
ServerKeyBits 768
SyslogFacility AUTH
LogLevel INFO
LoginGraceTime 120
PermitRootLogin yes
StrictModes yes
RSAAuthentication yes
PubkeyAuthentication yes
IgnoreRhosts yes
RhostsRSAAuthentication no
HostbasedAuthentication no
PermitEmptyPasswords no
PermitUserEnvironment yes
ChallengeResponseAuthentication no
X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog no
TCPKeepAlive yes
AcceptEnv LANG LC_*
Subsystem sftp internal-sftp
UsePAM yes

I have the same issues when connecting to a mac agent running mavericks (OpenSSH_6.2p2). Rollback to 2.57 fix the trouble.

Same for me. I am using CentOS 7 for both master and agent.

Also getting this today after upgrading to Jenkins 2.58 and connecting to a slave running CentOS 6. sshd logs the following on the slave which looks relevant:

fatal: dh_gen_key: group too small: 1024 (2*need 1024)

Downgrading to Jenkins 2.57 solves the problem.

Hi, do all of you with the problems have java 8 installed?

The java 1.8 was already required for slaves for Jenkins 2.57.

So, yes, I have java 1.8 installed on slave.

Yes, Java 8 on both master and slaves. It seems the problem is related to key exchange early in the SSH connection, before it is even able to try to run commands on the slave.

paladox, refer to the recent thread in the jenkins users mailing list where the same case is described. Especially look at the null pointer exception seen in the Jenkins log during startup.

Thanks. I probably didn't catch this when i was testing as i was using debian jessie.

This is my sshd configuration https://phabricator.wikimedia.org/P5358

Hi, a workaround is to put sha1 in front of all the other macs so that you try to use sha1 first. (untested but found this https://lists.mindrot.org/pipermail/openssh-unix-dev/2014-July/032790.html)

example:

MACs hmac-sha1,hmac-sha1-96,hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com

markewaite your trace back leads to this change https://github.com/jenkinsci/trilead-ssh2/commit/32b0250972bd5d68a6f8247b249c7c2dbee05194#diff-0bdd2ca1b1607b55760d9466dd9868f7 as that is what changes the line your traceback leads too.

This is not only an ubuntu/openssh problem. After upgrade to version 2.58 jenkins cannot start slave ssh agent on solaris host too.

Solaris 10 SSH Information: Sun_SSH_1.1.8, SSH-Protokolle 1.5/2.0, OpenSSL 0x1000114f

Here is the log from Jenkins:

[05/02/17 09:28:18] [SSH] Opening SSH connection to quark:22.
Key exchange was not finished, connection is closed.
java.io.IOException: There was a problem while connecting to quark:22
        at com.trilead.ssh2.Connection.connect(Connection.java:834)
        at com.trilead.ssh2.Connection.connect(Connection.java:703)
        at com.trilead.ssh2.Connection.connect(Connection.java:617)
        at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1265)
        at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:790)
        at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:785)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Key exchange was not finished, connection is closed.
        at com.trilead.ssh2.transport.KexManager.getOrWaitForConnectionInfo(KexManager.java:95)
        at com.trilead.ssh2.transport.TransportManager.getConnectionInfo(TransportManager.java:237)
        at com.trilead.ssh2.Connection.connect(Connection.java:786)
        ... 9 more
Caused by: java.io.IOException: Cannot read full block, EOF reached.
        at com.trilead.ssh2.crypto.cipher.CipherInputStream.getBlock(CipherInputStream.java:81)
        at com.trilead.ssh2.crypto.cipher.CipherInputStream.read(CipherInputStream.java:108)
        at com.trilead.ssh2.transport.TransportConnection.receiveMessage(TransportConnection.java:232)
        at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:706)
        at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:502)
        ... 1 more
[05/02/17 09:28:18] Launch failed - cleaning up connection
[05/02/17 09:28:18] [SSH] Connection closed. 

SSH agent on SLES 11 (OpenSSH_6.6.1p1, OpenSSL 0.9.8j-fips 07 Jan 2009) and SLES 12 (OpenSSH_6.6.1p1, OpenSSL 1.0.1i-fips 6 Aug 2014) hosts works without problems.

Just putting SHA1 in front of the MAC list did not fix it for me (Debian 7 slave). I actually had to remove all other MACs except hmac-sha1 to make it work.

Similar problem happens during connection to a slave on Solaris 11.

SSH Information: Sun_SSH_2.2, SSH protocols 1.5/2.0, OpenSSL 0x1000110f

Similar problem happens during connection to a slave on IBM AIX 7.1 and AIX 7.2.

SSh Informations are for both OS identical

OpenSSH_6.0p1, OpenSSL 1.0.1e 11 Feb 2013

@Dzianis Mazuronak hi, I got that error once when using Debian but it immediately managed to connect over ssh after that. Try clicking connect please.

Same problem.

Windows master, can't connect to agent running on CentOS 7.3. (OpenSSH_6.6.1p1, OpenSSL 1.0.1e-fips)

Broken by simple upgrade to jenkins 2.58.

Same problem.

Master on CentOS Linux release 7.3.1611 - 
Slaves on CentOS release 6.7 (Final)

Both on OpenSSL 1.0.1e-fips 11 Feb 2013

Same problem here with Solaris 11 slaves

Sun_SSH_2.0, SSH protocols 1.5/2.0, OpenSSL 0x100000af

sshd logs show:

May  3 16:03:33 solaris sshd[23376]: [ID 800047 auth.crit] fatal: dh_gen_key: group too small: 1024 (2*need 1024)
May  3 16:24:33 solaris sshd[23382]: [ID 800047 auth.crit] fatal: dh_gen_key: group too small: 1024 (2*need 1024)

paladox Recconect was simply not possible.

lklock and anyone else getting this error

dh_gen_key: group too small: 1024 (2*need 1024)

Hi, could you try https://github.com/jenkinsci/trilead-ssh2/pull/20 please.

You git clone https://github.com/mc1arke/trilead-ssh2.git

then git checkout jenkins-43979-sha512-issues

you then mvn install -Dmaven.test.skip=true

you then git clone https://github.com/jenkinsci/jenkins

then you edit file core/pom.xml

find this

https://github.com/jenkinsci/jenkins/blob/c33e7dc42094568a7d37aaa59f2bf5ab8332dc9a/core/pom.xml#L118 line

change it from build217-jenkins-9 to build217-jenkins-10-SNAPSHOT

then 

mvn install -Dmaven.test.skip=true

then you will find the war under war/target/ (will be jenkins.war in there) copy that to where ever jenkins runs the war on your os. On debian it's /usr/share/jenkins/

dmazuronak oh, i see a connect button when going to http://gerrit-jenkins.wmflabs.org/computer/jenkins-slave-01/

Did you mean it kept failing when you press connect?

Yes, it kept failing.

dmazuronak Could you try https://issues.jenkins-ci.org/browse/JENKINS-43979?focusedCommentId=298057&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-298057 please

Per m1clarke description of what the fix does here https://github.com/jenkinsci/trilead-ssh2/pull/20#issuecomment-298868970 it should hopefully fix your Caused by: java.io.IOException: Cannot read full block, EOF reached. error

We are running into the same issue and fixed it by updating OpenSSH on the slave. See https://askubuntu.com/questions/461494/update-openssh-server-in-12-04

After upgraded to version 2.58, getting this error:

dh_gen_key: group too small: 1024 (2*need 1024)

And this solution works for me https://issues.jenkins-ci.org/browse/JENKINS-43979?focusedCommentId=298057&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-298057 

But I also have to change host key verification strategy from "Known hosts file Verification Strategy" to "Non verifying Verification Strategy" in node settings because of error:

Caused by: java.lang.IllegalArgumentException: Unknown hostkey type ssh-rsa at com.trilead.ssh2.KnownHosts.decodeHostKey(KnownHosts.java:607) at com.trilead.ssh2.KnownHosts.verifyHostkey(KnownHosts.java:569) at hudson.plugins.sshslaves.verifiers.KnownHostsFileKeyVerificationStrategy.verify(KnownHostsFileKeyVerificationStrategy.java:62) at hudson.plugins.sshslaves.SSHLauncher$3.verifyServerHostKey(SSHLauncher.java:1279) at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:539)

paladox yeah it works for me ! gg

Thankyou for trying the fix

Aleksey R Hi, another fix on top of that one is to also try https://github.com/jenkinsci/trilead-ssh2/pull/21 please following the same steps above but on top of git checkout we want to git merge jenkins-42959-key-format-mismatch

Since we want both fixes you git checkout first then do the git merge which will add the fix on top of the other fix.

Hi paladox, fix from jenkins-42959-key-format-mismatch works too. Known hosts verification strategy now working as well.

After looking more into our stacktrace, it seems to look more like the stacktrace posted above by ycollette :

Instead of eventually failing over hostkey type ssh-rsa at the end, as in his stacktrace, it complains about ssh-ed25519. But otherwise an identical trace.

Caused by: java.lang.IllegalArgumentException: Unknown hostkey type ssh-ed25519

This worked flawlessly before. Reverting to jenkins 2.57 simply fixes it.

(Windows master; agent on CentOS 7.3)

Aleksey R Thankyou for testing.

K P Hi your issue sounds like it will be fixed with https://github.com/jenkinsci/trilead-ssh2/pull/21 as another user who had unkown hostley found after applying that pull it started working.

I've tried the https://github.com/jenkinsci/trilead-ssh2/pull/21 suggested by paladox.

Using that snapshot, it seems indeed that ssh-ed25519 is now recognized, but connection now fails with

[05/04/17 14:06:54] [SSH] The SSH key presented by the remote host does not match the key saved in the Known Hosts file against this host. Connections to this host will be denied until the two keys match.
Key exchange was not finished, connection is closed.

The known_hosts file on the jenkins master has a ssh-rsa key for the agent host.
The known_hosts file on the jenkins master has NO ssh-ed25519 key for the agent host.

• Adding the ssh-ed25519 key for that agent resolved resolved the above error, and allows us to start the agent again using the lastest snapshot.
• But then I'm lucky the stack trace I got with jenkins 2.58 pointed me into the right direction by complaining about ssh-ed25519.
• If I were only using the latest snapshot, I'd only gotten the above error message, which makes no mention about ssh-ed25519 at all.
  So the user would be really puzzled by the error, as his known_hosts file DOES contain a key for the agent host, namely its ssh-rsa key. And he be clueless about why his key wouldn't match. As the provided ssh-rsa key is correct, as it has always been...
• I've also tried configuring the verification strategy as "Manually provided key Verification Strategy", thereby specifying the ssh-rsa key.
  No luck. Doesn't work anymore either. Whereas this ssh-rsa key worked in jenkins 2.57.

Yet,

Jenkins 2.57 still connects perfectly well, even without the ssh-ed25519 key in the known_hosts of the master. Instead it still matches using the ssh-rsa key. It also worked perfectly well with a manually provided ssh-rsa key.

• So there's still in incompatibility between this snapshot and jenkins 2.57.
• As I'm now forced to adapt all key configurations for all such hosts - in the known_hosts file. (for some people's setups this might be a huge work, if feasible at all)

When choosing the manual option for those nodes, the jenkins node configuration doesn't even seem to accept ssh-ed25519 key?

• Even if one allows this backward compatibility and advises admins to copy around those additional keys for their agent hosts, the error messages give no clue at all which keys they'd need to copy: ecdsa, ed25519, rsa, ...

And I guess the release notes should at least contain a clear warning that a simple upgrade make break all agent connections, and may require quite some reconfiguration of keys as jenkins would no longer use the ssh-rsa - though correctly configured.

belpk Your issue with getting different key types is due to Trilead now supporting additional key types, and telling the SSH host which types it prefers. As RSA is viewed by Trilead as being less secure than ED25519, Trilead has RSA lower down the supported list than ED25519, which results in the server sending ED25519 keys since it also supports them. RSA used to be first in the key list so was sent by the server, hence it being the one already saved as being trusted. I'll update ssh-slaves to prefer any existing saved key's algorithm over the other (potentially more secure) algorithms it knows about. This will require you to update the ssh-slaves plugin once we release it, which is likely to be independent of the next release of Jenkins core containing the current Trilead fixes.

mc1arke, that seems to be our issue as well. However, I would prefer to use the more secure algorithms if possible. What will I need to do to accept the more secure key our servers are currently sending? It would be a pity if we're stuck to less-secure algorithms because they were once accepted. Will there be an easy way within Jenkins to "refresh" the key, so we get the most secure one and can accept this new one?

pjaytycy I could possibly add a feature at some point in the future to detect if the server offers newer key formats and then alerts the user that keys could be updated. Updating the saved keys is more of a challenge as we'd need to reconnect to SSH to get the updated key, and Jenkins may then only have read access to the known_hosts file so require manual intervention from someone with file-system access on the Jenkins master. I'll need to have a think about the best approach to this - I don't think OpenSSH's clients currently provide such a feature I can replicate the process from - so it's definitely not going to be part of the upcoming releases to fix the issues related to the Trilead upgrade.

Same issue here, except I have RHEL5/6. What do you mean by "applying pull"? Build jenkins.war from source? I think I'd rather wait until the next release.

Code changed in jenkins
User: Michael Clarke
Path:
src/com/trilead/ssh2/DHGexParameters.java
http://jenkins-ci.org/commit/trilead-ssh2/3308477c57120302881caf8458595bc81ce020ef
Log:
JENKINS-43979 Fix OpenSSH 5 dropping connections with SHA512 enabled

Code changed in jenkins
User: Michael Clarke
Path:
src/com/trilead/ssh2/DHGexParameters.java
http://jenkins-ci.org/commit/trilead-ssh2/63b2b8505466c5e90b3d8d76348226d45edfd920
Log:
Merge pull request #20 from mc1arke/jenkins-43979-sha512-issues

JENKINS-43979 Fix OpenSSH dropping connections with SHA512 enabled

Compare: https://github.com/jenkinsci/trilead-ssh2/compare/3be61af6df03...63b2b8505466

Code changed in jenkins
User: Michael Clarke
Path:
core/pom.xml
http://jenkins-ci.org/commit/jenkins/f36917c210d52ce7fc1f4bca71c9f1baf11c4647
Log:
[JENKINS-42959, JENKINS-44046, JENKINS-43979] - Bump Trilead to build217-jenkins-10 (#2872)

JENKINS-42959 Correctly compare key algorithms during key verification
[FIXED JENKINS-44046][FIXED JENKINS-43979] Use a larger default key size to fix issues when using SHA256 MACs during Diffe-Helman key exchange against older versions of OpenSSH

Fixes issues encountered following the recent move to the latest Trilead version. These fix 2 underlying issues:

diffe-helman-sha256 Key Exchange needs at least 2048 bits of data to generate a key against OpenSSH 6.4 and below.
Known hosts comparison incorrectly compares the hostname against the key value meaning key verification always fails when using known hosts files

Hi,

I confirm that applying https://github.com/jenkinsci/trilead-ssh2/pull/20

and https://github.com/jenkinsci/trilead-ssh2/pull/21 fixed the issue as well for my Solaris 11 nodes.

Thanks

Thanks. The fix should be released in jenkins 2.59.

Fixed in 2.59 (to be released within a day or so).

2.59 has been released now.

This has fixed the issue I've seen.  Thanks everyone who was able to determine the problem here!

Version 2.59 solved my communication problems.