[JENKINS-42959] Failed known_hosts verification for SSH agent - Jenkins Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Critical
Resolution: Fixed
Component/s: ssh-slaves-plugin
Labels:
- regression
Environment:
Versions:

Jenkins: 2.51
SSH Agent Plugin: 1.14
SSH Credentials plugin: 1.13
SSH Slaves Plugin : 1.15

Ubuntu 14.04, 16.04

Similar Issues:

Show

Description

SSH agent isn't launched after the latest update, complaining about missing records in the known hosts file.

Nevertheless, the records do exist. I've tried to connect manually with ssh, everything was fine, ssh reports that it has found the host in known_hosts.

Probable reason is the hashed host name in known_hosts.

Here is the log from the Jenkins slave launch page:

[03/21/17 11:40:34] [SSH] Opening SSH connection to xxx.xxx.xxx.xxx:22
[03/21/17 11:40:34] [SSH] WARNING: No entry currently exists in the Known Hosts file for this host.      Connections will be denied until this new host and its associated key is added to the Known Hosts file.
     Key exchange was not finished, connection is closed.
     java.io.IOException: There was a problem while connecting to xxx.xxx.xxx.xxx:22
	at com.trilead.ssh2.Connection.connect(Connection.java:818)
	at com.trilead.ssh2.Connection.connect(Connection.java:687)
	at com.trilead.ssh2.Connection.connect(Connection.java:601)
	at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1265)
	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:790)
	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:785)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
     Caused by: java.io.IOException: Key exchange was not finished, connection is closed.
	at com.trilead.ssh2.transport.KexManager.getOrWaitForConnectionInfo(KexManager.java:93)
	at com.trilead.ssh2.transport.TransportManager.getConnectionInfo(TransportManager.java:230)
	at com.trilead.ssh2.Connection.connect(Connection.java:770)
	... 9 more
     Caused by: java.io.IOException: The server hostkey was not accepted by the verifier callback
	at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
	at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
	at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
	... 1 more
     [03/21/17 11:40:34] Launch failed - cleaning up connection
     [03/21/17 11:40:34] [SSH] Connection closed.

And this is the debug output from ssh, showing that it has found a record in the second line of known_hosts.

  $ sudo -u jenkins -g jenkins ssh -v jenkins@xxx.xxx.xxx.xxx
    OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
    debug1: Reading configuration data /etc/ssh/ssh_config
    debug1: /etc/ssh/ssh_config line 19: Applying options for *
    debug1: Connecting to xxx.xxx.xxx.xxx [xxx.xxx.xxx.xxx] port 22.
    debug1: Connection established.
    debug1: identity file /var/lib/jenkins/.ssh/id_rsa type 1
...
    debug1: sending SSH2_MSG_KEX_ECDH_INIT
    debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
    debug1: Server host key: ECDSA xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
    debug1: Host 'xxx.xxx.xxx.xxx.xxx' is known and matches the ECDSA host key.
    debug1: Found key in /var/lib/jenkins/.ssh/known_hosts:2
    debug1: ssh_ecdsa_verify: signature correct
...
    Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-67-generic x86_64)

Additional logs from Jenkins system log:

Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
    kex_algo=diffie-hellman-group14-sha1
Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
    server_host_key_algo=ssh-rsa
    Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
    enc_algo_client_to_server=aes256-ctr
    Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
    enc_algo_server_to_client=aes256-ctr
    Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
    mac_algo_client_to_server=hmac-sha1
    Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
    mac_algo_server_to_client=hmac-sha1
    Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
    comp_algo_client_to_server=none
    Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
    comp_algo_server_to_client=none
    Mar 21, 2017 12:06:08 PM FINE com.trilead.ssh2.transport.TransportManager
    Receive thread: error in receiveLoop
    java.io.IOException: The server hostkey was not accepted by the verifier callback
            at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
            at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
            at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
            at java.lang.Thread.run(Thread.java:745)

    Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.TransportManager
    Receive thread: back from receiveLoop
    Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
    kex_algo=diffie-hellman-group14-sha1
    Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
    server_host_key_algo=ssh-rsa
    Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
    enc_algo_client_to_server=aes256-ctr
    Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
    enc_algo_server_to_client=aes256-ctr
    Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
    mac_algo_client_to_server=hmac-sha1
    Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
    mac_algo_server_to_client=hmac-sha1
    Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
    comp_algo_client_to_server=none
    Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
    comp_algo_server_to_client=none
    Mar 21, 2017 12:06:10 PM FINE com.trilead.ssh2.transport.TransportManager
    Receive thread: error in receiveLoop
    java.io.IOException: The server hostkey was not accepted by the verifier callback
            at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
            at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
            at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
            at java.lang.Thread.run(Thread.java:745)

    Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.TransportManager
    Receive thread: back from receiveLoop

File /var/lib/jenkins/.ssh/known_hosts contains strings, looking like being base64-encoded, delimited by '|'. Here is the sample.

|1|DAg  ...   o... 1ll9wI=| ...  ....  tIrM= ecdsa-sha2-nistp256 xxxxxx..... bmlzdHAyNTYAAAAIbm................. .................xxxxxxxxxxxoKEHF3Vr0q685jI2+6vWjvAAG4lz5Ckujy9k=

Github issue

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

J42959.png
43 kB
2017-03-22 07:48

Issue Links

is blocked by

JENKINS-44832 SSH Slaves plugin fails the host verification of slaves with an IllegalArgumentException

Resolved

is duplicated by

JENKINS-44803 Master 2.60.1-rc1 (2.58-2.64) ssh agents won't connect with "Manually provided key Verification Strategy"

Closed

relates to

JENKINS-44830 NullPointerException when upgrading to ssh-slaves 1.18

Resolved

links to

Add support for Ed25519 host and user keys

Add support for host and user ECDSA keys

Prefer algorithms currently supported from known_hosts entries

Specify preferred host keys during connect

(2 links to)

Activity

Ascending order - Click to sort in descending order

View 62 older comments

Daniel Beck added a comment - 2017-07-16 09:39

olenads Please provide more information, such as the exact error message (with stack trace), as well as what the known_hosts entry looks like.

Daniel Beck added a comment - 2017-07-16 09:39 olenads Please provide more information, such as the exact error message (with stack trace), as well as what the known_hosts entry looks like.

M Chon added a comment - 2017-08-15 01:47

I have these versions:

Jenkins 2.60.2

SSH Slaves Plugin 1.20

and am still getting this error:

Warning: no key algorithms provided; ~~JENKINS-42959~~ disabled [08/14/17 18:47:10] [SSH] Opening SSH connection to xxxxxx:22. [08/14/17 18:47:10] [SSH] WARNING: No entry currently exists in the Known Hosts file for this host. Connections will be denied until this new host and its associated key is added to the Known Hosts file.

M Chon added a comment - 2017-08-15 01:47 I have these versions: Jenkins 2.60.2 SSH Slaves Plugin 1.20 and am still getting this error: Warning: no key algorithms provided; JENKINS-42959 disabled [08/14/17 18:47:10] [SSH] Opening SSH connection to xxxxxx:22. [08/14/17 18:47:10] [SSH] WARNING: No entry currently exists in the Known Hosts file for this host. Connections will be denied until this new host and its associated key is added to the Known Hosts file.

Matthew Webber added a comment - 2017-08-15 15:39

This message means exactly what it says.

No entry currently exists in the Known Hosts file for this host. Connections will be denied until this new host and its associated key is added to the Known Hosts file

In your agent configuration, for "Host Key Verification Strategy" you probably have "Known hosts file verification strategy". The help text for that says:

Checks the known_hosts file (~/.ssh/known_hosts) for the user Jenkins is executing under, to see if an entry exists that matches the current connection.

This method does not make any updates to the Known Hosts file, instead using the file as a read-only source and expecting someone with suitable access to the appropriate user account on the Jenkins master to update the file as required, potentially using the ssh hostname command to initiate a connection and update the file appropriately.

So that's what you need to do. If you need more help, ask on the mailing list, not here, since it's not a bug.

Matthew Webber added a comment - 2017-08-15 15:39 This message means exactly what it says. No entry currently exists in the Known Hosts file for this host. Connections will be denied until this new host and its associated key is added to the Known Hosts file In your agent configuration, for "Host Key Verification Strategy" you probably have "Known hosts file verification strategy". The help text for that says: Checks the known_hosts file (~/.ssh/known_hosts) for the user Jenkins is executing under, to see if an entry exists that matches the current connection. This method does not make any updates to the Known Hosts file, instead using the file as a read-only source and expecting someone with suitable access to the appropriate user account on the Jenkins master to update the file as required, potentially using the ssh hostname command to initiate a connection and update the file appropriately. So that's what you need to do. If you need more help, ask on the mailing list, not here, since it's not a bug.

M Chon added a comment - 2017-08-17 17:50 - edited

Yes, and I've been asking our elusive Sys Admin to please ssh from one host to the other in order to populate the known_hosts. This never used to be required by the plugin in the past (or if it was, I was not aware), but I guess it is now.

Meanwhile, I noticed there is a dropdown menu under Node configuration, and I selected "Manually provided key Verification Strategy", and then I pasted the contents of id_rsa.pub into the "SSH Key" textbox, but got this error:

[08/17/17 10:48:58] [SSH] WARNING: The SSH key for this host does not match the key required in the connection configuration. Connections will be denied until until the host key matches the configuration key. Key exchange was not finished, connection is closed.

I then tried "Non verifying Verification Strategy", and got this error:

[08/17/17 10:46:28] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:e85865a0-e6eb-4de3-a77e-b0fbbc9ec8c4/method:publickey) [08/17/17 10:46:28] [SSH] Authentication failed. Authentication failed.

So it seems like the "Manually provided key" method will work for me, but there is an issue with key mismatch. So, I'll follow up with our sys admin.

M Chon added a comment - 2017-08-17 17:50 - edited Yes, and I've been asking our elusive Sys Admin to please ssh from one host to the other in order to populate the known_hosts. This never used to be required by the plugin in the past (or if it was, I was not aware), but I guess it is now. Meanwhile, I noticed there is a dropdown menu under Node configuration, and I selected "Manually provided key Verification Strategy", and then I pasted the contents of id_rsa.pub into the "SSH Key" textbox, but got this error: [08/17/17 10:48:58] [SSH] WARNING: The SSH key for this host does not match the key required in the connection configuration. Connections will be denied until until the host key matches the configuration key. Key exchange was not finished, connection is closed. I then tried "Non verifying Verification Strategy", and got this error: [08/17/17 10:46:28] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:e85865a0-e6eb-4de3-a77e-b0fbbc9ec8c4/method:publickey) [08/17/17 10:46:28] [SSH] Authentication failed. Authentication failed. So it seems like the "Manually provided key" method will work for me, but there is an issue with key mismatch. So, I'll follow up with our sys admin.

Michael Clarke added a comment - 2017-08-17 18:32

mcsf Which id_rsa.pub was it you copied in to the configuration? Your personal one (~/.ssh/id_rsa.pub), or the one for the host's public key (/etc/sshd/id_rsa.pub or similar)? It needs to be the latter for this to work.

Your final failure indicates that your private key (used for authentication) does not match any public key saved on the host, which no amount of key verification is going to fix.

Using Non verifying Verification Strategy is strongly discouraged, unless you can verify the identity of the host through another means. Have you considered using the "Manually Trusted Key Verification Strategy" to retrieve the remote host's key and then for you to verify the key against the contents of rsa_id.pub from the remote host? This should get round any delays with getting System Admins to perform actions on your behalf.

Michael Clarke added a comment - 2017-08-17 18:32 mcsf Which id_rsa.pub was it you copied in to the configuration? Your personal one (~/.ssh/id_rsa.pub), or the one for the host's public key (/etc/sshd/id_rsa.pub or similar)? It needs to be the latter for this to work. Your final failure indicates that your private key (used for authentication) does not match any public key saved on the host, which no amount of key verification is going to fix. Using Non verifying Verification Strategy is strongly discouraged , unless you can verify the identity of the host through another means. Have you considered using the "Manually Trusted Key Verification Strategy" to retrieve the remote host's key and then for you to verify the key against the contents of rsa_id.pub from the remote host? This should get round any delays with getting System Admins to perform actions on your behalf.

Jenkins

Failed known_hosts verification for SSH agent

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates