• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • ssh-slaves-plugin
    • Versions:

      Jenkins: 2.51
      SSH Agent Plugin: 1.14
      SSH Credentials plugin: 1.13
      SSH Slaves Plugin : 1.15

      Ubuntu 14.04, 16.04

      SSH agent isn't launched after the latest update, complaining about missing records in the known hosts file.

      Nevertheless, the records do exist. I've tried to connect manually with ssh, everything was fine, ssh reports that it has found the host in known_hosts.

      Probable reason is the hashed host name in known_hosts.

      Here is the log from the Jenkins slave launch page:

      [03/21/17 11:40:34] [SSH] Opening SSH connection to xxx.xxx.xxx.xxx:22
      [03/21/17 11:40:34] [SSH] WARNING: No entry currently exists in the Known Hosts file for this host.      Connections will be denied until this new host and its associated key is added to the Known Hosts file.
           Key exchange was not finished, connection is closed.
           java.io.IOException: There was a problem while connecting to xxx.xxx.xxx.xxx:22
      	at com.trilead.ssh2.Connection.connect(Connection.java:818)
      	at com.trilead.ssh2.Connection.connect(Connection.java:687)
      	at com.trilead.ssh2.Connection.connect(Connection.java:601)
      	at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1265)
      	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:790)
      	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:785)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
           Caused by: java.io.IOException: Key exchange was not finished, connection is closed.
      	at com.trilead.ssh2.transport.KexManager.getOrWaitForConnectionInfo(KexManager.java:93)
      	at com.trilead.ssh2.transport.TransportManager.getConnectionInfo(TransportManager.java:230)
      	at com.trilead.ssh2.Connection.connect(Connection.java:770)
      	... 9 more
           Caused by: java.io.IOException: The server hostkey was not accepted by the verifier callback
      	at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
      	at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
      	at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
      	... 1 more
           [03/21/17 11:40:34] Launch failed - cleaning up connection
           [03/21/17 11:40:34] [SSH] Connection closed.
      
      

      And this is the debug output from ssh, showing that it has found a record in the second line of known_hosts.

        $ sudo -u jenkins -g jenkins ssh -v jenkins@xxx.xxx.xxx.xxx
          OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
          debug1: Reading configuration data /etc/ssh/ssh_config
          debug1: /etc/ssh/ssh_config line 19: Applying options for *
          debug1: Connecting to xxx.xxx.xxx.xxx [xxx.xxx.xxx.xxx] port 22.
          debug1: Connection established.
          debug1: identity file /var/lib/jenkins/.ssh/id_rsa type 1
      ...
          debug1: sending SSH2_MSG_KEX_ECDH_INIT
          debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
          debug1: Server host key: ECDSA xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
          debug1: Host 'xxx.xxx.xxx.xxx.xxx' is known and matches the ECDSA host key.
          debug1: Found key in /var/lib/jenkins/.ssh/known_hosts:2
          debug1: ssh_ecdsa_verify: signature correct
      ...
          Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-67-generic x86_64)
      

      Additional logs from Jenkins system log:

      Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          kex_algo=diffie-hellman-group14-sha1
      Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          server_host_key_algo=ssh-rsa
          Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          enc_algo_client_to_server=aes256-ctr
          Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          enc_algo_server_to_client=aes256-ctr
          Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          mac_algo_client_to_server=hmac-sha1
          Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          mac_algo_server_to_client=hmac-sha1
          Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          comp_algo_client_to_server=none
          Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.KexManager
          comp_algo_server_to_client=none
          Mar 21, 2017 12:06:08 PM FINE com.trilead.ssh2.transport.TransportManager
          Receive thread: error in receiveLoop
          java.io.IOException: The server hostkey was not accepted by the verifier callback
                  at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
                  at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
                  at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
                  at java.lang.Thread.run(Thread.java:745)
      
          Mar 21, 2017 12:06:08 PM FINER com.trilead.ssh2.transport.TransportManager
          Receive thread: back from receiveLoop
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          kex_algo=diffie-hellman-group14-sha1
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          server_host_key_algo=ssh-rsa
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          enc_algo_client_to_server=aes256-ctr
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          enc_algo_server_to_client=aes256-ctr
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          mac_algo_client_to_server=hmac-sha1
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          mac_algo_server_to_client=hmac-sha1
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          comp_algo_client_to_server=none
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.KexManager
          comp_algo_server_to_client=none
          Mar 21, 2017 12:06:10 PM FINE com.trilead.ssh2.transport.TransportManager
          Receive thread: error in receiveLoop
          java.io.IOException: The server hostkey was not accepted by the verifier callback
                  at com.trilead.ssh2.transport.KexManager.handleMessage(KexManager.java:591)
                  at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:777)
                  at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
                  at java.lang.Thread.run(Thread.java:745)
      
          Mar 21, 2017 12:06:10 PM FINER com.trilead.ssh2.transport.TransportManager
          Receive thread: back from receiveLoop
      

      File /var/lib/jenkins/.ssh/known_hosts contains strings, looking like being base64-encoded, delimited by '|'. Here is the sample.

      |1|DAg  ...   o... 1ll9wI=| ...  ....  tIrM= ecdsa-sha2-nistp256 xxxxxx..... bmlzdHAyNTYAAAAIbm................. .................xxxxxxxxxxxoKEHF3Vr0q685jI2+6vWjvAAG4lz5Ckujy9k=
      

      Github issue

          [JENKINS-42959] Failed known_hosts verification for SSH agent

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          pom.xml
          src/main/java/hudson/plugins/sshslaves/SSHLauncher.java
          src/test/java/hudson/plugins/sshslaves/SSHLauncherTest.java
          http://jenkins-ci.org/commit/ssh-slaves-plugin/367bfdfa1d297f88a840e429b20acba0a34e0032
          Log:
          Merge pull request #61 from jglick/NPE-JENKINS-44830

          JENKINS-44830 NPE after upgrading caused by fix of JENKINS-42959

          Compare: https://github.com/jenkinsci/ssh-slaves-plugin/compare/ce18ed6aa6b4...367bfdfa1d29

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: pom.xml src/main/java/hudson/plugins/sshslaves/SSHLauncher.java src/test/java/hudson/plugins/sshslaves/SSHLauncherTest.java http://jenkins-ci.org/commit/ssh-slaves-plugin/367bfdfa1d297f88a840e429b20acba0a34e0032 Log: Merge pull request #61 from jglick/NPE- JENKINS-44830 JENKINS-44830 NPE after upgrading caused by fix of JENKINS-42959 Compare: https://github.com/jenkinsci/ssh-slaves-plugin/compare/ce18ed6aa6b4...367bfdfa1d29

          It's fixed for me now - tested on Jenkins 2.66, SSH Slaves plugin 1.20

          Matthew Webber added a comment - It's fixed for me now - tested on Jenkins 2.66, SSH Slaves plugin 1.20

          The problem seems to be back for me Jenkins 2.69, SSH Slaves plugin 1.20

          Olena Gaydarenko added a comment - The problem seems to be back for me Jenkins 2.69, SSH Slaves plugin 1.20

          I am having the same problem on Jenkins 2.69

          Olena Gaydarenko added a comment - I am having the same problem on Jenkins 2.69

          Having just upgraded from 2.68 to 2.69, with SSH Slaves plugin 1.20, it's still WORKING for me.

          Matthew Webber added a comment - Having just upgraded from 2.68 to 2.69, with SSH Slaves plugin 1.20, it's still WORKING for me.

          Daniel Beck added a comment -

          olenads Please provide more information, such as the exact error message (with stack trace), as well as what the known_hosts entry looks like.

          Daniel Beck added a comment - olenads Please provide more information, such as the exact error message (with stack trace), as well as what the known_hosts entry looks like.

          M Chon added a comment -

          I have these versions:

          Jenkins 2.60.2

          SSH Slaves Plugin 1.20

          and am still getting this error:

           

          Warning: no key algorithms provided; JENKINS-42959 disabled [08/14/17 18:47:10] [SSH] Opening SSH connection to xxxxxx:22. [08/14/17 18:47:10] [SSH] WARNING: No entry currently exists in the Known Hosts file for this host. Connections will be denied until this new host and its associated key is added to the Known Hosts file.

          M Chon added a comment - I have these versions: Jenkins 2.60.2 SSH Slaves Plugin 1.20 and am still getting this error:   Warning: no key algorithms provided; JENKINS-42959 disabled [08/14/17 18:47:10] [SSH] Opening SSH connection to xxxxxx:22. [08/14/17 18:47:10] [SSH] WARNING: No entry currently exists in the Known Hosts file for this host. Connections will be denied until this new host and its associated key is added to the Known Hosts file.

          This message means exactly what it says.

          No entry currently exists in the Known Hosts file for this host. Connections will be denied until this new host and its associated key is added to the Known Hosts file

          In your agent configuration, for "Host Key Verification Strategy" you probably have "Known hosts file verification strategy". The help text for that says:

          Checks the known_hosts file (~/.ssh/known_hosts) for the user Jenkins is executing under, to see if an entry exists that matches the current connection.

          This method does not make any updates to the Known Hosts file, instead using the file as a read-only source and expecting someone with suitable access to the appropriate user account on the Jenkins master to update the file as required, potentially using the ssh hostname command to initiate a connection and update the file appropriately.

           So that's what you need to do. If you need more help, ask on the mailing list, not here, since it's not a bug.

          Matthew Webber added a comment - This message means exactly what it says. No entry currently exists in the Known Hosts file for this host. Connections will be denied until this new host and its associated key is added to the Known Hosts file In your agent configuration, for "Host Key Verification Strategy" you probably have "Known hosts file verification strategy". The help text for that says: Checks the known_hosts file (~/.ssh/known_hosts) for the user Jenkins is executing under, to see if an entry exists that matches the current connection. This method does not make any updates to the Known Hosts file, instead using the file as a read-only source and expecting someone with suitable access to the appropriate user account on the Jenkins master to update the file as required, potentially using the ssh hostname command to initiate a connection and update the file appropriately.  So that's what you need to do. If you need more help, ask on the mailing list, not here, since it's not a bug.

          M Chon added a comment - - edited

          Yes, and I've been asking our elusive Sys Admin to please ssh from one host to the other in order to populate the known_hosts. This never used to be required by the plugin in the past (or if it was, I was not aware), but I guess it is now.

           

          Meanwhile, I noticed there is a dropdown menu under Node configuration, and I selected "Manually provided key Verification Strategy", and then I pasted the contents of id_rsa.pub into the "SSH Key" textbox, but got this error:

           

          [08/17/17 10:48:58] [SSH] WARNING: The SSH key for this host does not match the key required in the connection configuration. Connections will be denied until until the host key matches the configuration key. Key exchange was not finished, connection is closed.

          I then tried "Non verifying Verification Strategy", and got this error:

          [08/17/17 10:46:28] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:e85865a0-e6eb-4de3-a77e-b0fbbc9ec8c4/method:publickey) [08/17/17 10:46:28] [SSH] Authentication failed. Authentication failed.

           

          So it seems like the "Manually provided key" method will work for me, but there is an issue with key mismatch. So, I'll follow up with our sys admin.

          M Chon added a comment - - edited Yes, and I've been asking our elusive Sys Admin to please ssh from one host to the other in order to populate the known_hosts. This never used to be required by the plugin in the past (or if it was, I was not aware), but I guess it is now.   Meanwhile, I noticed there is a dropdown menu under Node configuration, and I selected "Manually provided key Verification Strategy", and then I pasted the contents of id_rsa.pub into the "SSH Key" textbox, but got this error:   [08/17/17 10:48:58] [SSH] WARNING: The SSH key for this host does not match the key required in the connection configuration. Connections will be denied until until the host key matches the configuration key. Key exchange was not finished, connection is closed. I then tried "Non verifying Verification Strategy", and got this error: [08/17/17 10:46:28] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:e85865a0-e6eb-4de3-a77e-b0fbbc9ec8c4/method:publickey) [08/17/17 10:46:28] [SSH] Authentication failed. Authentication failed.   So it seems like the "Manually provided key" method will work for me, but there is an issue with key mismatch. So, I'll follow up with our sys admin.

          mcsf Which id_rsa.pub was it you copied in to the configuration? Your personal one (~/.ssh/id_rsa.pub), or the one for the host's public key (/etc/sshd/id_rsa.pub or similar)? It needs to be the latter for this to work.

          Your final failure indicates that your private key (used for authentication) does not match any public key saved on the host, which no amount of key verification is going to fix.

          Using Non verifying Verification Strategy is strongly discouraged, unless you can verify the identity of the host through another means. Have you considered using the "Manually Trusted Key Verification Strategy" to retrieve the remote host's key and then for you to verify the key against the contents of rsa_id.pub from the remote host? This should get round any delays with getting System Admins to perform actions on your behalf.

          Michael Clarke added a comment - mcsf Which id_rsa.pub was it you copied in to the configuration? Your personal one (~/.ssh/id_rsa.pub), or the one for the host's public key (/etc/sshd/id_rsa.pub or similar)? It needs to be the latter for this to work. Your final failure indicates that your private key (used for authentication) does not match any public key saved on the host, which no amount of key verification is going to fix. Using Non verifying Verification Strategy is strongly discouraged , unless you can verify the identity of the host through another means. Have you considered using the "Manually Trusted Key Verification Strategy" to retrieve the remote host's key and then for you to verify the key against the contents of rsa_id.pub from the remote host? This should get round any delays with getting System Admins to perform actions on your behalf.

            mc1arke Michael Clarke
            wl2776 Vladimir Eremeev
            Votes:
            55 Vote for this issue
            Watchers:
            75 Start watching this issue

              Created:
              Updated:
              Resolved: