Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44832

SSH Slaves plugin fails the host verification of slaves with an IllegalArgumentException

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: ssh-slaves-plugin
    • Labels:
    • Environment:
      Java 64bit 1.8.0.131
      Jenkins 2.46.3
      ssh-slaves-plugin 1.18 (and 1.19)
    • Similar Issues:

      Description

      When we upgrade the ssh slaves plugin on the latest stable jenkins we get :

      ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
      java.lang.IllegalArgumentException
        at com.trilead.ssh2.Connection.setServerHostKeyAlgorithms(Connection.java:1311)
        at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:796)
        at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
      [06/13/17 13:10:39] Launch failed - cleaning up connection
      [06/13/17 13:10:39] [SSH] Connection closed.
      

       

        Attachments

          Issue Links

            Activity

            admincrowdiugo IUGO Admin created issue -
            admincrowdiugo IUGO Admin made changes -
            Field Original Value New Value
            Description When we upgrade the ssh slaves plugin on the latest stable jenkins we get :

             

            ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins. java.lang.IllegalArgumentException at com.trilead.ssh2.Connection.setServerHostKeyAlgorithms(Connection.java:1311) at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:789) at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:785) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) [06/12/17 13:16:57] Launch failed - cleaning up connection [06/12/17 13:16:57] [SSH] Connection closed.
             
            running jdk 1.8.132. 
            h3. Jenkins v2.46.3
             
            When we upgrade the ssh slaves plugin on the latest stable jenkins we get :

             

            ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins. java.lang.IllegalArgumentException at com.trilead.ssh2.Connection.setServerHostKeyAlgorithms(Connection.java:1311) at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:789) at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:785) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) [06/12/17 13:16:57] Launch failed - cleaning up connection [06/12/17 13:16:57] [SSH] Connection closed.
              
             running jdk 1.8.131.

            java -version
            java version "1.8.0_131"
            Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
            Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode) 
            h3. Jenkins v2.46.3

             
            Hide
            nfalco Nikolas Falco added a comment -

            Here too with version 1.19

            the only workaround is setup node verification to "Non verifying Verification Strategy" with the risk of Man-in-the-Middle

            Maybe we should assign this task to Jesse Glick

            Show
            nfalco Nikolas Falco added a comment - Here too with version 1.19 the only workaround is setup node verification to "Non verifying Verification Strategy" with the risk of Man-in-the-Middle Maybe we should assign this task to Jesse Glick
            Hide
            nfalco Nikolas Falco added a comment -

            Format the issue and move down to critical because workaround exists

            Show
            nfalco Nikolas Falco added a comment - Format the issue and move down to critical because workaround exists
            nfalco Nikolas Falco made changes -
            Description When we upgrade the ssh slaves plugin on the latest stable jenkins we get :

             

            ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins. java.lang.IllegalArgumentException at com.trilead.ssh2.Connection.setServerHostKeyAlgorithms(Connection.java:1311) at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:789) at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:785) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) [06/12/17 13:16:57] Launch failed - cleaning up connection [06/12/17 13:16:57] [SSH] Connection closed.
              
             running jdk 1.8.131.

            java -version
            java version "1.8.0_131"
            Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
            Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode) 
            h3. Jenkins v2.46.3

             
            When we upgrade the ssh slaves plugin on the latest stable jenkins we get :

            {noformat}
            ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
            java.lang.IllegalArgumentException
              at com.trilead.ssh2.Connection.setServerHostKeyAlgorithms(Connection.java:1311)
              at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:796)
              at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
            [06/13/17 13:10:39] Launch failed - cleaning up connection
            [06/13/17 13:10:39] [SSH] Connection closed.
            {noformat}

             
            Environment Java 64bit 1.8.0.131
            Jenkins 2.46.3
            Labels regression
            Priority Blocker [ 1 ] Critical [ 2 ]
            Summary SSH Slaves plugin 1.18 breaks all slaves - illegalargumentexception SSH Slaves plugin fails the host verification of slaves with an IllegalArgumentException
            nfalco Nikolas Falco made changes -
            Environment Java 64bit 1.8.0.131
            Jenkins 2.46.3
            Java 64bit 1.8.0.131
            Jenkins 2.46.3
            ssh-slaves-plugin 1.18 (and 1.19)
            jglick Jesse Glick made changes -
            Link This issue blocks JENKINS-42959 [ JENKINS-42959 ]
            Hide
            jglick Jesse Glick added a comment -

            Caused by fix of JENKINS-42959.

            Show
            jglick Jesse Glick added a comment - Caused by fix of JENKINS-42959 .
            jglick Jesse Glick made changes -
            Assignee Kohsuke Kawaguchi [ kohsuke ] Jesse Glick [ jglick ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            Hide
            jglick Jesse Glick added a comment -

            Not sure how to reproduce from scratch—from code inspection it looks like there are some conditions where getPreferredKeyAlgorithms could return null or an empty list, which is invalid. What is the verification strategy for the affected slave?

            Anyway I think I can just disable the fix of JENKINS-42959 in cases where it would cause this exception, perhaps.

            Show
            jglick Jesse Glick added a comment - Not sure how to reproduce from scratch—from code inspection it looks like there are some conditions where getPreferredKeyAlgorithms could return null or an empty list, which is invalid. What is the verification strategy for the affected slave? Anyway I think I can just disable the fix of JENKINS-42959 in cases where it would cause this exception, perhaps.
            Hide
            jglick Jesse Glick added a comment -
            		if ((algos == null) || (algos.length == 0))
            			throw new IllegalArgumentException();
            
            Show
            jglick Jesse Glick added a comment - if ((algos == null ) || (algos.length == 0)) throw new IllegalArgumentException();
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 63 (Web Link)" [ 17129 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/hudson/plugins/sshslaves/SSHLauncher.java
            src/main/java/hudson/plugins/sshslaves/verifiers/SshHostKeyVerificationStrategy.java
            http://jenkins-ci.org/commit/ssh-slaves-plugin/b9620eebd571c47f25e4c190ef6ea620b41f5c5a
            Log:
            [FIXED JENKINS-44832] IAE under unknown conditions due to empty or null list of preferred key algorithms.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/hudson/plugins/sshslaves/SSHLauncher.java src/main/java/hudson/plugins/sshslaves/verifiers/SshHostKeyVerificationStrategy.java http://jenkins-ci.org/commit/ssh-slaves-plugin/b9620eebd571c47f25e4c190ef6ea620b41f5c5a Log: [FIXED JENKINS-44832] IAE under unknown conditions due to empty or null list of preferred key algorithms.
            nfalco Nikolas Falco made changes -
            Attachment slave setting.png [ 38451 ]
            nfalco Nikolas Falco made changes -
            Attachment known_hosts_on_master.png [ 38452 ]
            Hide
            nfalco Nikolas Falco added a comment -

            The actual configuration for nodes is "Know host file Verification Strategy" (that means ~/.ssh/known_hosts file)

            With version 1.17 all slaves work without issue updating to 1.19 no way to connect. I got that error.

            I've try also to change the Verification Strategy save and back to Know host (maybe serialisation is changed) but with no luck.

            Actual setting for all slaves

            A known_hosts file on master

            Jesse Glick let me know if you need other file to profile the issue.

            Show
            nfalco Nikolas Falco added a comment - The actual configuration for nodes is "Know host file Verification Strategy" (that means ~/.ssh/known_hosts file) With version 1.17 all slaves work without issue updating to 1.19 no way to connect. I got that error. I've try also to change the Verification Strategy save and back to Know host (maybe serialisation is changed) but with no luck. Actual setting for all slaves A known_hosts file on master Jesse Glick let me know if you need other file to profile the issue.
            Hide
            jglick Jesse Glick added a comment -

            Based on the screenshot from Nikolas Falco I presume that KnownHosts.getPreferredServerHostkeyAlgorithmOrder is the culprit here. Indeed there are various documented conditions under which it would return null.

            Show
            jglick Jesse Glick added a comment - Based on the screenshot from Nikolas Falco I presume that KnownHosts.getPreferredServerHostkeyAlgorithmOrder is the culprit here. Indeed there are various documented conditions under which it would return null.
            Hide
            jglick Jesse Glick added a comment -

            Releasing as 1.20.

            Show
            jglick Jesse Glick added a comment - Releasing as 1.20.
            jglick Jesse Glick made changes -
            Resolution Fixed [ 1 ]
            Status In Review [ 10005 ] Resolved [ 5 ]
            Hide
            jglick Jesse Glick added a comment -

            Potentially the better fix would be to return super.getPreferredKeyAlgorithms(computer) in this case. At any rate, my patch ought to restore the 1.17 behavior in this situation, which I guess is good enough for now. Michael Clarke can look into refinements later.

            Show
            jglick Jesse Glick added a comment - Potentially the better fix would be to return super.getPreferredKeyAlgorithms(computer) in this case. At any rate, my patch ought to restore the 1.17 behavior in this situation, which I guess is good enough for now. Michael Clarke can look into refinements later.
            nfalco Nikolas Falco made changes -
            Attachment image-2017-06-13-16-27-57-910.png [ 38453 ]
            Hide
            nfalco Nikolas Falco added a comment -

            I post also the config.xml of my 1.17 configuration (I'm back)

            Show
            nfalco Nikolas Falco added a comment - I post also the config.xml of my 1.17 configuration (I'm back)
            Hide
            jglick Jesse Glick added a comment -

            Nothing new there—the only relevant aspect of the configuration in Jenkins is that you are using the known-hosts option. The question (if anyone cares) is why getPreferredServerHostkeyAlgorithmOrder failed. UnknownHostException trying to look up the slave? More likely, no known-hosts entry for that machine in recommendHostkeyAlgorithms, or multiple entries with different key types. Again I will leave it to Michael Clarke to evaluate what the desirable behavior is—I am just releasing a hotfix.

            At any rate, try updating to 1.20 when it becomes available on the update center, which should be soon.

            Show
            jglick Jesse Glick added a comment - Nothing new there—the only relevant aspect of the configuration in Jenkins is that you are using the known-hosts option. The question (if anyone cares) is why getPreferredServerHostkeyAlgorithmOrder failed. UnknownHostException trying to look up the slave? More likely, no known-hosts entry for that machine in recommendHostkeyAlgorithms , or multiple entries with different key types. Again I will leave it to Michael Clarke to evaluate what the desirable behavior is—I am just releasing a hotfix. At any rate, try updating to 1.20 when it becomes available on the update center, which should be soon.
            Hide
            nfalco Nikolas Falco added a comment -

            I've download the hpi from the Jenkins JFrog repository

            It works, follow the log of one node:

            Warning: no key algorithms provided; JENKINS-42959 disabled
            [06/13/17 17:58:56] [SSH] Opening SSH connection to 10.0.7.188:22.
            [06/13/17 17:58:57] [SSH] SSH host key matches key in Known Hosts file. Connection will be allowed.
            [06/13/17 17:58:58] [SSH] Authentication successful.
            [06/13/17 17:58:58] [SSH] The remote users environment is:
            
            Show
            nfalco Nikolas Falco added a comment - I've download the hpi from the Jenkins JFrog repository It works, follow the log of one node: Warning: no key algorithms provided; JENKINS-42959 disabled [06/13/17 17:58:56] [SSH] Opening SSH connection to 10.0.7.188:22. [06/13/17 17:58:57] [SSH] SSH host key matches key in Known Hosts file. Connection will be allowed. [06/13/17 17:58:58] [SSH] Authentication successful. [06/13/17 17:58:58] [SSH] The remote users environment is:

              People

              Assignee:
              jglick Jesse Glick
              Reporter:
              admincrowdiugo IUGO Admin
              Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: