Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46754

2.73+ SSH agent sometimes will not start if using passphrase-protected ed25519 key

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • core
    • Jenkins 2.73.1 RC
      Jenkins plugins as stored in my lts-with-plugins branch SHA f45cc34ca0

      The Jenkins 2.73.1 LTS release fails to connect my ssh agents which use an ed25519 passphrase protected private key.  These agents connected successfully with Jenkins 2.60.3 LTS and earlier.

      I've confirmed that dsa passphrase protected private keys work in all cases and that rsa passphrase protected private keys work in all cases. The rsa private keys and ed25519 private keys which are not passphrase protected work in all cases.

      It appears to only be ed25519 private keys which are passphrase protected that have a problem in two of my six tested configurations with 2.73.1 LTS.  Those same configurations work as expected with 2.60.3 LTS.

      Failures include a stack trace:

      [09/08/17 08:56:01] SSH Launch of mark-pc2-beemarkwaite on mark-pc2.markwaite.net failed in 113 ms
      Sep 08, 2017 8:56:01 AM com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator authenticate
      WARNING: Uncaught exception escaped doAuthenticate method
      java.lang.NoSuchMethodError: org.mindrot.jbcrypt.BCrypt.pbkdf([B[BI[B)V
      at com.trilead.ssh2.signature.OpenSshCertificateDecoder.generateKayAndIvPbkdf2(OpenSshCertificateDecoder.java:135)
      at com.trilead.ssh2.signature.OpenSshCertificateDecoder.createKeyPair(OpenSshCertificateDecoder.java:78)
      at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:493)
      at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:225)
      at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:483)
      at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:109)
      at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:438)
      at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:458)
      at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1321)
      at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:804)
      at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:793)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:748)
      

       
      The other agent fails with a similar stack trace in the log file:

      Sep 08, 2017 9:06:13 AM com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator authenticate
      WARNING: Uncaught exception escaped doAuthenticate method
      java.lang.NoSuchMethodError: org.mindrot.jbcrypt.BCrypt.pbkdf([B[BI[B)V
      	at com.trilead.ssh2.signature.OpenSshCertificateDecoder.generateKayAndIvPbkdf2(OpenSshCertificateDecoder.java:135)
      	at com.trilead.ssh2.signature.OpenSshCertificateDecoder.createKeyPair(OpenSshCertificateDecoder.java:78)
      	at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:493)
      	at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:225)
      	at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:483)
      	at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:109)
      	at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:438)
      	at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:458)
      	at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1321)
      	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:804)
      	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:793)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:748)
      
      [09/08/17 09:06:13] SSH Launch of debian9-a-coleen on debian9-a.markwaite.net failed in 135 ms
      

      Problem does not appear in 2.71, 2.72, 2.73, or 2.75 on the two failing machines.
      Problem is visible in 2.73.1-rc, 2.76, and 2.77 on the two failing machines.

          [JENKINS-46754] 2.73+ SSH agent sometimes will not start if using passphrase-protected ed25519 key

          Code changed in jenkins
          User: Daniel Beck
          Path:
          content/_data/changelogs/lts.yml
          content/doc/upgrade-guide/2.73.adoc
          http://jenkins-ci.org/commit/jenkins.io/ac48867bd1839ede182ef2a98b7c8f6da3c91c90
          Log:
          Note details on JENKINS-46754

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: content/_data/changelogs/lts.yml content/doc/upgrade-guide/2.73.adoc http://jenkins-ci.org/commit/jenkins.io/ac48867bd1839ede182ef2a98b7c8f6da3c91c90 Log: Note details on JENKINS-46754

          Code changed in jenkins
          User: R. Tyler Croy
          Path:
          content/_data/changelogs/lts.yml
          content/doc/upgrade-guide/2.73.adoc
          http://jenkins-ci.org/commit/jenkins.io/78b392bef2fe9c9aba2e3db309c89bdca2942aaa
          Log:
          Merge pull request #1131 from daniel-beck/JENKINS-46754

          Note details on JENKINS-46754

          Compare: https://github.com/jenkins-infra/jenkins.io/compare/352a9939f8c3...78b392bef2fe

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: R. Tyler Croy Path: content/_data/changelogs/lts.yml content/doc/upgrade-guide/2.73.adoc http://jenkins-ci.org/commit/jenkins.io/78b392bef2fe9c9aba2e3db309c89bdca2942aaa Log: Merge pull request #1131 from daniel-beck/ JENKINS-46754 Note details on JENKINS-46754 Compare: https://github.com/jenkins-infra/jenkins.io/compare/352a9939f8c3...78b392bef2fe

          Jesse Glick added a comment - - edited

          Affected private keys are those that require a passphrase and start with the line:

          -----BEGIN OPENSSH PRIVATE KEY-----
          

          Currently it seems ssh-keygen will only use this format when -t ed25519 is specified (older types such as RSA have distinct headers and internal formats), but in principle other (future?) key types could share this new format.

          Jesse Glick added a comment - - edited Affected private keys are those that require a passphrase and start with the line: -----BEGIN OPENSSH PRIVATE KEY----- Currently it seems ssh-keygen will only use this format when -t ed25519 is specified (older types such as RSA have distinct headers and internal formats), but in principle other (future?) key types could share this new format.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          core/pom.xml
          test/src/test/java/jenkins/ClassPathTest.java
          http://jenkins-ci.org/commit/jenkins/1784f90806c1c1f39e307c722a3dd4f63850877e
          Log:
          [FIXED JENKINS-46754] Remove org.mindrot:jbcrypt:0.4 since we already bundle org.connectbot.jbcrypt:jbcrypt:1.0.0.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/pom.xml test/src/test/java/jenkins/ClassPathTest.java http://jenkins-ci.org/commit/jenkins/1784f90806c1c1f39e307c722a3dd4f63850877e Log: [FIXED JENKINS-46754] Remove org.mindrot:jbcrypt:0.4 since we already bundle org.connectbot.jbcrypt:jbcrypt:1.0.0.

          Jesse Glick added a comment -

          Merged toward 2.79.

          Jesse Glick added a comment - Merged toward 2.79.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          src/main/java/org/jenkinsci/test/acceptance/docker/fixtures/SshAgentContainer.java
          src/main/java/org/jenkinsci/test/acceptance/plugins/ssh_credentials/SshPrivateKeyCredential.java
          src/main/java/org/jenkinsci/test/acceptance/plugins/ssh_slaves/SshSlaveLauncher.java
          src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshAgentContainer/Dockerfile
          src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshAgentContainer/ed25519.pass
          src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshAgentContainer/ed25519.priv
          src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshAgentContainer/ed25519.pub
          src/test/java/plugins/SshSlavesPluginTest.java
          http://jenkins-ci.org/commit/acceptance-test-harness/7544f951fb4b854cd5db89c60ea48da9178c0f6a
          Log:
          JENKINS-46754 Reproduce bug and demonstrate fix.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/test/acceptance/docker/fixtures/SshAgentContainer.java src/main/java/org/jenkinsci/test/acceptance/plugins/ssh_credentials/SshPrivateKeyCredential.java src/main/java/org/jenkinsci/test/acceptance/plugins/ssh_slaves/SshSlaveLauncher.java src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshAgentContainer/Dockerfile src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshAgentContainer/ed25519.pass src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshAgentContainer/ed25519.priv src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshAgentContainer/ed25519.pub src/test/java/plugins/SshSlavesPluginTest.java http://jenkins-ci.org/commit/acceptance-test-harness/7544f951fb4b854cd5db89c60ea48da9178c0f6a Log: JENKINS-46754 Reproduce bug and demonstrate fix.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          src/main/java/org/jenkinsci/test/acceptance/docker/fixtures/SshAgentContainer.java
          src/main/java/org/jenkinsci/test/acceptance/plugins/ssh_credentials/SshPrivateKeyCredential.java
          src/main/java/org/jenkinsci/test/acceptance/plugins/ssh_slaves/SshSlaveLauncher.java
          src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshAgentContainer/Dockerfile
          src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshAgentContainer/ed25519.pass
          src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshAgentContainer/ed25519.priv
          src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshAgentContainer/ed25519.pub
          src/test/java/plugins/SshSlavesPluginTest.java
          http://jenkins-ci.org/commit/acceptance-test-harness/3a09d8b9b0b2317c0c3c5a690aa4564a9693a63d
          Log:
          Merge pull request #354 from jglick/jbcrypt-JENKINS-46754

          JENKINS-46754 Reproduce bug and demonstrate fix

          Compare: https://github.com/jenkinsci/acceptance-test-harness/compare/539505e2ff4c...3a09d8b9b0b2

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/test/acceptance/docker/fixtures/SshAgentContainer.java src/main/java/org/jenkinsci/test/acceptance/plugins/ssh_credentials/SshPrivateKeyCredential.java src/main/java/org/jenkinsci/test/acceptance/plugins/ssh_slaves/SshSlaveLauncher.java src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshAgentContainer/Dockerfile src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshAgentContainer/ed25519.pass src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshAgentContainer/ed25519.priv src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/SshAgentContainer/ed25519.pub src/test/java/plugins/SshSlavesPluginTest.java http://jenkins-ci.org/commit/acceptance-test-harness/3a09d8b9b0b2317c0c3c5a690aa4564a9693a63d Log: Merge pull request #354 from jglick/jbcrypt- JENKINS-46754 JENKINS-46754 Reproduce bug and demonstrate fix Compare: https://github.com/jenkinsci/acceptance-test-harness/compare/539505e2ff4c...3a09d8b9b0b2

          R. Tyler Croy added a comment -

          I have written a script, linked via Gist, which will help administrators identify whether their system is problematic.

          R. Tyler Croy added a comment - I have written a script, linked via Gist, which will help administrators identify whether their system is problematic.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          core/pom.xml
          test/src/test/java/jenkins/ClassPathTest.java
          http://jenkins-ci.org/commit/jenkins/fa96a02a3e39c0fa1d561ef254f6d36e40ed3b5e
          Log:
          [FIXED JENKINS-46754] Remove org.mindrot:jbcrypt:0.4 since we already bundle org.connectbot.jbcrypt:jbcrypt:1.0.0.

          (cherry picked from commit 1784f90806c1c1f39e307c722a3dd4f63850877e)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/pom.xml test/src/test/java/jenkins/ClassPathTest.java http://jenkins-ci.org/commit/jenkins/fa96a02a3e39c0fa1d561ef254f6d36e40ed3b5e Log: [FIXED JENKINS-46754] Remove org.mindrot:jbcrypt:0.4 since we already bundle org.connectbot.jbcrypt:jbcrypt:1.0.0. (cherry picked from commit 1784f90806c1c1f39e307c722a3dd4f63850877e)

          Code changed in jenkins
          User: Jesse Glick
          Path:
          core/src/main/java/hudson/slaves/SlaveComputer.java
          core/src/main/java/jenkins/model/Jenkins.java
          pom.xml
          http://jenkins-ci.org/commit/jenkins/f9ad963d1fb7e9840cd79bf084c3ab180708aca0
          Log:
          Revert "JENKINS-46754 Revert "Upgrade Remoting to 3.11 (#2988)""

          This reverts commit f6ef88211b22d0aec54431820cfb5e5a9fa91610.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/slaves/SlaveComputer.java core/src/main/java/jenkins/model/Jenkins.java pom.xml http://jenkins-ci.org/commit/jenkins/f9ad963d1fb7e9840cd79bf084c3ab180708aca0 Log: Revert " JENKINS-46754 Revert "Upgrade Remoting to 3.11 (#2988)"" This reverts commit f6ef88211b22d0aec54431820cfb5e5a9fa91610.

            jglick Jesse Glick
            markewaite Mark Waite
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: