Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61356

Trilead SSH does not support PKCS#8 encoded private keys

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • trilead-api-plugin

      The Jenkins SSH agent plugin uses trilead-ssh which only supports the legacy PEM and new SSH2 private key file formats. In certain hardened environments, SSH keys are mandated to be encoded in PKCS#8 (presumably because they're more universal and allow for password protection and encrypting the key) to the point where OpenSSH and ssh-keygen are patched to only support PKCS#8 private keys.

      Attached to this ticket are various encodings of private keys as we troubleshooted the initial issue described below. Note that private keys that have a header line with BEGIN PRIVATE KEY are indicative of the file being encoded in PKCS#8. If the file header contains BEGIN RSA PRIVATE KEY (or EC or whatever other algorithm), that is the legacy PEM format. If the file header contains BEGIN OPENSSH PRIVATE KEY, then that is the new SSH2 format. The latter two encoding formats are already supported in trilead-ssh.

      Original Details

      • My Jenkins exists on an Amazon EC2 instance.
      • I'm trying to add Red Hat 7.7 slave via ssh.
      • I can ssh directly using terminal from my master to slave and vice versa using that key file
      jenkins@master:/var/lib/jenkins
      $ ssh -i .ssh/id_rsa -q 10.193.177.232
      
      jenkins@slave:/var/lib/jenkins
      $ ssh -i .ssh/id_rsa -q 10.193.177.209
      

      I’m also able to make a successful SSH connection when I perform a SSH connection test from the Manage Jenkins -> Configure System section of the Master (FYI - see attached screenshot).

      But when I try to configure my slave via launch agent method, it gives me the following error:

      caused by: java.io.IOException: PEM problem: it is of unknown type
              at com.trilead.ssh2.crypto.PEMDecoder.decpdeKeyPair(PEMDecoder.java:500)
      
      • How I've configured the slave (one of the method)?
        • I've saved slave machine credentials as "ssh key username with private key"
        • Launch method: Launch agents via SSH
        • Hostname: slave machine private ip address.
        • Host Key Verification Strategy: Manually trusted key Verification Strategy

      Note:

      • I cannot regenerate ssh keys as I'll not be able to access my machines again and the infrastructure is very strict. 
      • All SSH key are 4096 bits in length and RSA.
      • I cannot use credentials as username/password as either there is no password.
      • I've tried everything whatever mentioned in other jenkins tickets for same issue or on internet, but my issue is not getting resolved.

      Please see the following attachments for log details and screenshots.

       

      This is a total blocker. Please advise. Thank you.

        1. 61356.tar.gz
          6 kB
        2. Screenshot 2020-04-24 at 17.22.03.png
          Screenshot 2020-04-24 at 17.22.03.png
          66 kB
        3. 61356-key.txt
          3 kB
        4. 61356-key.pub.txt
          0.7 kB
        5. 61356-key.txt
          3 kB
        6. script-console-output.PNG
          script-console-output.PNG
          53 kB
        7. Unable to launch agent using plugin-ver.1.17.4.PNG
          Unable to launch agent using plugin-ver.1.17.4.PNG
          24 kB
        8. ver1.17.4.PNG
          ver1.17.4.PNG
          24 kB
        9. new privkey.txt
          2 kB
        10. ssh-cred.PNG
          ssh-cred.PNG
          4 kB
        11. com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.txt.txt
          6 kB
        12. Launch Agent error output.PNG
          Launch Agent error output.PNG
          88 kB
        13. Slave screenshot.PNG
          Slave screenshot.PNG
          54 kB
        14. SSH test from Master with the same SSH credential.PNG
          SSH test from Master with the same SSH credential.PNG
          23 kB
        15. slave-log.log
          2 kB

          [JENKINS-61356] Trilead SSH does not support PKCS#8 encoded private keys

          Matt Sicker added a comment -

          Seems like openjdk doesn't disable strong crypto; it's oraclejdk that did. Some info here: https://support.cloudbees.com/hc/en-us/articles/115003785531-SSL-handshake-failures-due-to-incompatible-Cipher-Suite

          Looking at the code, though, it seems like trilead-ssh is using its own PEM parsing code rather than the standard Java API. I'll note the method in question: https://github.com/jenkinsci/trilead-ssh2/blob/master/src/com/trilead/ssh2/crypto/PEMDecoder.java#L478

          Could you try executing the following in a script console: com.trilead.ssh2.signature.KeyAlgorithmManager.getSupportedAlgorithms().dump()

          Matt Sicker added a comment - Seems like openjdk doesn't disable strong crypto; it's oraclejdk that did. Some info here: https://support.cloudbees.com/hc/en-us/articles/115003785531-SSL-handshake-failures-due-to-incompatible-Cipher-Suite Looking at the code, though, it seems like trilead-ssh is using its own PEM parsing code rather than the standard Java API. I'll note the method in question: https://github.com/jenkinsci/trilead-ssh2/blob/master/src/com/trilead/ssh2/crypto/PEMDecoder.java#L478 Could you try executing the following in a script console: com.trilead.ssh2.signature.KeyAlgorithmManager.getSupportedAlgorithms().dump()

          Matt Sicker added a comment -

          Adding trilead as an affected component since the stacktrace relates to it. The linked file hasn't been updated in three years, so perhaps you're using a newer key algorithm?

          Matt Sicker added a comment - Adding trilead as an affected component since the stacktrace relates to it. The linked file hasn't been updated in three years, so perhaps you're using a newer key algorithm?

          Charles Smith added a comment -

          jvz

          Thank you for all of the updated information. 

          Do I need to rerun a similar Maven test on com.trilead.ssh2.signature.KeyAlgorithmManager.getSupportedAlgorithms().dump() like I did previously, or is there another way to execute this?

          Charles Smith added a comment - jvz Thank you for all of the updated information.  Do I need to rerun a similar Maven test on  com.trilead.ssh2.signature.KeyAlgorithmManager.getSupportedAlgorithms().dump()  like I did previously, or is there another way to execute this?

          Matt Sicker added a comment -

          You can run that from the Jenkins script console: https://jenkins.io/doc/book/managing/script-console/

          Matt Sicker added a comment - You can run that from the Jenkins script console: https://jenkins.io/doc/book/managing/script-console/

          Charles Smith added a comment - - edited

          jvz

          GM Matt - Below is the result after running the com.trilead.ssh2.signature.KeyAlgorithmManager.getSupportedAlgorithms().dump() from the script console:

          Result
          
          Result: <java.util.Collections$UnmodifiableCollection@42327252 c=[com.trilead.ssh2.signature.ED25519KeyAlgorithm@4eff3ac, com.trilead.ssh2.signature.ECDSAKeyAlgorithm$ECDSASha2Nistp521@551eb181, com.trilead.ssh2.signature.ECDSAKeyAlgorithm$ECDSASha2Nistp384@4ca06dc6, com.trilead.ssh2.signature.ECDSAKeyAlgorithm$ECDSASha2Nistp256@796bd6b5, com.trilead.ssh2.signature.RSAKeyAlgorithm@26238a7, com.trilead.ssh2.signature.DSAKeyAlgorithm@225013d3]>
          

          Is this the expected result? I cannot tell from the output?

          Charles Smith added a comment - - edited jvz GM Matt - Below is the result after running the  com.trilead.ssh2.signature.KeyAlgorithmManager.getSupportedAlgorithms().dump() from the script console: Result Result: <java.util.Collections$UnmodifiableCollection@42327252 c=[com.trilead.ssh2.signature.ED25519KeyAlgorithm@4eff3ac, com.trilead.ssh2.signature.ECDSAKeyAlgorithm$ECDSASha2Nistp521@551eb181, com.trilead.ssh2.signature.ECDSAKeyAlgorithm$ECDSASha2Nistp384@4ca06dc6, com.trilead.ssh2.signature.ECDSAKeyAlgorithm$ECDSASha2Nistp256@796bd6b5, com.trilead.ssh2.signature.RSAKeyAlgorithm@26238a7, com.trilead.ssh2.signature.DSAKeyAlgorithm@225013d3]> Is this the expected result? I cannot tell from the output?

          Matt Sicker added a comment - - edited

          That looks correct to me. I tried to reproduce this issue, but I'm not getting any PEM errors. Are you including the

          -----BEGIN RSA PRIVATE KEY-----

          and

          -----END RSA PRIVATE KEY-----

          lines in the key when you save it as an SSH credential? That's about the only thing I can think of that would cause the log message you're seeing to be output.

          Edit: my reproduction attempt was with a master running on macOS and using an outbound SSH agent to connect to an Ubuntu 18.04 server. I tried using your provided key as well, though that just gives me a public key refused connection error as expected.

          Matt Sicker added a comment - - edited That looks correct to me. I tried to reproduce this issue, but I'm not getting any PEM errors. Are you including the -----BEGIN RSA PRIVATE KEY----- and -----END RSA PRIVATE KEY----- lines in the key when you save it as an SSH credential? That's about the only thing I can think of that would cause the log message you're seeing to be output. Edit: my reproduction attempt was with a master running on macOS and using an outbound SSH agent to connect to an Ubuntu 18.04 server. I tried using your provided key as well, though that just gives me a public key refused connection error as expected.

          Matt Sicker added a comment -

          Oh here's a hypothesis: try adding "RSA" to your BEGIN/END PRIVATE KEY lines and see if you get the same error. If that works, then I believe we've isolated the issue to trilead's PEM decoder being overly strict.

          Matt Sicker added a comment - Oh here's a hypothesis: try adding "RSA" to your BEGIN/END PRIVATE KEY lines and see if you get the same error. If that works, then I believe we've isolated the issue to trilead's PEM decoder being overly strict.

          Charles Smith added a comment -

          jvz

          I have just tried adding the "RSA" to the privkey with:

          -----BEGIN RSA PRIVATE KEY-----
          

          and

          -----END RSA PRIVATE KEY-----
          

          and still receiving the PEM error:

          ERROR: Server rejected the 1 private key(s) for jenkins  (credentialId:jenkins-test-ssh-key/method:publickey)
          ERROR: Failed to authenticate as jenkins  with credential=jenkins-test-ssh-key
          java.io.IOException: Publickey authentication failed.
          	at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:358)
          	at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:472)
          	at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:109)
          	at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:436)
          	at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:473)
          	at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:863)
          	at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:435)
          	at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:422)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	at java.lang.Thread.run(Thread.java:748)
          Caused by: java.io.IOException: PEM problem: it is of unknown type
          	at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:500)
          	at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:292)
          	... 11 more
          [04/15/20 14:07:48] [SSH] Authentication failed.
          Authentication failed.
          [04/15/20 14:07:48] Launch failed - cleaning up connection
          

          I noticed that in one of your earlier replies you were using a "noformat" option:

          { { noformat} -----BEGIN RSA PRIVATE KEY----- {noformat } } and { { noformat} -----END RSA PRIVATE KEY----- {noformat } } 
          

          Do I need to add the 'noformat" option to the privkey?

          Charles Smith added a comment - jvz I have just tried adding the "RSA" to the privkey with: -----BEGIN RSA PRIVATE KEY----- and -----END RSA PRIVATE KEY----- and still receiving the PEM error: ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:jenkins-test-ssh-key/method:publickey) ERROR: Failed to authenticate as jenkins with credential=jenkins-test-ssh-key java.io.IOException: Publickey authentication failed. at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:358) at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:472) at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:109) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:436) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:473) at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:863) at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:435) at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:422) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) Caused by: java.io.IOException: PEM problem: it is of unknown type at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:500) at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:292) ... 11 more [04/15/20 14:07:48] [SSH] Authentication failed. Authentication failed. [04/15/20 14:07:48] Launch failed - cleaning up connection I noticed that in one of your earlier replies you were using a " noformat " option: { { noformat} -----BEGIN RSA PRIVATE KEY----- {noformat } } and { { noformat} -----END RSA PRIVATE KEY----- {noformat } } Do I need to add the ' noformat " option to the privkey?

          Daniel Beck added a comment -

          "noformat" is the formatting code for

          this 

          in this issue tracker. Matt formatted his message after it looked bad with just basic text.

          Daniel Beck added a comment - "noformat" is the formatting code for this in this issue tracker. Matt formatted his message after it looked bad with just basic text.

          Matt Sicker added a comment -

          I did the noformat thing because otherwise jira was interpreting some of the dashes as strikeout text.

          Matt Sicker added a comment - I did the noformat thing because otherwise jira was interpreting some of the dashes as strikeout text.

          Charles Smith added a comment - - edited

          jvz danielbeck

          Oh ok thanks for the confirmation. I'm not using the private key that I provided earlier. If you want to test with the public key then I can send it to you?

          I just tried a new separate 2048bit and 4096bit key with the RSA suggestion and still receive the PEM error message. I'm n ot sure if they're may be some unseen characters being added when I copy the new priv keys into the credential field.

          Charles Smith added a comment - - edited jvz danielbeck Oh ok thanks for the confirmation. I'm not using the private key that I provided earlier. If you want to test with the public key then I can send it to you? I just tried a new separate 2048bit and 4096bit key with the RSA suggestion and still receive the PEM error message. I'm n ot sure if they're may be some unseen characters being added when I copy the new priv keys into the credential field.

          Matt Sicker added a comment -

          I think we've isolated this to being a problem in trilead, or at least in our fork of it. The last active developer there is ifernandezcalvo. Perhaps this is something supported upstream that's not merged into our fork?

          Matt Sicker added a comment - I think we've isolated this to being a problem in trilead, or at least in our fork of it. The last active developer there is ifernandezcalvo . Perhaps this is something supported upstream that's not merged into our fork?

          Reading the issue I do not know which version of trilead-ssh2 or trilead-api plugins are installed. I am gonna try with an RSA key generated with the following command

          ssh-keygen -t rsa –b 4096

          that it is the only command I saw in the comments but IIRC this exact command is the one I use so I would not replicate the issue. I will provide the test environment in a while after my tests.

          Ivan Fernandez Calvo added a comment - Reading the issue I do not know which version of trilead-ssh2 or trilead-api plugins are installed. I am gonna try with an RSA key generated with the following command ssh-keygen -t rsa –b 4096 that it is the only command I saw in the comments but IIRC this exact command is the one I use so I would not replicate the issue. I will provide the test environment in a while after my tests.

          Matt Sicker added a comment -

          It seems like it might be related to the message digests allowed. SHA-1 has been deprecated in openssh, so maybe it's possible that trilead doesn't support SHA-256 or something?

          Matt Sicker added a comment - It seems like it might be related to the message digests allowed. SHA-1 has been deprecated in openssh, so maybe it's possible that trilead doesn't support SHA-256 or something?

          Ivan Fernandez Calvo added a comment - SHA-256 is supported https://github.com/jenkinsci/trilead-ssh2/blob/3322e54a48f500487ddb6862be42e289e7d87e08/src/com/trilead/ssh2/transport/KexManager.java#L435-L439

          This is my test environment https://github.com/kuisathaverat/jenkins-issues/tree/master/JENKINS-61356 and the key is generated with this ssh-keygen version

          ssh -V
          OpenSSH_8.1p1, LibreSSL 2.7.3
          

          The key generated with this command

          ssh-keygen -t rsa –b 4096 -N ""
          

          I use JCasC to configure a credential with the key
          https://github.com/kuisathaverat/jenkins-issues/blob/master/JENKINS-61356/jenkins/jenkins_home/jenkins.yaml#L11-L59

          and I've added the public key to the authorized key in the agent,
          https://github.com/kuisathaverat/jenkins-issues/blob/master/JENKINS-61356/ssh-agent/ssh/authorized_keys

          I cannot replicate the issue on the latest Jenkins core (2.30) and the latest plugins

          Ivan Fernandez Calvo added a comment - This is my test environment https://github.com/kuisathaverat/jenkins-issues/tree/master/JENKINS-61356 and the key is generated with this ssh-keygen version ssh -V OpenSSH_8.1p1, LibreSSL 2.7.3 The key generated with this command ssh-keygen -t rsa –b 4096 -N "" I use JCasC to configure a credential with the key https://github.com/kuisathaverat/jenkins-issues/blob/master/JENKINS-61356/jenkins/jenkins_home/jenkins.yaml#L11-L59 and I've added the public key to the authorized key in the agent, https://github.com/kuisathaverat/jenkins-issues/blob/master/JENKINS-61356/ssh-agent/ssh/authorized_keys I cannot replicate the issue on the latest Jenkins core (2.30) and the latest plugins

          Ivan Fernandez Calvo added a comment - - edited

          Thinking on common pitfalls I make a change on my configuration to create the credential with the PUBLIC key (totally wrong thing) and guess what the same error

          SSHLauncher{host='ssh-agent', port=22, credentialsId='12187fc5-6f2d-4343-ad41-7a4a8c683b85', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.NonVerifyingKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
          [04/16/20 19:32:34] [SSH] Opening SSH connection to ssh-agent:22.
          [04/16/20 19:32:34] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
          ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:12187fc5-6f2d-4343-ad41-7a4a8c683b85/method:publickey)
          ERROR: Failed to authenticate as jenkins with credential=12187fc5-6f2d-4343-ad41-7a4a8c683b85
          java.io.IOException: Publickey authentication failed.
          	at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:358)
          	at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:472)
          	at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:109)
          	at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:436)
          	at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:473)
          	at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:863)
          	at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:435)
          	at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:422)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	at java.lang.Thread.run(Thread.java:748)
          Caused by: java.io.IOException: PEM problem: it is of unknown type
          	at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:500)
          	at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:292)
          	... 11 more
          [04/16/20 19:32:34] [SSH] Authentication failed.
          Authentication failed.
          [04/16/20 19:32:34] Launch failed - cleaning up connection
          [04/16/20 19:32:34] [SSH] Connection closed.
          

          clsmith4 Are you sure you are creating the Jenkins credential with the PRIVATE key(something like this one https://github.com/kuisathaverat/jenkins-issues/blob/master/JENKINS-61356/ssh-agent/ssh/rsa-key) and the authorized_keys with the PUBLIC key (https://github.com/kuisathaverat/jenkins-issues/blob/master/JENKINS-61356/ssh-agent/ssh/rsa-key.pub)

          Other common pitfalls with the same result are to copy and paste the key in an editor that does not support Unix line breaks and save the key with windows linebreaks, add spaces on any part of the key, in general, anything that modifies the original ARMOR format of the key break the key.

          To have a key pair that is confirmed that fails could help to be sure that the problem is not in the plugin or Jenkins

          Ivan Fernandez Calvo added a comment - - edited Thinking on common pitfalls I make a change on my configuration to create the credential with the PUBLIC key (totally wrong thing) and guess what the same error SSHLauncher{host= 'ssh-agent' , port=22, credentialsId= '12187fc5-6f2d-4343-ad41-7a4a8c683b85' , jvmOptions= '', javaPath=' ', prefixStartSlaveCmd=' ', suffixStartSlaveCmd=' ', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.NonVerifyingKeyVerificationStrategy, tcpNoDelay= true , trackCredentials= true } [04/16/20 19:32:34] [SSH] Opening SSH connection to ssh-agent:22. [04/16/20 19:32:34] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:12187fc5-6f2d-4343-ad41-7a4a8c683b85/method:publickey) ERROR: Failed to authenticate as jenkins with credential=12187fc5-6f2d-4343-ad41-7a4a8c683b85 java.io.IOException: Publickey authentication failed. at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:358) at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:472) at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:109) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:436) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:473) at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:863) at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:435) at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:422) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) Caused by: java.io.IOException: PEM problem: it is of unknown type at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:500) at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:292) ... 11 more [04/16/20 19:32:34] [SSH] Authentication failed. Authentication failed. [04/16/20 19:32:34] Launch failed - cleaning up connection [04/16/20 19:32:34] [SSH] Connection closed. clsmith4 Are you sure you are creating the Jenkins credential with the PRIVATE key(something like this one https://github.com/kuisathaverat/jenkins-issues/blob/master/JENKINS-61356/ssh-agent/ssh/rsa-key ) and the authorized_keys with the PUBLIC key ( https://github.com/kuisathaverat/jenkins-issues/blob/master/JENKINS-61356/ssh-agent/ssh/rsa-key.pub ) Other common pitfalls with the same result are to copy and paste the key in an editor that does not support Unix line breaks and save the key with windows linebreaks, add spaces on any part of the key, in general, anything that modifies the original ARMOR format of the key break the key. To have a key pair that is confirmed that fails could help to be sure that the problem is not in the plugin or Jenkins

          Charles Smith added a comment - - edited

          ifernandezcalvo

          Thank you for all your updates on this issue.  

          Yes, I'm 1000% sure that it's the PRIVATE key(not PUBLIC) that I'm copying into the SSH credential field that asks for the PRIVATE key. I've created another keypair(61356-key & 61356-key.pub) and able to ssh without issue from the Master to the Build(slave) system using this test keypair. However, the same keypair isn't working when used by the shh credential plugin to launch the build agent. I've attached the keypair for your testing/visibility.

           

          If this is an issue with copying/pasting the keypair with added Unix line breaks, then what is a workaround for this? I'm a Unix guy so I copy the PRIVATE key from my Putty terminal directly into the SSH credential field. Is there another way of doing this to test this theory?

           

          Other than that, we're currently running RHEL7.8:

          $ ssh -V
          OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
          

           

          We also run security McAfee software that may potentially be an issue when creating the keypair:

          # systemctl status mfeespd.service; systemctl status mfetpd.service
          ● mfeespd.service - McAfee Endpoint Security Platform for Linux
             Loaded: loaded (/usr/lib/systemd/system/mfeespd.service; enabled; vendor preset: disabled)
             Active: active (running) since Wed 2020-04-15 14:13:48 CDT; 5 days ago
               Docs: man:mfeespd(8)
           Main PID: 4574 (mfeespd)
              Tasks: 9
             Memory: 2.9M
             CGroup: /system.slice/mfeespd.service
                     └─4574 /opt/McAfee/ens/esp/bin/mfeespdApr 15 14:13:46 ansible-baseline-rhel7-test.uscis.dhs.gov systemd[1]: Starting M...
          Apr 15 14:13:46 ansible-baseline-rhel7-test.uscis.dhs.gov prepare-mfeesp.sh[4561]: ...
          Apr 15 14:13:48 ansible-baseline-rhel7-test.uscis.dhs.gov systemd[1]: Started Mc...
          Hint: Some lines were ellipsized, use -l to show in full.
          ● mfetpd.service - McAfee Endpoint Security for Linux Threat Prevention
             Loaded: loaded (/usr/lib/systemd/system/mfetpd.service; enabled; vendor preset: disabled)
             Active: active (running) since Wed 2020-04-15 14:13:49 CDT; 5 days ago
               Docs: man:mfetpd(8)
           Main PID: 4732 (mfetpd)
              Tasks: 49
             Memory: 1.4G
             CGroup: /system.slice/mfetpd.service
                     ├─ 4732 /opt/McAfee/ens/tp/bin/mfetpd
                     ├─ 4777 /opt/McAfee/ens/tp/bin/mfetpd
                     ├─20076 /opt/McAfee/ens/tp/bin/mfetpd
                     └─20084 /opt/McAfee/ens/tp/bin/mfetpdApr 15 14:13:49 ansible-baseline-rhel7-test.uscis.dhs.gov systemd[1]: Starting M...
          Apr 15 14:13:49 ansible-baseline-rhel7-test.uscis.dhs.gov systemd[1]: Started Mc...
          Apr 15 14:13:50 ansible-baseline-rhel7-test.uscis.dhs.gov mfetpd[4732]: error: F...
          Hint: Some lines were ellipsized, use -l to show in full.
          
          

          However, the keypair works when I use it to ssh from the shell.

           

          Charles Smith added a comment - - edited ifernandezcalvo Thank you for all your updates on this issue.   Yes, I'm 1000% sure that it's the PRIVATE key(not PUBLIC) that I'm copying into the SSH credential field that asks for the PRIVATE key. I've created another keypair( 61356-key & 61356-key.pub ) and able to ssh without issue from the Master to the Build(slave) system using this test keypair. However, the same keypair isn't working when used by the shh credential plugin to launch the build agent. I've attached the keypair for your testing/visibility.   If this is an issue with copying/pasting the keypair with added Unix line breaks, then what is a workaround for this? I'm a Unix guy so I copy the PRIVATE key from my Putty terminal directly into the SSH credential field. Is there another way of doing this to test this theory?   Other than that, we're currently running RHEL7.8: $ ssh -V OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017   We also run security McAfee software that may potentially be an issue when creating the keypair: # systemctl status mfeespd.service; systemctl status mfetpd.service ● mfeespd.service - McAfee Endpoint Security Platform for Linux Loaded: loaded (/usr/lib/systemd/system/mfeespd.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2020-04-15 14:13:48 CDT; 5 days ago Docs: man:mfeespd(8) Main PID: 4574 (mfeespd) Tasks: 9 Memory: 2.9M CGroup: /system.slice/mfeespd.service └─4574 /opt/McAfee/ens/esp/bin/mfeespdApr 15 14:13:46 ansible-baseline-rhel7-test.uscis.dhs.gov systemd[1]: Starting M... Apr 15 14:13:46 ansible-baseline-rhel7-test.uscis.dhs.gov prepare-mfeesp.sh[4561]: ... Apr 15 14:13:48 ansible-baseline-rhel7-test.uscis.dhs.gov systemd[1]: Started Mc... Hint: Some lines were ellipsized, use -l to show in full. ● mfetpd.service - McAfee Endpoint Security for Linux Threat Prevention Loaded: loaded (/usr/lib/systemd/system/mfetpd.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2020-04-15 14:13:49 CDT; 5 days ago Docs: man:mfetpd(8) Main PID: 4732 (mfetpd) Tasks: 49 Memory: 1.4G CGroup: /system.slice/mfetpd.service ├─ 4732 /opt/McAfee/ens/tp/bin/mfetpd ├─ 4777 /opt/McAfee/ens/tp/bin/mfetpd ├─20076 /opt/McAfee/ens/tp/bin/mfetpd └─20084 /opt/McAfee/ens/tp/bin/mfetpdApr 15 14:13:49 ansible-baseline-rhel7-test.uscis.dhs.gov systemd[1]: Starting M... Apr 15 14:13:49 ansible-baseline-rhel7-test.uscis.dhs.gov systemd[1]: Started Mc... Apr 15 14:13:50 ansible-baseline-rhel7-test.uscis.dhs.gov mfetpd[4732]: error: F... Hint: Some lines were ellipsized, use -l to show in full. However, the keypair works when I use it to ssh from the shell.  

          Charles Smith added a comment -

          jvz ifernandezcalvo

          Hello All - Just checking for any more responses or updates on this? Where to go from here? How to test if erroneous characters are being added to the priv key section of that plugin? Any more information would be appreciated.  Thanks.

          Charles Smith added a comment - jvz ifernandezcalvo Hello All - Just checking for any more responses or updates on this? Where to go from here? How to test if erroneous characters are being added to the priv key section of that plugin? Any more information would be appreciated.  Thanks.

          Matt Sicker added a comment -

          Can you try generating a key using openssl genrsa -out private.pem 4096 and see if that works? I wonder if there's been some sort of key format change in ssh-keygen. I really wish this were using OpenSSH, but it's not.

          Matt Sicker added a comment - Can you try generating a key using openssl genrsa -out private.pem 4096 and see if that works? I wonder if there's been some sort of key format change in ssh-keygen. I really wish this were using OpenSSH, but it's not.

          Matt Sicker added a comment -

          In fact, I wonder if that's the issue. It says the PEM is of an unknown type, but an SSH private key is usually encoded via RFC4716 rather than PEM (which ssh-keygen(1) says is the "legacy" format). There's also PCKS8.

          Version: OpenSSH_8.1p1, LibreSSL 2.7.3

          Matt Sicker added a comment - In fact, I wonder if that's the issue. It says the PEM is of an unknown type, but an SSH private key is usually encoded via RFC4716 rather than PEM (which ssh-keygen(1) says is the "legacy" format). There's also PCKS8. Version: OpenSSH_8.1p1, LibreSSL 2.7.3

          Matt Sicker added a comment -

          Based on version differences, it seems like your fips-patched version of OpenSSH is likely what's causing the issue. There might be a missing SSH key format being supported by trilead?

          Matt Sicker added a comment - Based on version differences, it seems like your fips-patched version of OpenSSH is likely what's causing the issue. There might be a missing SSH key format being supported by trilead?

          Charles Smith added a comment -

          jvz

          Thanks for the updates. How do you want me to extract the public key from the pem file? 

          Using ssh-keygen:

          ssh-keygen-y -f key.pem > key.pub
          

          Or using openssl:

          openssl rsa -in key.pem -pubout -out pubkey.pem
          

          Note that when using openssl, the public key doesn't have the standard ssh-rsa heading that normal ssh public keys use.

          The fips-patched version of OpenSSH makes sense. I've attached a test keypair earlier that you can try and test with to see if the issue can be replicated. I'm not using that keypair.

          Charles Smith added a comment - jvz Thanks for the updates. How do you want me to extract the public key from the pem file?  Using ssh-keygen : ssh-keygen-y -f key.pem > key.pub Or using openssl : openssl rsa -in key.pem -pubout -out pubkey.pem Note that when using openssl , the public key doesn't have the standard ssh-rsa  heading that normal ssh public keys use. The fips-patched version of OpenSSH makes sense. I've attached a test keypair earlier that you can try and test with to see if the issue can be replicated. I'm not using that keypair.

          Matt Sicker added a comment -

          You can convert an existing key with ssh-keygen -p -f private_key_file -m pem

          Matt Sicker added a comment - You can convert an existing key with ssh-keygen -p -f private_key_file -m pem

          Matt Sicker added a comment -

          And I meant to try and see if the RSA keys generated from openssl are decoded properly by Jenkins compared to the ones being generated via ssh-keygen.

          Matt Sicker added a comment - And I meant to try and see if the RSA keys generated from openssl are decoded properly by Jenkins compared to the ones being generated via ssh-keygen.

          Matt Sicker added a comment -

          Alright, I've done some digging, and here's what I've found. The key you attached (61356) is encoded in PKCS#8 which can be seen by the "BEGIN PRIVATE KEY" header. What is usually used in SSH is PKCS#1 (i.e., no special encoding apparently) which has the header "BEGIN RSA PRIVATE KEY". The former format encodes the algorithm in the base64 data while the latter is more of a naked format with the algorithm in the header. I've also seen "BEGIN OPENSSH PRIVATE KEY" which appears to be the newer format which is used by ed25519 and other ciphers.

          One workaround you could try (in production, too!) would be to convert your private key to the old PEM format. You can use openssl rsa -in private.key -out private.pem -outform PEM to convert the key encoding format which should be supported by trilead. If that doesn't work, then we may have an interesting problem.

          Matt Sicker added a comment - Alright, I've done some digging, and here's what I've found. The key you attached (61356) is encoded in PKCS#8 which can be seen by the "BEGIN PRIVATE KEY" header. What is usually used in SSH is PKCS#1 (i.e., no special encoding apparently) which has the header "BEGIN RSA PRIVATE KEY". The former format encodes the algorithm in the base64 data while the latter is more of a naked format with the algorithm in the header. I've also seen "BEGIN OPENSSH PRIVATE KEY" which appears to be the newer format which is used by ed25519 and other ciphers. One workaround you could try (in production, too!) would be to convert your private key to the old PEM format. You can use openssl rsa -in private.key -out private.pem -outform PEM to convert the key encoding format which should be supported by trilead. If that doesn't work, then we may have an interesting problem.

          Matt Sicker added a comment -

          Looking at trilead, I see that RSAKeyAlgorithm only supports SHA1, and it doesn't even support aes-128/256-gcm, so it seems somewhat dated in algorithm support as it is. I'm kind of wondering why this library is the default rather than Apache SSHD (which was used for some other SSH functionality in Jenkins).

          I see there's support for the old pem file format and the new openssh (RFC 4716) format, but no explicit support for PKCS#8. Could you try out the other formats for the same key to see if it works? If it doesn't, then I think we have some missing crypto primitives here.

          Matt Sicker added a comment - Looking at trilead, I see that RSAKeyAlgorithm only supports SHA1, and it doesn't even support aes-128/256-gcm, so it seems somewhat dated in algorithm support as it is. I'm kind of wondering why this library is the default rather than Apache SSHD (which was used for some other SSH functionality in Jenkins). I see there's support for the old pem file format and the new openssh (RFC 4716) format, but no explicit support for PKCS#8. Could you try out the other formats for the same key to see if it works? If it doesn't, then I think we have some missing crypto primitives here.

          Charles Smith added a comment -

          jvz

          Thanks for the new information.

          Just to confirm with you:

          1.) Do you want me to use the ssh-keygen or openssl command to generate the public key? I've already created the key.pem file with the openssl genrsa command?

          2.) Do you want me to run openssl rsa -in private.key -out private.pem -outform PEM to change the private key format to pem and then extract the public key from the converted pem file?

           

          Sorry I want to make sure that I do this right.

          Charles Smith added a comment - jvz Thanks for the new information. Just to confirm with you: 1.) Do you want me to use the ssh-keygen or openssl command to generate the public key? I've already created the  key.pem file with the openssl genrsa command? 2.) Do you want me to run openssl rsa -in private.key -out private.pem -outform PEM to change the private key format to pem and then extract the public key from the converted pem file?   Sorry I want to make sure that I do this right.

          Matt Sicker added a comment -

          Take your original private key (or any of the private keys you've created through ssh-keygen) and use the openssl command to convert its format to PEM. Then try that PEM key in Jenkins.

          Matt Sicker added a comment - Take your original private key (or any of the private keys you've created through ssh-keygen ) and use the openssl command to convert its format to PEM. Then try that PEM key in Jenkins.

          Charles Smith added a comment -

          jvz

          I'm unable to get this to work using the 61356-key private key.

          To replicate:

          I ran the following to convert the key to PEM format(61356-key.pem):

          jenkins@dbis-admin:/var/lib/jenkins/.ssh/test
          $ openssl rsa -in 61356-key -out 61356-key.pem -outform PEM
          writing RSA key
          
          jenkins@dbis-admin:/var/lib/jenkins/.ssh/test
          $ ls -l
          total 8
          -rw-------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key
          -rw-------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key.pem
          
          

          Then, copied(selected) the new key in vi format and pasted it into the SSH credential key box. I launch the build agent and get the same PEM error. Both the original private key and pem file are the same exact (i.e. no changes to the BEGIN PRIVATE KEY headers.)

          Charles Smith added a comment - jvz I'm unable to get this to work using the 61356-key private key. To replicate: I ran the following to convert the key to PEM format( 61356-key.pem ): jenkins@dbis-admin:/ var /lib/jenkins/.ssh/test $ openssl rsa -in 61356-key -out 61356-key.pem -outform PEM writing RSA key jenkins@dbis-admin:/ var /lib/jenkins/.ssh/test $ ls -l total 8 -rw-------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key -rw-------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key.pem Then, copied(selected) the new key in vi format and pasted it into the SSH credential key box. I launch the build agent and get the same PEM error. Both the original private key and pem file are the same exact ( i.e. no changes to the BEGIN PRIVATE KEY headers. )

          Matt Sicker added a comment -

          Well I'll be damned. Seems like this fips version just doesn't let you use the old format. Could you try exporting it in the RFC4716 format? ssh-keygen -f 61356-key -p -m RFC4716 and save without a password. Check the updated key file and make sure it has a "BEGIN OPENSSH PRIVATE KEY" header instead of "BEGIN PRIVATE KEY". If it still has the BEGIN PRIVATE KEY, then it really seems locked down to a single key format, period.

          Matt Sicker added a comment - Well I'll be damned. Seems like this fips version just doesn't let you use the old format. Could you try exporting it in the RFC4716 format? ssh-keygen -f 61356-key -p -m RFC4716 and save without a password. Check the updated key file and make sure it has a "BEGIN OPENSSH PRIVATE KEY" header instead of "BEGIN PRIVATE KEY". If it still has the BEGIN PRIVATE KEY, then it really seems locked down to a single key format, period.

          Charles Smith added a comment -

          jvz

          Yes it's confirmed. The 61356-key still has the "BEGIN PRIVATE KEY" header after attempting to change the format to RFC4716:

          
          

          $ ls -l
          total 8
          rw------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key
          rw------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key.pem

          $ grep "BEGIN\ PRIVATE\ KEY" 61356-key
          ----BEGIN PRIVATE KEY----

          $ date
          Thu Apr 23 22:58:12 EDT 2020

          $ ssh-keygen -f 61356-key -p -m RFC4716
          Enter new passphrase (empty for no passphrase):
          Enter same passphrase again:
          Your identification has been saved with the new passphrase.

          $ ls -l
          total 8
          rw------. 1 jenkins jenkins 3272 Apr 23 22:58 61356-key
          rw------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key.pem

          $ grep "BEGIN\ PRIVATE\ KEY" 61356-key
          ----BEGIN PRIVATE KEY----

          ${code}
           

          Charles Smith added a comment - jvz Yes it's confirmed. The  61356-key still has the " BEGIN PRIVATE KEY " header after attempting to change the format to RFC4716: $ ls -l total 8 rw ------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key rw ------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key.pem $ grep "BEGIN\ PRIVATE\ KEY" 61356-key ---- BEGIN PRIVATE KEY ---- $ date Thu Apr 23 22:58:12 EDT 2020 $ ssh-keygen -f 61356-key -p -m RFC4716 Enter new passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved with the new passphrase. $ ls -l total 8 rw ------. 1 jenkins jenkins 3272 Apr 23 22:58 61356-key rw ------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key.pem $ grep "BEGIN\ PRIVATE\ KEY" 61356-key ---- BEGIN PRIVATE KEY ---- ${code}  

          Ivan Fernandez Calvo added a comment - - edited

          I make a test with the original keys, then I've converted the original key with openssl and the issue is resolved.
          Here is the environment I made for the tests https://github.com/kuisathaverat/jenkins-issues/tree/master/JENKINS-61356

          This are the steps to start the environment

          git clone https://github.com/kuisathaverat/jenkins-issues.git
          cd jenkins-issues/JENKINS-61356
          make clean start
          

          The original key is in https://github.com/kuisathaverat/jenkins-issues/blob/master/JENKINS-61356/ssh-agent/ssh/rsa-key.org

          I've converted it with the following commands, then I've updated my JCasC configuration and everything works as expected.

          openssl version
          LibreSSL 2.8.3
          
          openssl rsa -in rsa-key.org -out rsa-key
          

          The PKCS#8 (BEGIN PRIVATE KEY) is not supported. There are other formats supported see https://github.com/jenkinsci/trilead-ssh2/tree/master/test/com/trilead/ssh2/signature

          Ivan Fernandez Calvo added a comment - - edited I make a test with the original keys, then I've converted the original key with openssl and the issue is resolved. Here is the environment I made for the tests https://github.com/kuisathaverat/jenkins-issues/tree/master/JENKINS-61356 This are the steps to start the environment git clone https: //github.com/kuisathaverat/jenkins-issues.git cd jenkins-issues/JENKINS-61356 make clean start The original key is in https://github.com/kuisathaverat/jenkins-issues/blob/master/JENKINS-61356/ssh-agent/ssh/rsa-key.org I've converted it with the following commands, then I've updated my JCasC configuration and everything works as expected. openssl version LibreSSL 2.8.3 openssl rsa -in rsa-key.org -out rsa-key The PKCS#8 (BEGIN PRIVATE KEY) is not supported. There are other formats supported see https://github.com/jenkinsci/trilead-ssh2/tree/master/test/com/trilead/ssh2/signature

          Charles Smith added a comment -

          ifernandezcalvo

          That's the problem the key never gets converted when I run those openssl commands. I noticed that you haven't tested yet with the attached 61356-key keypairs that I have attached to this ticket. Try your test on that keypair as I cannot get it to convert.

          # openssl version
          OpenSSL 1.0.2k-fips  26 Jan 2017
          

          Charles Smith added a comment - ifernandezcalvo That's the problem the key never gets converted when I run those openssl commands. I noticed that you haven't tested yet with the attached  61356-key keypairs that I have attached to this ticket. Try your test on that keypair as I cannot get it to convert. # openssl version OpenSSL 1.0.2k-fips 26 Jan 2017

          Matt Sicker added a comment - - edited

          I've attached the 61356 key converted into the ssh2 format. It's the newer version of the same file.

          Matt Sicker added a comment - - edited I've attached the 61356 key converted into the ssh2 format. It's the newer version of the same file.

          Ivan Fernandez Calvo added a comment - I tested the files uploaded 3 days ago https://issues.jenkins-ci.org/secure/attachment/51008/61356-key.pub.txt and https://issues.jenkins-ci.org/secure/attachment/51007/61356-key.txt

          Ivan Fernandez Calvo added a comment - - edited

          I just updated the test repo with three agents using the three different formats we have in the issue:

          • "----BEGIN PRIVATE KEY----" failure
          • "----BEGIN OPENSSH PRIVATE KEY----" success
          • "----BEGIN RSA PRIVATE KEY----" success

          Ivan Fernandez Calvo added a comment - - edited I just updated the test repo with three agents using the three different formats we have in the issue: "---- BEGIN PRIVATE KEY ----" failure "---- BEGIN OPENSSH PRIVATE KEY ----" success "---- BEGIN RSA PRIVATE KEY ----" success

          Matt Sicker added a comment -

          Alright, then I think we've isolated the issue. trilead doesn't support PKCS#8 private keys.

          Matt Sicker added a comment - Alright, then I think we've isolated the issue. trilead doesn't support PKCS#8 private keys.

          Charles Smith added a comment -

          When I use your converted 61356 key to launch the build agent, I now receive a different error:

          SSHLauncher{host='prod-slave-01.tt.com', port=22, credentialsId='jenkins-test-ssh-key', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=60, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
          [04/24/20 11:13:58] [SSH] Opening SSH connection to dbis-slave-01.uscis.dhs.gov:22.
          [04/24/20 11:13:58] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
          ERROR: Server rejected the 1 private key(s) for jenkins  (credentialId:jenkins-test-ssh-key/method:publickey)
          [04/24/20 11:13:59] [SSH] Authentication failed.
          Authentication failed.
          [04/24/20 11:13:59] Launch failed - cleaning up connection
          [04/24/20 11:13:59] [SSH] Connection closed.
          

          However, I'm able to successfully SSH into the build agent from the jenkins master shell using the same converted key. Which openssl command did you use to confirm that the privkeys are PKCS#8?

          Charles Smith added a comment - When I use your converted 61356 key to launch the build agent, I now receive a different error: SSHLauncher{host= 'prod-slave-01.tt.com' , port=22, credentialsId= 'jenkins-test-ssh-key' , jvmOptions= '', javaPath=' ', prefixStartSlaveCmd=' ', suffixStartSlaveCmd=' ', launchTimeoutSeconds=60, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay= true , trackCredentials= true } [04/24/20 11:13:58] [SSH] Opening SSH connection to dbis-slave-01.uscis.dhs.gov:22. [04/24/20 11:13:58] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed. ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:jenkins-test-ssh-key/method:publickey) [04/24/20 11:13:59] [SSH] Authentication failed. Authentication failed. [04/24/20 11:13:59] Launch failed - cleaning up connection [04/24/20 11:13:59] [SSH] Connection closed. However, I'm able to successfully SSH into the build agent from the jenkins master shell using the same converted key. Which openssl command did you use to confirm that the privkeys are PKCS#8?

          Matt Sicker added a comment -

          I converted the key using the command I pasted earlier:

          ssh-keygen -f 61356-key.txt -p -m RFC4716
          

          When prompted, I used an empty password. I tried using some other flags besides -p to convert the private key, but the other flags seem to only output the public key.

          And to confirm that the key is PKCS#8, you can tell because it has the header "BEGIN PRIVATE KEY" with no specific type of key mentioned in the header. That data is encoded in the base64 contents.

          Matt Sicker added a comment - I converted the key using the command I pasted earlier: ssh-keygen -f 61356-key.txt -p -m RFC4716 When prompted, I used an empty password. I tried using some other flags besides -p to convert the private key, but the other flags seem to only output the public key. And to confirm that the key is PKCS#8, you can tell because it has the header "BEGIN PRIVATE KEY" with no specific type of key mentioned in the header. That data is encoded in the base64 contents.

          Charles Smith added a comment -

          jvz

          Ok that command doesn't work on my system when trying to convert the key. Is this because of the fips enabled versions of SSH and SSL that's being used?

          Charles Smith added a comment - jvz Ok that command doesn't work on my system when trying to convert the key. Is this because of the fips enabled versions of SSH and SSL that's being used?

          Matt Sicker added a comment -

          I don't really know anything about fips, but it's certainly possible.

          Matt Sicker added a comment - I don't really know anything about fips, but it's certainly possible.

          Charles Smith added a comment -

          jvz

          As a last ditch effort for me, can you please create a new test ssh keypair that isn't PKCS#8? (i.e. - not converted key)?

          I would like to test one last time with the 2 keypairs:

          • "----BEGIN OPENSSH PRIVATE KEY----" success
          • "----BEGIN RSA PRIVATE KEY----" success

          I can't convert the key on my system to either of those formats.

          Charles Smith added a comment - jvz As a last ditch effort for me, can you please create a new test ssh keypair that isn't PKCS#8? ( i.e. - not converted key )? I would like to test one last time with the 2 keypairs: "---- BEGIN OPENSSH PRIVATE KEY ----" success "---- BEGIN RSA PRIVATE KEY ----" success I can't convert the key on my system to either of those formats.

          Matt Sicker added a comment -

          I've uploaded a tar.gz file with two newly generated RSA 4096 key pairs in both the legacy PEM and RFC4716 format from ssh-keygen.

          Matt Sicker added a comment - I've uploaded a tar.gz file with two newly generated RSA 4096 key pairs in both the legacy PEM and RFC4716 format from ssh-keygen .

          Charles Smith added a comment -

          jvz

          I created a new ssh credential and used the private key with "----BEGIN RSA PRIVATE KEY----"  and now the build agent successfully starts online:

           

          SSHLauncher{host='dbis-slave-01.uscis.dhs.gov', port=22, credentialsId='jenkins-rsa-test-ssh-key', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=60, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
          [04/24/20 13:47:11] [SSH] Opening SSH connection to dbis-slave-01.uscis.dhs.gov:22.
          [04/24/20 13:47:11] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
          [04/24/20 13:47:12] [SSH] Authentication successful.
          [04/24/20 13:47:12] [SSH] The remote user's environment is:
          BASH=/usr/bin/bash
          BASHOPTS=cmdhist:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
          BASH_ALIASES=()
          BASH_ARGC=()
          BASH_ARGV=()
          BASH_CMDS=()
          BASH_EXECUTION_STRING=set
          BASH_LINENO=()
          BASH_SOURCE=()
          BASH_VERSINFO=([0]="4" [1]="2" [2]="46" [3]="2" [4]="release" [5]="x86_64-redhat-linux-gnu")
          BASH_VERSION='4.2.46(2)-release'
          DIRSTACK=()
          EUID=987
          GROUPS=()
          HOME=/var/lib/jenkins
          HOSTNAME=dbis-slave-01.uscis.dhs.gov
          HOSTTYPE=x86_64
          IFS=$' \t\n'
          LOGNAME=jenkins
          MACHTYPE=x86_64-redhat-linux-gnu
          MAIL=/var/mail/jenkins
          OPTERR=1
          OPTIND=1
          OSTYPE=linux-gnu
          PATH=/usr/local/bin:/usr/bin
          PPID=27318
          PS4='+ '
          PWD=/var/lib/jenkins
          SELINUX_LEVEL_REQUESTED=
          SELINUX_ROLE_REQUESTED=
          SELINUX_USE_CURRENT_RANGE=
          SHELL=/bin/bash
          SHELLOPTS=braceexpand:hashall:interactive-comments
          SHLVL=1
          SSH_CLIENT='10.193.177.209 43519 22'
          SSH_CONNECTION='10.193.177.209 43519 10.193.177.232 22'
          TERM=dumb
          UID=987
          USER=jenkins
          XDG_RUNTIME_DIR=/run/user/987
          XDG_SESSION_ID=9633
          _=bash
          [04/24/20 13:47:12] [SSH] Checking java version of /var/lib/jenkins/jdk/bin/java
          Couldn't figure out the Java version of /var/lib/jenkins/jdk/bin/java
          bash: /var/lib/jenkins/jdk/bin/java: No such file or directory[04/24/20 13:47:12] [SSH] Checking java version of java
          [04/24/20 13:47:12] [SSH] java -version returned 1.8.0_252.
          [04/24/20 13:47:12] [SSH] Starting sftp client.
          [04/24/20 13:47:12] [SSH] Copying latest remoting.jar...
          [04/24/20 13:47:12] [SSH] Copied 1,522,914 bytes.
          Expanded the channel window size to 4MB
          [04/24/20 13:47:12] [SSH] Starting agent process: cd "/var/lib/jenkins" && java  -jar remoting.jar -workDir /var/lib/jenkins -jar-cache /var/lib/jenkins/remoting/jarCache
          Apr 24, 2020 12:47:13 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
          INFO: Using /var/lib/jenkins/remoting as a remoting work directory
          Apr 24, 2020 12:47:13 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
          INFO: Both error and output logs will be printed to /var/lib/jenkins/remoting
          <===[JENKINS REMOTING CAPACITY]===>channel started
          Remoting version: 4.2
          This is a Unix agent
          Apr 24, 2020 12:47:15 PM hudson.remoting.UserRequest perform
          WARNING: LinkageError while performing UserRequest:jenkins.slaves.StandardOutputSwapper$ChannelSwapper@16267446
          java.lang.UnsatisfiedLinkError: /tmp/resource-2287829487335364078/com/sun/jna/linux-x86-64/libjnidispatch.so: /tmp/resource-2287829487335364078/com/sun/jna/linux-x86-64/libjnidispatch.so: failed to map segment from shared object: Operation not permitted
          	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
          	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1934)
          	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1817)
          	at java.lang.Runtime.load0(Runtime.java:809)
          	at java.lang.System.load(System.java:1088)
          	at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1018)
          	at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988)
          	at com.sun.jna.Native.<clinit>(Native.java:195)
          	at hudson.util.jna.GNUCLibrary.<clinit>(GNUCLibrary.java:115)
          	at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.swap(StandardOutputSwapper.java:60)
          	at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:45)
          	at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:39)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
          	at hudson.remoting.Request$2.run(Request.java:369)
          	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	at java.lang.Thread.run(Thread.java:748)Agent successfully connected and online
          

          So the key with the "----BEGIN RSA PRIVATE KEY----" seems to work, but it does give a java error:

          java.lang.UnsatisfiedLinkError: /tmp/resource-2287829487335364078/com/sun/jna/linux-x86-64/libjnidispatch.so: /tmp/resource-2287829487335364078/com/sun/jna/linux-x86-64/libjnidispatch.so: failed to map segment from shared object: Operation not permitted
          

          Can you tell me what command was used to create that key if different than previously specified? I'll need to somehow figure out how to generate this type of keypair on my system. 

           

           

          Charles Smith added a comment - jvz I created a new ssh credential and used the private key with "---- BEGIN RSA PRIVATE KEY ----"  and now the build agent successfully starts online:   SSHLauncher{host= 'dbis-slave-01.uscis.dhs.gov' , port=22, credentialsId= 'jenkins-rsa-test-ssh-key' , jvmOptions= '', javaPath=' ', prefixStartSlaveCmd=' ', suffixStartSlaveCmd=' ', launchTimeoutSeconds=60, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay= true , trackCredentials= true } [04/24/20 13:47:11] [SSH] Opening SSH connection to dbis-slave-01.uscis.dhs.gov:22. [04/24/20 13:47:11] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed. [04/24/20 13:47:12] [SSH] Authentication successful. [04/24/20 13:47:12] [SSH] The remote user's environment is: BASH=/usr/bin/bash BASHOPTS=cmdhist:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath BASH_ALIASES=() BASH_ARGC=() BASH_ARGV=() BASH_CMDS=() BASH_EXECUTION_STRING=set BASH_LINENO=() BASH_SOURCE=() BASH_VERSINFO=([0]= "4" [1]= "2" [2]= "46" [3]= "2" [4]= "release" [5]= "x86_64-redhat-linux-gnu" ) BASH_VERSION= '4.2.46(2)-release' DIRSTACK=() EUID=987 GROUPS=() HOME=/ var /lib/jenkins HOSTNAME=dbis-slave-01.uscis.dhs.gov HOSTTYPE=x86_64 IFS=$ ' \t\n' LOGNAME=jenkins MACHTYPE=x86_64-redhat-linux-gnu MAIL=/ var /mail/jenkins OPTERR=1 OPTIND=1 OSTYPE=linux-gnu PATH=/usr/local/bin:/usr/bin PPID=27318 PS4= '+ ' PWD=/ var /lib/jenkins SELINUX_LEVEL_REQUESTED= SELINUX_ROLE_REQUESTED= SELINUX_USE_CURRENT_RANGE= SHELL=/bin/bash SHELLOPTS=braceexpand:hashall:interactive-comments SHLVL=1 SSH_CLIENT= '10.193.177.209 43519 22' SSH_CONNECTION= '10.193.177.209 43519 10.193.177.232 22' TERM=dumb UID=987 USER=jenkins XDG_RUNTIME_DIR=/run/user/987 XDG_SESSION_ID=9633 _=bash [04/24/20 13:47:12] [SSH] Checking java version of / var /lib/jenkins/jdk/bin/java Couldn't figure out the Java version of / var /lib/jenkins/jdk/bin/java bash: / var /lib/jenkins/jdk/bin/java: No such file or directory[04/24/20 13:47:12] [SSH] Checking java version of java [04/24/20 13:47:12] [SSH] java -version returned 1.8.0_252. [04/24/20 13:47:12] [SSH] Starting sftp client. [04/24/20 13:47:12] [SSH] Copying latest remoting.jar... [04/24/20 13:47:12] [SSH] Copied 1,522,914 bytes. Expanded the channel window size to 4MB [04/24/20 13:47:12] [SSH] Starting agent process: cd "/ var /lib/jenkins" && java -jar remoting.jar -workDir / var /lib/jenkins -jar-cache / var /lib/jenkins/remoting/jarCache Apr 24, 2020 12:47:13 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using / var /lib/jenkins/remoting as a remoting work directory Apr 24, 2020 12:47:13 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to / var /lib/jenkins/remoting <===[JENKINS REMOTING CAPACITY]===>channel started Remoting version: 4.2 This is a Unix agent Apr 24, 2020 12:47:15 PM hudson.remoting.UserRequest perform WARNING: LinkageError while performing UserRequest:jenkins.slaves.StandardOutputSwapper$ChannelSwapper@16267446 java.lang.UnsatisfiedLinkError: /tmp/resource-2287829487335364078/com/sun/jna/linux-x86-64/libjnidispatch.so: /tmp/resource-2287829487335364078/com/sun/jna/linux-x86-64/libjnidispatch.so: failed to map segment from shared object: Operation not permitted at java.lang. ClassLoader $NativeLibrary.load(Native Method) at java.lang. ClassLoader .loadLibrary0( ClassLoader .java:1934) at java.lang. ClassLoader .loadLibrary( ClassLoader .java:1817) at java.lang. Runtime .load0( Runtime .java:809) at java.lang. System .load( System .java:1088) at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1018) at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988) at com.sun.jna.Native.<clinit>(Native.java:195) at hudson.util.jna.GNUCLibrary.<clinit>(GNUCLibrary.java:115) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.swap(StandardOutputSwapper.java:60) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:45) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:39) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748)Agent successfully connected and online So the key with the "---- BEGIN RSA PRIVATE KEY ----" seems to work, but it does give a java error: java.lang.UnsatisfiedLinkError: /tmp/resource-2287829487335364078/com/sun/jna/linux-x86-64/libjnidispatch.so: /tmp/resource-2287829487335364078/com/sun/jna/linux-x86-64/libjnidispatch.so: failed to map segment from shared object: Operation not permitted Can you tell me what command was used to create that key if different than previously specified? I'll need to somehow figure out how to generate this type of keypair on my system.     

          Matt Sicker added a comment -

          Commands I used to generate keys:

          ssh-keygen -f pemkey -m PEM -t rsa -b 4096
          ssh-keygen -f ssh2key -m RFC4716 -t rsa -b 4096
          

          As for the linkage error you got, that looks like a seccomp error I think? Never seen that before.

          Matt Sicker added a comment - Commands I used to generate keys: ssh-keygen -f pemkey -m PEM -t rsa -b 4096 ssh-keygen -f ssh2key -m RFC4716 -t rsa -b 4096 As for the linkage error you got, that looks like a seccomp error I think? Never seen that before.

          Matt Sicker added a comment -

          I've updated the original ticket to better reflect our understanding of the underlying issue.

          Matt Sicker added a comment - I've updated the original ticket to better reflect our understanding of the underlying issue.

          For me is not a critical issue, there are several formats supported, those formats can be generated with any modern version of OpenSSL or OpenSSH. The issue of the user is thigh attached to the restrictions and versions they use in their environment, so the overall impact on users is minimum and has a solution in most of the cases.
          I've documented the issue on the troubleshooting guide and in the configuration documents.

          Ivan Fernandez Calvo added a comment - For me is not a critical issue, there are several formats supported, those formats can be generated with any modern version of OpenSSL or OpenSSH. The issue of the user is thigh attached to the restrictions and versions they use in their environment, so the overall impact on users is minimum and has a solution in most of the cases. I've documented the issue on the troubleshooting guide and in the configuration documents.

          Matt Sicker added a comment - - edited

          Makes sense. I've updated it to minor. I've also started talking with some other Jenkins developers to try to find out whether or not we can migrate to Apache MINA SSH as our SSH client library as it's still actively developed. That may or may not be relevant in the future as default OpenSSH ciphers change over time. For example, there's no AES/GCM support, and I don't remember if ChaCha20-Poly1305 is already supported or not. Then there will be updated key exchanges and all the other supporting primitives. Not sure how much of that we want to continue maintaining if we can use an OSS dependency with a community around it.

          Matt Sicker added a comment - - edited Makes sense. I've updated it to minor. I've also started talking with some other Jenkins developers to try to find out whether or not we can migrate to Apache MINA SSH as our SSH client library as it's still actively developed. That may or may not be relevant in the future as default OpenSSH ciphers change over time. For example, there's no AES/GCM support, and I don't remember if ChaCha20-Poly1305 is already supported or not. Then there will be updated key exchanges and all the other supporting primitives. Not sure how much of that we want to continue maintaining if we can use an OSS dependency with a community around it.

          Matt Sicker added a comment -

          Adding this to the Mina SSH epic as it supports more key types, so this should get swept up as part of the migration.

          Matt Sicker added a comment - Adding this to the Mina SSH epic as it supports more key types, so this should get swept up as part of the migration.

          As workaround is possible to convert PKCS#8 format to PKCS#1, the key is stored encrypted in the Jenkins Controller so input format is not really relevant.

          openssl rsa -in private_pkcs8.pem -out private_pkcs1.pem -traditional

          Ivan Fernandez Calvo added a comment - As workaround is possible to convert PKCS#8 format to PKCS#1, the key is stored encrypted in the Jenkins Controller so input format is not really relevant. openssl rsa -in private_pkcs8.pem -out private_pkcs1.pem -traditional

            ifernandezcalvo Ivan Fernandez Calvo
            clsmith4 Charles Smith
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: