Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61356

Trilead SSH does not support PKCS#8 encoded private keys

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • trilead-api-plugin

      The Jenkins SSH agent plugin uses trilead-ssh which only supports the legacy PEM and new SSH2 private key file formats. In certain hardened environments, SSH keys are mandated to be encoded in PKCS#8 (presumably because they're more universal and allow for password protection and encrypting the key) to the point where OpenSSH and ssh-keygen are patched to only support PKCS#8 private keys.

      Attached to this ticket are various encodings of private keys as we troubleshooted the initial issue described below. Note that private keys that have a header line with BEGIN PRIVATE KEY are indicative of the file being encoded in PKCS#8. If the file header contains BEGIN RSA PRIVATE KEY (or EC or whatever other algorithm), that is the legacy PEM format. If the file header contains BEGIN OPENSSH PRIVATE KEY, then that is the new SSH2 format. The latter two encoding formats are already supported in trilead-ssh.

      Original Details

      • My Jenkins exists on an Amazon EC2 instance.
      • I'm trying to add Red Hat 7.7 slave via ssh.
      • I can ssh directly using terminal from my master to slave and vice versa using that key file
      jenkins@master:/var/lib/jenkins
      $ ssh -i .ssh/id_rsa -q 10.193.177.232
      
      jenkins@slave:/var/lib/jenkins
      $ ssh -i .ssh/id_rsa -q 10.193.177.209
      

      I’m also able to make a successful SSH connection when I perform a SSH connection test from the Manage Jenkins -> Configure System section of the Master (FYI - see attached screenshot).

      But when I try to configure my slave via launch agent method, it gives me the following error:

      caused by: java.io.IOException: PEM problem: it is of unknown type
              at com.trilead.ssh2.crypto.PEMDecoder.decpdeKeyPair(PEMDecoder.java:500)
      
      • How I've configured the slave (one of the method)?
        • I've saved slave machine credentials as "ssh key username with private key"
        • Launch method: Launch agents via SSH
        • Hostname: slave machine private ip address.
        • Host Key Verification Strategy: Manually trusted key Verification Strategy

      Note:

      • I cannot regenerate ssh keys as I'll not be able to access my machines again and the infrastructure is very strict. 
      • All SSH key are 4096 bits in length and RSA.
      • I cannot use credentials as username/password as either there is no password.
      • I've tried everything whatever mentioned in other jenkins tickets for same issue or on internet, but my issue is not getting resolved.

      Please see the following attachments for log details and screenshots.

       

      This is a total blocker. Please advise. Thank you.

        1. 61356.tar.gz
          6 kB
        2. Screenshot 2020-04-24 at 17.22.03.png
          Screenshot 2020-04-24 at 17.22.03.png
          66 kB
        3. 61356-key.txt
          3 kB
        4. 61356-key.pub.txt
          0.7 kB
        5. 61356-key.txt
          3 kB
        6. script-console-output.PNG
          script-console-output.PNG
          53 kB
        7. Unable to launch agent using plugin-ver.1.17.4.PNG
          Unable to launch agent using plugin-ver.1.17.4.PNG
          24 kB
        8. ver1.17.4.PNG
          ver1.17.4.PNG
          24 kB
        9. new privkey.txt
          2 kB
        10. ssh-cred.PNG
          ssh-cred.PNG
          4 kB
        11. com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.txt.txt
          6 kB
        12. Launch Agent error output.PNG
          Launch Agent error output.PNG
          88 kB
        13. Slave screenshot.PNG
          Slave screenshot.PNG
          54 kB
        14. SSH test from Master with the same SSH credential.PNG
          SSH test from Master with the same SSH credential.PNG
          23 kB
        15. slave-log.log
          2 kB

          [JENKINS-61356] Trilead SSH does not support PKCS#8 encoded private keys

          Charles Smith created issue -
          Charles Smith made changes -
          Attachment New: Launch Agent error output.PNG [ 50573 ]
          Attachment New: Slave screenshot.PNG [ 50574 ]
          Attachment New: SSH test from Master with the same SSH credential.PNG [ 50575 ]
          Attachment New: slave-log.log [ 50576 ]
          Charles Smith made changes -
          Description Original: * My Jenkins exists on an Amazon EC2 instance.
           * I'm trying to add Red Hat 7.7 slave via ssh.
           * While creating an instance, a <filename>.pem file is generated for storing .ssh keys which is a private key. No separate pub ssh key is there. 
           * I can ssh directly using terminal from my master to slave and vice versa using that key file

          {code:java}
          jenkins@master:/var/lib/jenkins
          $ ssh -i .ssh/id_rsa -q 10.193.177.232

          jenkins@slave:/var/lib/jenkins
          $ ssh -i .ssh/id_rsa -q 10.193.177.209
          {code}
          I’m also able to make a successful SSH connection when I perform a SSH connection test from the *Manage Jenkins -> Configure System* section of the Master (_FYI - see attached screenshot_).

          But when I try to configure my slave via launch agent method, it gives me the following error:
          {code:java}
          caused by: java.io.IOException: PEM problem: it is of unknown type
                  at com.trilead.ssh2.crypto.PEMDecoder.decpdeKeyPair(PEMDecoder.java:500)
          {code}
           * How I've configured the slave (one of the method)?
           ** I've saved slave machine credentials as "*ssh key username with private key*"
           ** Launch method: Launch agents via SSH
           ** Hostname: slave machine private ip address.
           ** Host Key Verification Strategy: *Manually trusted key Verification Strategy*

          Note:
           * I cannot regenerate ssh keys as I'll not be able to access my machines again and the infrastructure is very strict. 
           * All SSH key are 4096 bits in length and RSA.
           * I cannot use credentials as username/password as either there is no password.
           * I've tried everything whatever mentioned in other jenkins tickets for same issue or on internet, but my issue is not getting resolved.

          Please see the following attachments for log details and screenshots.

           

          This is a total blocker. Please advise. Thank you.
          New: * My Jenkins exists on an Amazon EC2 instance.
           * I'm trying to add Red Hat 7.7 slave via ssh.
           * I can ssh directly using terminal from my master to slave and vice versa using that key file

          {code:java}
          jenkins@master:/var/lib/jenkins
          $ ssh -i .ssh/id_rsa -q 10.193.177.232

          jenkins@slave:/var/lib/jenkins
          $ ssh -i .ssh/id_rsa -q 10.193.177.209
          {code}
          I’m also able to make a successful SSH connection when I perform a SSH connection test from the *Manage Jenkins -> Configure System* section of the Master (_FYI - see attached screenshot_).

          But when I try to configure my slave via launch agent method, it gives me the following error:
          {code:java}
          caused by: java.io.IOException: PEM problem: it is of unknown type
                  at com.trilead.ssh2.crypto.PEMDecoder.decpdeKeyPair(PEMDecoder.java:500)
          {code}
           * How I've configured the slave (one of the method)?
           ** I've saved slave machine credentials as "*ssh key username with private key*"
           ** Launch method: Launch agents via SSH
           ** Hostname: slave machine private ip address.
           ** Host Key Verification Strategy: *Manually trusted key Verification Strategy*

          Note:
           * I cannot regenerate ssh keys as I'll not be able to access my machines again and the infrastructure is very strict. 
           * All SSH key are 4096 bits in length and RSA.
           * I cannot use credentials as username/password as either there is no password.
           * I've tried everything whatever mentioned in other jenkins tickets for same issue or on internet, but my issue is not getting resolved.

          Please see the following attachments for log details and screenshots.

           

          This is a total blocker. Please advise. Thank you.
          Charles Smith made changes -
          Priority Original: Major [ 3 ] New: Critical [ 2 ]

          Charles Smith added a comment -

          jvz

          Hello Matt, can you give me some type of update on this issue? I've tried all the recommendations that I could find on this site and on google with no success. If you require anymore info please let me know. Thanks.

          Charles Smith added a comment - jvz Hello Matt, can you give me some type of update on this issue? I've tried all the recommendations that I could find on this site and on google with no success. If you require anymore info please let me know. Thanks.

          Matt Sicker added a comment -

          Did this issue appear in a particular version of the plugin? Or do you have a test that demonstrates the issue?

          Matt Sicker added a comment - Did this issue appear in a particular version of the plugin? Or do you have a test that demonstrates the issue?

          Charles Smith added a comment -

          jvz

          Hi Matt, This is the first build agent that has been added to our Jenkins instance. We are running the latest plugins and Jenkins version. As far as a test, all you have to do is create a new build agent and attempt to launch it and it produces the PEM error. Please let me know if you need anymore information apart from what has already been provided.

          Charles Smith added a comment - jvz Hi Matt, This is the first build agent that has been added to our Jenkins instance. We are running the latest plugins and Jenkins version. As far as a test, all you have to do is create a new build agent and attempt to launch it and it produces the PEM error. Please let me know if you need anymore information apart from what has already been provided.

          Matt Sicker added a comment -

          Take a look at this test: https://github.com/jenkinsci/ssh-credentials-plugin/blob/master/src/test/java/com/cloudbees/jenkins/plugins/sshcredentials/impl/TrileadSSHPublicKeyAuthenticatorTest.java

          If you run that test locally but instead put the contents of your private key in the getPrivateKey() method, does the test still pass?

          Matt Sicker added a comment - Take a look at this test: https://github.com/jenkinsci/ssh-credentials-plugin/blob/master/src/test/java/com/cloudbees/jenkins/plugins/sshcredentials/impl/TrileadSSHPublicKeyAuthenticatorTest.java If you run that test locally but instead put the contents of your private key in the getPrivateKey() method, does the test still pass?

          Charles Smith added a comment -

          jvz

          My apologies Matt, but I'm having a lot of trouble trying to compile that java program before running it 'TrileadSSHPublicKeyAuthenticatorTest.java'. Every time I attempt to compile the program it errors out with the following:

          javac TrileadSSHPublicKeyAuthenticatorTest.java
          symbol:   method assertNotNull(Object)
            location: class TrileadSSHPublicKeyAuthenticatorTest
          TrileadSSHPublicKeyAuthenticatorTest.java:343: error: cannot find symbol
                  assertNotNull(factory);
                  ^
            symbol:   method assertNotNull(Object)
            location: class TrileadSSHPublicKeyAuthenticatorTest
          Note: TrileadSSHPublicKeyAuthenticatorTest.java uses unchecked or unsafe operations.
          Note: Recompile with -Xlint:unchecked for details.
          68 errors
          1 warning
          
          

          Can you let me know if I'm doing this correctly? Or is there an easier way that I can test this program and the getPrivateKey() method? I don't have access to an IDE so will need to run this test from the server manually with javac and java commands.

          Charles Smith added a comment - jvz My apologies Matt, but I'm having a lot of trouble trying to compile that java program before running it ' TrileadSSHPublicKeyAuthenticatorTest.java '. Every time I attempt to compile the program it errors out with the following: javac TrileadSSHPublicKeyAuthenticatorTest.java symbol: method assertNotNull( Object ) location: class TrileadSSHPublicKeyAuthenticatorTest TrileadSSHPublicKeyAuthenticatorTest.java:343: error: cannot find symbol assertNotNull(factory); ^ symbol: method assertNotNull( Object ) location: class TrileadSSHPublicKeyAuthenticatorTest Note: TrileadSSHPublicKeyAuthenticatorTest.java uses unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. 68 errors 1 warning Can you let me know if I'm doing this correctly? Or is there an easier way that I can test this program and the getPrivateKey() method? I don't have access to an IDE so will need to run this test from the server manually with javac and java commands.

          Matt Sicker added a comment -

          You'll need to git clone the repo, edit that file to change the key to your key, then you can run mvn test -Dtest=TrileadSSHPublicKeyAuthenticatorTest to run just that test.

          Matt Sicker added a comment - You'll need to git clone the repo, edit that file to change the key to your key, then you can run mvn test -Dtest=TrileadSSHPublicKeyAuthenticatorTest to run just that test.

            ifernandezcalvo Ivan Fernandez Calvo
            clsmith4 Charles Smith
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: