Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61356

Trilead SSH does not support PKCS#8 encoded private keys

    XMLWordPrintable

Details

    • Improvement
    • Status: Open (View Workflow)
    • Minor
    • Resolution: Unresolved
    • trilead-api-plugin

    Description

      The Jenkins SSH agent plugin uses trilead-ssh which only supports the legacy PEM and new SSH2 private key file formats. In certain hardened environments, SSH keys are mandated to be encoded in PKCS#8 (presumably because they're more universal and allow for password protection and encrypting the key) to the point where OpenSSH and ssh-keygen are patched to only support PKCS#8 private keys.

      Attached to this ticket are various encodings of private keys as we troubleshooted the initial issue described below. Note that private keys that have a header line with BEGIN PRIVATE KEY are indicative of the file being encoded in PKCS#8. If the file header contains BEGIN RSA PRIVATE KEY (or EC or whatever other algorithm), that is the legacy PEM format. If the file header contains BEGIN OPENSSH PRIVATE KEY, then that is the new SSH2 format. The latter two encoding formats are already supported in trilead-ssh.

      Original Details

      • My Jenkins exists on an Amazon EC2 instance.
      • I'm trying to add Red Hat 7.7 slave via ssh.
      • I can ssh directly using terminal from my master to slave and vice versa using that key file
      jenkins@master:/var/lib/jenkins
      $ ssh -i .ssh/id_rsa -q 10.193.177.232
      
      jenkins@slave:/var/lib/jenkins
      $ ssh -i .ssh/id_rsa -q 10.193.177.209
      

      I’m also able to make a successful SSH connection when I perform a SSH connection test from the Manage Jenkins -> Configure System section of the Master (FYI - see attached screenshot).

      But when I try to configure my slave via launch agent method, it gives me the following error:

      caused by: java.io.IOException: PEM problem: it is of unknown type
              at com.trilead.ssh2.crypto.PEMDecoder.decpdeKeyPair(PEMDecoder.java:500)
      
      • How I've configured the slave (one of the method)?
        • I've saved slave machine credentials as "ssh key username with private key"
        • Launch method: Launch agents via SSH
        • Hostname: slave machine private ip address.
        • Host Key Verification Strategy: Manually trusted key Verification Strategy

      Note:

      • I cannot regenerate ssh keys as I'll not be able to access my machines again and the infrastructure is very strict. 
      • All SSH key are 4096 bits in length and RSA.
      • I cannot use credentials as username/password as either there is no password.
      • I've tried everything whatever mentioned in other jenkins tickets for same issue or on internet, but my issue is not getting resolved.

      Please see the following attachments for log details and screenshots.

       

      This is a total blocker. Please advise. Thank you.

      Attachments

        1. 61356.tar.gz
          6 kB
        2. 61356-key.pub.txt
          0.7 kB
        3. 61356-key.txt
          3 kB
        4. 61356-key.txt
          3 kB
        5. com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.txt.txt
          6 kB
        6. Launch Agent error output.PNG
          Launch Agent error output.PNG
          88 kB
        7. new privkey.txt
          2 kB
        8. Screenshot 2020-04-24 at 17.22.03.png
          Screenshot 2020-04-24 at 17.22.03.png
          66 kB
        9. script-console-output.PNG
          script-console-output.PNG
          53 kB
        10. slave-log.log
          2 kB
        11. Slave screenshot.PNG
          Slave screenshot.PNG
          54 kB
        12. ssh-cred.PNG
          ssh-cred.PNG
          4 kB
        13. SSH test from Master with the same SSH credential.PNG
          SSH test from Master with the same SSH credential.PNG
          23 kB
        14. Unable to launch agent using plugin-ver.1.17.4.PNG
          Unable to launch agent using plugin-ver.1.17.4.PNG
          24 kB
        15. ver1.17.4.PNG
          ver1.17.4.PNG
          24 kB

        Issue Links

          Activity

            jvz Matt Sicker added a comment -

            Commands I used to generate keys:

            ssh-keygen -f pemkey -m PEM -t rsa -b 4096
            ssh-keygen -f ssh2key -m RFC4716 -t rsa -b 4096
            

            As for the linkage error you got, that looks like a seccomp error I think? Never seen that before.

            jvz Matt Sicker added a comment - Commands I used to generate keys: ssh-keygen -f pemkey -m PEM -t rsa -b 4096 ssh-keygen -f ssh2key -m RFC4716 -t rsa -b 4096 As for the linkage error you got, that looks like a seccomp error I think? Never seen that before.
            jvz Matt Sicker added a comment -

            I've updated the original ticket to better reflect our understanding of the underlying issue.

            jvz Matt Sicker added a comment - I've updated the original ticket to better reflect our understanding of the underlying issue.

            For me is not a critical issue, there are several formats supported, those formats can be generated with any modern version of OpenSSL or OpenSSH. The issue of the user is thigh attached to the restrictions and versions they use in their environment, so the overall impact on users is minimum and has a solution in most of the cases.
            I've documented the issue on the troubleshooting guide and in the configuration documents.

            ifernandezcalvo Ivan Fernandez Calvo added a comment - For me is not a critical issue, there are several formats supported, those formats can be generated with any modern version of OpenSSL or OpenSSH. The issue of the user is thigh attached to the restrictions and versions they use in their environment, so the overall impact on users is minimum and has a solution in most of the cases. I've documented the issue on the troubleshooting guide and in the configuration documents.
            jvz Matt Sicker added a comment - - edited

            Makes sense. I've updated it to minor. I've also started talking with some other Jenkins developers to try to find out whether or not we can migrate to Apache MINA SSH as our SSH client library as it's still actively developed. That may or may not be relevant in the future as default OpenSSH ciphers change over time. For example, there's no AES/GCM support, and I don't remember if ChaCha20-Poly1305 is already supported or not. Then there will be updated key exchanges and all the other supporting primitives. Not sure how much of that we want to continue maintaining if we can use an OSS dependency with a community around it.

            jvz Matt Sicker added a comment - - edited Makes sense. I've updated it to minor. I've also started talking with some other Jenkins developers to try to find out whether or not we can migrate to Apache MINA SSH as our SSH client library as it's still actively developed. That may or may not be relevant in the future as default OpenSSH ciphers change over time. For example, there's no AES/GCM support, and I don't remember if ChaCha20-Poly1305 is already supported or not. Then there will be updated key exchanges and all the other supporting primitives. Not sure how much of that we want to continue maintaining if we can use an OSS dependency with a community around it.
            jvz Matt Sicker added a comment -

            Adding this to the Mina SSH epic as it supports more key types, so this should get swept up as part of the migration.

            jvz Matt Sicker added a comment - Adding this to the Mina SSH epic as it supports more key types, so this should get swept up as part of the migration.

            People

              ifernandezcalvo Ivan Fernandez Calvo
              clsmith4 Charles Smith
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: