Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61356

Trilead SSH does not support PKCS#8 encoded private keys

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • trilead-api-plugin

      The Jenkins SSH agent plugin uses trilead-ssh which only supports the legacy PEM and new SSH2 private key file formats. In certain hardened environments, SSH keys are mandated to be encoded in PKCS#8 (presumably because they're more universal and allow for password protection and encrypting the key) to the point where OpenSSH and ssh-keygen are patched to only support PKCS#8 private keys.

      Attached to this ticket are various encodings of private keys as we troubleshooted the initial issue described below. Note that private keys that have a header line with BEGIN PRIVATE KEY are indicative of the file being encoded in PKCS#8. If the file header contains BEGIN RSA PRIVATE KEY (or EC or whatever other algorithm), that is the legacy PEM format. If the file header contains BEGIN OPENSSH PRIVATE KEY, then that is the new SSH2 format. The latter two encoding formats are already supported in trilead-ssh.

      Original Details

      • My Jenkins exists on an Amazon EC2 instance.
      • I'm trying to add Red Hat 7.7 slave via ssh.
      • I can ssh directly using terminal from my master to slave and vice versa using that key file
      jenkins@master:/var/lib/jenkins
      $ ssh -i .ssh/id_rsa -q 10.193.177.232
      
      jenkins@slave:/var/lib/jenkins
      $ ssh -i .ssh/id_rsa -q 10.193.177.209
      

      I’m also able to make a successful SSH connection when I perform a SSH connection test from the Manage Jenkins -> Configure System section of the Master (FYI - see attached screenshot).

      But when I try to configure my slave via launch agent method, it gives me the following error:

      caused by: java.io.IOException: PEM problem: it is of unknown type
              at com.trilead.ssh2.crypto.PEMDecoder.decpdeKeyPair(PEMDecoder.java:500)
      
      • How I've configured the slave (one of the method)?
        • I've saved slave machine credentials as "ssh key username with private key"
        • Launch method: Launch agents via SSH
        • Hostname: slave machine private ip address.
        • Host Key Verification Strategy: Manually trusted key Verification Strategy

      Note:

      • I cannot regenerate ssh keys as I'll not be able to access my machines again and the infrastructure is very strict. 
      • All SSH key are 4096 bits in length and RSA.
      • I cannot use credentials as username/password as either there is no password.
      • I've tried everything whatever mentioned in other jenkins tickets for same issue or on internet, but my issue is not getting resolved.

      Please see the following attachments for log details and screenshots.

       

      This is a total blocker. Please advise. Thank you.

        1. ver1.17.4.PNG
          24 kB
          Charles Smith
        2. Unable to launch agent using plugin-ver.1.17.4.PNG
          24 kB
          Charles Smith
        3. SSH test from Master with the same SSH credential.PNG
          23 kB
          Charles Smith
        4. ssh-cred.PNG
          4 kB
          Charles Smith
        5. Slave screenshot.PNG
          54 kB
          Charles Smith
        6. slave-log.log
          2 kB
          Charles Smith
        7. script-console-output.PNG
          53 kB
          Charles Smith
        8. Screenshot 2020-04-24 at 17.22.03.png
          66 kB
          Ivan Fernandez Calvo
        9. new privkey.txt
          2 kB
          Charles Smith
        10. Launch Agent error output.PNG
          88 kB
          Charles Smith
        11. com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.txt.txt
          6 kB
          Charles Smith
        12. 61356-key.txt
          3 kB
          Charles Smith
        13. 61356-key.txt
          3 kB
          Matt Sicker
        14. 61356-key.pub.txt
          0.7 kB
          Charles Smith
        15. 61356.tar.gz
          6 kB
          Matt Sicker

          [JENKINS-61356] Trilead SSH does not support PKCS#8 encoded private keys

          Matt Sicker added a comment -

          I don't really know anything about fips, but it's certainly possible.

          Matt Sicker added a comment - I don't really know anything about fips, but it's certainly possible.

          Charles Smith added a comment -

          jvz

          As a last ditch effort for me, can you please create a new test ssh keypair that isn't PKCS#8? (i.e. - not converted key)?

          I would like to test one last time with the 2 keypairs:

          • "----BEGIN OPENSSH PRIVATE KEY----" success
          • "----BEGIN RSA PRIVATE KEY----" success

          I can't convert the key on my system to either of those formats.

          Charles Smith added a comment - jvz As a last ditch effort for me, can you please create a new test ssh keypair that isn't PKCS#8? ( i.e. - not converted key )? I would like to test one last time with the 2 keypairs: "---- BEGIN OPENSSH PRIVATE KEY ----" success "---- BEGIN RSA PRIVATE KEY ----" success I can't convert the key on my system to either of those formats.

          Matt Sicker added a comment -

          I've uploaded a tar.gz file with two newly generated RSA 4096 key pairs in both the legacy PEM and RFC4716 format from ssh-keygen.

          Matt Sicker added a comment - I've uploaded a tar.gz file with two newly generated RSA 4096 key pairs in both the legacy PEM and RFC4716 format from ssh-keygen .

          Charles Smith added a comment -

          jvz

          I created a new ssh credential and used the private key with "----BEGIN RSA PRIVATE KEY----"  and now the build agent successfully starts online:

           

          SSHLauncher{host='dbis-slave-01.uscis.dhs.gov', port=22, credentialsId='jenkins-rsa-test-ssh-key', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=60, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
          [04/24/20 13:47:11] [SSH] Opening SSH connection to dbis-slave-01.uscis.dhs.gov:22.
          [04/24/20 13:47:11] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
          [04/24/20 13:47:12] [SSH] Authentication successful.
          [04/24/20 13:47:12] [SSH] The remote user's environment is:
          BASH=/usr/bin/bash
          BASHOPTS=cmdhist:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
          BASH_ALIASES=()
          BASH_ARGC=()
          BASH_ARGV=()
          BASH_CMDS=()
          BASH_EXECUTION_STRING=set
          BASH_LINENO=()
          BASH_SOURCE=()
          BASH_VERSINFO=([0]="4" [1]="2" [2]="46" [3]="2" [4]="release" [5]="x86_64-redhat-linux-gnu")
          BASH_VERSION='4.2.46(2)-release'
          DIRSTACK=()
          EUID=987
          GROUPS=()
          HOME=/var/lib/jenkins
          HOSTNAME=dbis-slave-01.uscis.dhs.gov
          HOSTTYPE=x86_64
          IFS=$' \t\n'
          LOGNAME=jenkins
          MACHTYPE=x86_64-redhat-linux-gnu
          MAIL=/var/mail/jenkins
          OPTERR=1
          OPTIND=1
          OSTYPE=linux-gnu
          PATH=/usr/local/bin:/usr/bin
          PPID=27318
          PS4='+ '
          PWD=/var/lib/jenkins
          SELINUX_LEVEL_REQUESTED=
          SELINUX_ROLE_REQUESTED=
          SELINUX_USE_CURRENT_RANGE=
          SHELL=/bin/bash
          SHELLOPTS=braceexpand:hashall:interactive-comments
          SHLVL=1
          SSH_CLIENT='10.193.177.209 43519 22'
          SSH_CONNECTION='10.193.177.209 43519 10.193.177.232 22'
          TERM=dumb
          UID=987
          USER=jenkins
          XDG_RUNTIME_DIR=/run/user/987
          XDG_SESSION_ID=9633
          _=bash
          [04/24/20 13:47:12] [SSH] Checking java version of /var/lib/jenkins/jdk/bin/java
          Couldn't figure out the Java version of /var/lib/jenkins/jdk/bin/java
          bash: /var/lib/jenkins/jdk/bin/java: No such file or directory[04/24/20 13:47:12] [SSH] Checking java version of java
          [04/24/20 13:47:12] [SSH] java -version returned 1.8.0_252.
          [04/24/20 13:47:12] [SSH] Starting sftp client.
          [04/24/20 13:47:12] [SSH] Copying latest remoting.jar...
          [04/24/20 13:47:12] [SSH] Copied 1,522,914 bytes.
          Expanded the channel window size to 4MB
          [04/24/20 13:47:12] [SSH] Starting agent process: cd "/var/lib/jenkins" && java  -jar remoting.jar -workDir /var/lib/jenkins -jar-cache /var/lib/jenkins/remoting/jarCache
          Apr 24, 2020 12:47:13 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
          INFO: Using /var/lib/jenkins/remoting as a remoting work directory
          Apr 24, 2020 12:47:13 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
          INFO: Both error and output logs will be printed to /var/lib/jenkins/remoting
          <===[JENKINS REMOTING CAPACITY]===>channel started
          Remoting version: 4.2
          This is a Unix agent
          Apr 24, 2020 12:47:15 PM hudson.remoting.UserRequest perform
          WARNING: LinkageError while performing UserRequest:jenkins.slaves.StandardOutputSwapper$ChannelSwapper@16267446
          java.lang.UnsatisfiedLinkError: /tmp/resource-2287829487335364078/com/sun/jna/linux-x86-64/libjnidispatch.so: /tmp/resource-2287829487335364078/com/sun/jna/linux-x86-64/libjnidispatch.so: failed to map segment from shared object: Operation not permitted
          	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
          	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1934)
          	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1817)
          	at java.lang.Runtime.load0(Runtime.java:809)
          	at java.lang.System.load(System.java:1088)
          	at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1018)
          	at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988)
          	at com.sun.jna.Native.<clinit>(Native.java:195)
          	at hudson.util.jna.GNUCLibrary.<clinit>(GNUCLibrary.java:115)
          	at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.swap(StandardOutputSwapper.java:60)
          	at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:45)
          	at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:39)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
          	at hudson.remoting.Request$2.run(Request.java:369)
          	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	at java.lang.Thread.run(Thread.java:748)Agent successfully connected and online
          

          So the key with the "----BEGIN RSA PRIVATE KEY----" seems to work, but it does give a java error:

          java.lang.UnsatisfiedLinkError: /tmp/resource-2287829487335364078/com/sun/jna/linux-x86-64/libjnidispatch.so: /tmp/resource-2287829487335364078/com/sun/jna/linux-x86-64/libjnidispatch.so: failed to map segment from shared object: Operation not permitted
          

          Can you tell me what command was used to create that key if different than previously specified? I'll need to somehow figure out how to generate this type of keypair on my system. 

           

           

          Charles Smith added a comment - jvz I created a new ssh credential and used the private key with "---- BEGIN RSA PRIVATE KEY ----"  and now the build agent successfully starts online:   SSHLauncher{host= 'dbis-slave-01.uscis.dhs.gov' , port=22, credentialsId= 'jenkins-rsa-test-ssh-key' , jvmOptions= '', javaPath=' ', prefixStartSlaveCmd=' ', suffixStartSlaveCmd=' ', launchTimeoutSeconds=60, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay= true , trackCredentials= true } [04/24/20 13:47:11] [SSH] Opening SSH connection to dbis-slave-01.uscis.dhs.gov:22. [04/24/20 13:47:11] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed. [04/24/20 13:47:12] [SSH] Authentication successful. [04/24/20 13:47:12] [SSH] The remote user's environment is: BASH=/usr/bin/bash BASHOPTS=cmdhist:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath BASH_ALIASES=() BASH_ARGC=() BASH_ARGV=() BASH_CMDS=() BASH_EXECUTION_STRING=set BASH_LINENO=() BASH_SOURCE=() BASH_VERSINFO=([0]= "4" [1]= "2" [2]= "46" [3]= "2" [4]= "release" [5]= "x86_64-redhat-linux-gnu" ) BASH_VERSION= '4.2.46(2)-release' DIRSTACK=() EUID=987 GROUPS=() HOME=/ var /lib/jenkins HOSTNAME=dbis-slave-01.uscis.dhs.gov HOSTTYPE=x86_64 IFS=$ ' \t\n' LOGNAME=jenkins MACHTYPE=x86_64-redhat-linux-gnu MAIL=/ var /mail/jenkins OPTERR=1 OPTIND=1 OSTYPE=linux-gnu PATH=/usr/local/bin:/usr/bin PPID=27318 PS4= '+ ' PWD=/ var /lib/jenkins SELINUX_LEVEL_REQUESTED= SELINUX_ROLE_REQUESTED= SELINUX_USE_CURRENT_RANGE= SHELL=/bin/bash SHELLOPTS=braceexpand:hashall:interactive-comments SHLVL=1 SSH_CLIENT= '10.193.177.209 43519 22' SSH_CONNECTION= '10.193.177.209 43519 10.193.177.232 22' TERM=dumb UID=987 USER=jenkins XDG_RUNTIME_DIR=/run/user/987 XDG_SESSION_ID=9633 _=bash [04/24/20 13:47:12] [SSH] Checking java version of / var /lib/jenkins/jdk/bin/java Couldn't figure out the Java version of / var /lib/jenkins/jdk/bin/java bash: / var /lib/jenkins/jdk/bin/java: No such file or directory[04/24/20 13:47:12] [SSH] Checking java version of java [04/24/20 13:47:12] [SSH] java -version returned 1.8.0_252. [04/24/20 13:47:12] [SSH] Starting sftp client. [04/24/20 13:47:12] [SSH] Copying latest remoting.jar... [04/24/20 13:47:12] [SSH] Copied 1,522,914 bytes. Expanded the channel window size to 4MB [04/24/20 13:47:12] [SSH] Starting agent process: cd "/ var /lib/jenkins" && java -jar remoting.jar -workDir / var /lib/jenkins -jar-cache / var /lib/jenkins/remoting/jarCache Apr 24, 2020 12:47:13 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using / var /lib/jenkins/remoting as a remoting work directory Apr 24, 2020 12:47:13 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to / var /lib/jenkins/remoting <===[JENKINS REMOTING CAPACITY]===>channel started Remoting version: 4.2 This is a Unix agent Apr 24, 2020 12:47:15 PM hudson.remoting.UserRequest perform WARNING: LinkageError while performing UserRequest:jenkins.slaves.StandardOutputSwapper$ChannelSwapper@16267446 java.lang.UnsatisfiedLinkError: /tmp/resource-2287829487335364078/com/sun/jna/linux-x86-64/libjnidispatch.so: /tmp/resource-2287829487335364078/com/sun/jna/linux-x86-64/libjnidispatch.so: failed to map segment from shared object: Operation not permitted at java.lang. ClassLoader $NativeLibrary.load(Native Method) at java.lang. ClassLoader .loadLibrary0( ClassLoader .java:1934) at java.lang. ClassLoader .loadLibrary( ClassLoader .java:1817) at java.lang. Runtime .load0( Runtime .java:809) at java.lang. System .load( System .java:1088) at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1018) at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988) at com.sun.jna.Native.<clinit>(Native.java:195) at hudson.util.jna.GNUCLibrary.<clinit>(GNUCLibrary.java:115) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.swap(StandardOutputSwapper.java:60) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:45) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:39) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748)Agent successfully connected and online So the key with the "---- BEGIN RSA PRIVATE KEY ----" seems to work, but it does give a java error: java.lang.UnsatisfiedLinkError: /tmp/resource-2287829487335364078/com/sun/jna/linux-x86-64/libjnidispatch.so: /tmp/resource-2287829487335364078/com/sun/jna/linux-x86-64/libjnidispatch.so: failed to map segment from shared object: Operation not permitted Can you tell me what command was used to create that key if different than previously specified? I'll need to somehow figure out how to generate this type of keypair on my system.     

          Matt Sicker added a comment -

          Commands I used to generate keys:

          ssh-keygen -f pemkey -m PEM -t rsa -b 4096
          ssh-keygen -f ssh2key -m RFC4716 -t rsa -b 4096
          

          As for the linkage error you got, that looks like a seccomp error I think? Never seen that before.

          Matt Sicker added a comment - Commands I used to generate keys: ssh-keygen -f pemkey -m PEM -t rsa -b 4096 ssh-keygen -f ssh2key -m RFC4716 -t rsa -b 4096 As for the linkage error you got, that looks like a seccomp error I think? Never seen that before.

          Matt Sicker added a comment -

          I've updated the original ticket to better reflect our understanding of the underlying issue.

          Matt Sicker added a comment - I've updated the original ticket to better reflect our understanding of the underlying issue.

          For me is not a critical issue, there are several formats supported, those formats can be generated with any modern version of OpenSSL or OpenSSH. The issue of the user is thigh attached to the restrictions and versions they use in their environment, so the overall impact on users is minimum and has a solution in most of the cases.
          I've documented the issue on the troubleshooting guide and in the configuration documents.

          Ivan Fernandez Calvo added a comment - For me is not a critical issue, there are several formats supported, those formats can be generated with any modern version of OpenSSL or OpenSSH. The issue of the user is thigh attached to the restrictions and versions they use in their environment, so the overall impact on users is minimum and has a solution in most of the cases. I've documented the issue on the troubleshooting guide and in the configuration documents.

          Matt Sicker added a comment - - edited

          Makes sense. I've updated it to minor. I've also started talking with some other Jenkins developers to try to find out whether or not we can migrate to Apache MINA SSH as our SSH client library as it's still actively developed. That may or may not be relevant in the future as default OpenSSH ciphers change over time. For example, there's no AES/GCM support, and I don't remember if ChaCha20-Poly1305 is already supported or not. Then there will be updated key exchanges and all the other supporting primitives. Not sure how much of that we want to continue maintaining if we can use an OSS dependency with a community around it.

          Matt Sicker added a comment - - edited Makes sense. I've updated it to minor. I've also started talking with some other Jenkins developers to try to find out whether or not we can migrate to Apache MINA SSH as our SSH client library as it's still actively developed. That may or may not be relevant in the future as default OpenSSH ciphers change over time. For example, there's no AES/GCM support, and I don't remember if ChaCha20-Poly1305 is already supported or not. Then there will be updated key exchanges and all the other supporting primitives. Not sure how much of that we want to continue maintaining if we can use an OSS dependency with a community around it.

          Matt Sicker added a comment -

          Adding this to the Mina SSH epic as it supports more key types, so this should get swept up as part of the migration.

          Matt Sicker added a comment - Adding this to the Mina SSH epic as it supports more key types, so this should get swept up as part of the migration.

          As workaround is possible to convert PKCS#8 format to PKCS#1, the key is stored encrypted in the Jenkins Controller so input format is not really relevant.

          openssl rsa -in private_pkcs8.pem -out private_pkcs1.pem -traditional

          Ivan Fernandez Calvo added a comment - As workaround is possible to convert PKCS#8 format to PKCS#1, the key is stored encrypted in the Jenkins Controller so input format is not really relevant. openssl rsa -in private_pkcs8.pem -out private_pkcs1.pem -traditional

            ifernandezcalvo Ivan Fernandez Calvo
            clsmith4 Charles Smith
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: