Details
-
Improvement
-
Status: Open (View Workflow)
-
Minor
-
Resolution: Unresolved
Description
The Jenkins SSH agent plugin uses trilead-ssh which only supports the legacy PEM and new SSH2 private key file formats. In certain hardened environments, SSH keys are mandated to be encoded in PKCS#8 (presumably because they're more universal and allow for password protection and encrypting the key) to the point where OpenSSH and ssh-keygen are patched to only support PKCS#8 private keys.
Attached to this ticket are various encodings of private keys as we troubleshooted the initial issue described below. Note that private keys that have a header line with BEGIN PRIVATE KEY are indicative of the file being encoded in PKCS#8. If the file header contains BEGIN RSA PRIVATE KEY (or EC or whatever other algorithm), that is the legacy PEM format. If the file header contains BEGIN OPENSSH PRIVATE KEY, then that is the new SSH2 format. The latter two encoding formats are already supported in trilead-ssh.
Original Details
- My Jenkins exists on an Amazon EC2 instance.
- I'm trying to add Red Hat 7.7 slave via ssh.
- I can ssh directly using terminal from my master to slave and vice versa using that key file
jenkins@master:/var/lib/jenkins $ ssh -i .ssh/id_rsa -q 10.193.177.232 jenkins@slave:/var/lib/jenkins $ ssh -i .ssh/id_rsa -q 10.193.177.209
I’m also able to make a successful SSH connection when I perform a SSH connection test from the Manage Jenkins -> Configure System section of the Master (FYI - see attached screenshot).
But when I try to configure my slave via launch agent method, it gives me the following error:
caused by: java.io.IOException: PEM problem: it is of unknown type at com.trilead.ssh2.crypto.PEMDecoder.decpdeKeyPair(PEMDecoder.java:500)
- How I've configured the slave (one of the method)?
- I've saved slave machine credentials as "ssh key username with private key"
- Launch method: Launch agents via SSH
- Hostname: slave machine private ip address.
- Host Key Verification Strategy: Manually trusted key Verification Strategy
Note:
- I cannot regenerate ssh keys as I'll not be able to access my machines again and the infrastructure is very strict.
- All SSH key are 4096 bits in length and RSA.
- I cannot use credentials as username/password as either there is no password.
- I've tried everything whatever mentioned in other jenkins tickets for same issue or on internet, but my issue is not getting resolved.
Please see the following attachments for log details and screenshots.
This is a total blocker. Please advise. Thank you.
Attachments
Activity
I've updated the original ticket to better reflect our understanding of the underlying issue.
For me is not a critical issue, there are several formats supported, those formats can be generated with any modern version of OpenSSL or OpenSSH. The issue of the user is thigh attached to the restrictions and versions they use in their environment, so the overall impact on users is minimum and has a solution in most of the cases.
I've documented the issue on the troubleshooting guide and in the configuration documents.
Makes sense. I've updated it to minor. I've also started talking with some other Jenkins developers to try to find out whether or not we can migrate to Apache MINA SSH as our SSH client library as it's still actively developed. That may or may not be relevant in the future as default OpenSSH ciphers change over time. For example, there's no AES/GCM support, and I don't remember if ChaCha20-Poly1305 is already supported or not. Then there will be updated key exchanges and all the other supporting primitives. Not sure how much of that we want to continue maintaining if we can use an OSS dependency with a community around it.
Adding this to the Mina SSH epic as it supports more key types, so this should get swept up as part of the migration.
Commands I used to generate keys:
As for the linkage error you got, that looks like a seccomp error I think? Never seen that before.