-
Improvement
-
Resolution: Unresolved
-
Minor
-
HideJenkins: Jenkins ver. 2.204.4
Plugins: SSH plugin 2.6.1 * SSH Build Agents 1.31.0 * SSH Credentials Plugin 1.18.1 *SSH Agent Plugin 1.19
OS: Master(RHEL7.7) Slave(RHEL7.7)
Java: Version 1.8.0_242 (Master & Slave)
Private key(SSH): RSA 4096 bit key (PKCS#8 encoded)ShowJenkins: Jenkins ver. 2.204.4 Plugins: SSH plugin 2.6.1 * SSH Build Agents 1.31.0 * SSH Credentials Plugin 1.18.1 *SSH Agent Plugin 1.19 OS: Master(RHEL7.7) Slave(RHEL7.7) Java: Version 1.8.0_242 (Master & Slave) Private key(SSH): RSA 4096 bit key (PKCS#8 encoded)
-
Powered by SuggestiMate
The Jenkins SSH agent plugin uses trilead-ssh which only supports the legacy PEM and new SSH2 private key file formats. In certain hardened environments, SSH keys are mandated to be encoded in PKCS#8 (presumably because they're more universal and allow for password protection and encrypting the key) to the point where OpenSSH and ssh-keygen are patched to only support PKCS#8 private keys.
Attached to this ticket are various encodings of private keys as we troubleshooted the initial issue described below. Note that private keys that have a header line with BEGIN PRIVATE KEY are indicative of the file being encoded in PKCS#8. If the file header contains BEGIN RSA PRIVATE KEY (or EC or whatever other algorithm), that is the legacy PEM format. If the file header contains BEGIN OPENSSH PRIVATE KEY, then that is the new SSH2 format. The latter two encoding formats are already supported in trilead-ssh.
Original Details
- My Jenkins exists on an Amazon EC2 instance.
- I'm trying to add Red Hat 7.7 slave via ssh.
- I can ssh directly using terminal from my master to slave and vice versa using that key file
jenkins@master:/var/lib/jenkins $ ssh -i .ssh/id_rsa -q 10.193.177.232 jenkins@slave:/var/lib/jenkins $ ssh -i .ssh/id_rsa -q 10.193.177.209
I’m also able to make a successful SSH connection when I perform a SSH connection test from the Manage Jenkins -> Configure System section of the Master (FYI - see attached screenshot).
But when I try to configure my slave via launch agent method, it gives me the following error:
caused by: java.io.IOException: PEM problem: it is of unknown type at com.trilead.ssh2.crypto.PEMDecoder.decpdeKeyPair(PEMDecoder.java:500)
- How I've configured the slave (one of the method)?
- I've saved slave machine credentials as "ssh key username with private key"
- Launch method: Launch agents via SSH
- Hostname: slave machine private ip address.
- Host Key Verification Strategy: Manually trusted key Verification Strategy
Note:
- I cannot regenerate ssh keys as I'll not be able to access my machines again and the infrastructure is very strict.
- All SSH key are 4096 bits in length and RSA.
- I cannot use credentials as username/password as either there is no password.
- I've tried everything whatever mentioned in other jenkins tickets for same issue or on internet, but my issue is not getting resolved.
Please see the following attachments for log details and screenshots.
This is a total blocker. Please advise. Thank you.
- 61356.tar.gz
- 6 kB
- 61356-key.txt
- 3 kB
- 61356-key.pub.txt
- 0.7 kB
- 61356-key.txt
- 3 kB
- ver1.17.4.PNG
- 24 kB
- new privkey.txt
- 2 kB
- ssh-cred.PNG
- 4 kB
- Slave screenshot.PNG
- 54 kB
- slave-log.log
- 2 kB
[JENKINS-61356] Trilead SSH does not support PKCS#8 encoded private keys
Starting in Java 8u151, unlimited strength cryptography is supported without additional configuration. Make sure you have the latest updates there.
If you can supply a test to reproduce the issue, that will help us address the issue sooner. It is still in our backlog.
Which other test would you suggest me trying? I've already tried multiple different SSH keys, downgrading the ssh-credential plugin, and the maven test of the ssh-credential plugin without success. Is there anything else that we can try at this juncture as this is a major blocker for us?
$ java -version openjdk version "1.8.0_242" OpenJDK Runtime Environment (build 1.8.0_242-b08) OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)
Oh I'm sorry, I confused this ticket with someone else. We already tested this together in the earlier comments. It looks like you should have an appropriate JVM for handling "large" RSA keys, so that likely isn't the issue.
I'll check if any of my co-workers know anything about this issue. It's possible that this issue is fairly new and hasn't been noticed by anyone else yet.
I just noticed that you may need to run Security.setProperty("crypto.policy", "unlimited"); before startup somewhere. Alternatively, the policy file is still available on Oracle's site: https://www.oracle.com/java/technologies/javase-jce8-downloads.html
I'll also note that you can set that crypto.policy property in the java.security file which is in JAVA_HOME/jre/lib/security/
jvz Something we should do for our Docker images if we don't already…?
Seems like openjdk doesn't disable strong crypto; it's oraclejdk that did. Some info here: https://support.cloudbees.com/hc/en-us/articles/115003785531-SSL-handshake-failures-due-to-incompatible-Cipher-Suite
Looking at the code, though, it seems like trilead-ssh is using its own PEM parsing code rather than the standard Java API. I'll note the method in question: https://github.com/jenkinsci/trilead-ssh2/blob/master/src/com/trilead/ssh2/crypto/PEMDecoder.java#L478
Could you try executing the following in a script console: com.trilead.ssh2.signature.KeyAlgorithmManager.getSupportedAlgorithms().dump()
Adding trilead as an affected component since the stacktrace relates to it. The linked file hasn't been updated in three years, so perhaps you're using a newer key algorithm?
Thank you for all of the updated information.
Do I need to rerun a similar Maven test on com.trilead.ssh2.signature.KeyAlgorithmManager.getSupportedAlgorithms().dump() like I did previously, or is there another way to execute this?
You can run that from the Jenkins script console: https://jenkins.io/doc/book/managing/script-console/
GM Matt - Below is the result after running the com.trilead.ssh2.signature.KeyAlgorithmManager.getSupportedAlgorithms().dump() from the script console:
Result Result: <java.util.Collections$UnmodifiableCollection@42327252 c=[com.trilead.ssh2.signature.ED25519KeyAlgorithm@4eff3ac, com.trilead.ssh2.signature.ECDSAKeyAlgorithm$ECDSASha2Nistp521@551eb181, com.trilead.ssh2.signature.ECDSAKeyAlgorithm$ECDSASha2Nistp384@4ca06dc6, com.trilead.ssh2.signature.ECDSAKeyAlgorithm$ECDSASha2Nistp256@796bd6b5, com.trilead.ssh2.signature.RSAKeyAlgorithm@26238a7, com.trilead.ssh2.signature.DSAKeyAlgorithm@225013d3]>
Is this the expected result? I cannot tell from the output?
That looks correct to me. I tried to reproduce this issue, but I'm not getting any PEM errors. Are you including the
-----BEGIN RSA PRIVATE KEY-----
and
-----END RSA PRIVATE KEY-----
lines in the key when you save it as an SSH credential? That's about the only thing I can think of that would cause the log message you're seeing to be output.
Edit: my reproduction attempt was with a master running on macOS and using an outbound SSH agent to connect to an Ubuntu 18.04 server. I tried using your provided key as well, though that just gives me a public key refused connection error as expected.
Oh here's a hypothesis: try adding "RSA" to your BEGIN/END PRIVATE KEY lines and see if you get the same error. If that works, then I believe we've isolated the issue to trilead's PEM decoder being overly strict.
I have just tried adding the "RSA" to the privkey with:
-----BEGIN RSA PRIVATE KEY-----
and
-----END RSA PRIVATE KEY-----
and still receiving the PEM error:
ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:jenkins-test-ssh-key/method:publickey) ERROR: Failed to authenticate as jenkins with credential=jenkins-test-ssh-key java.io.IOException: Publickey authentication failed. at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:358) at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:472) at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:109) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:436) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:473) at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:863) at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:435) at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:422) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: PEM problem: it is of unknown type at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:500) at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:292) ... 11 more [04/15/20 14:07:48] [SSH] Authentication failed. Authentication failed. [04/15/20 14:07:48] Launch failed - cleaning up connection
I noticed that in one of your earlier replies you were using a "noformat" option:
{ { noformat} -----BEGIN RSA PRIVATE KEY----- {noformat } } and { { noformat} -----END RSA PRIVATE KEY----- {noformat } }
Do I need to add the 'noformat" option to the privkey?
"noformat" is the formatting code for
this
in this issue tracker. Matt formatted his message after it looked bad with just basic text.
I did the noformat thing because otherwise jira was interpreting some of the dashes as strikeout text.
Oh ok thanks for the confirmation. I'm not using the private key that I provided earlier. If you want to test with the public key then I can send it to you?
I just tried a new separate 2048bit and 4096bit key with the RSA suggestion and still receive the PEM error message. I'm n ot sure if they're may be some unseen characters being added when I copy the new priv keys into the credential field.
I think we've isolated this to being a problem in trilead, or at least in our fork of it. The last active developer there is ifernandezcalvo. Perhaps this is something supported upstream that's not merged into our fork?
Reading the issue I do not know which version of trilead-ssh2 or trilead-api plugins are installed. I am gonna try with an RSA key generated with the following command
ssh-keygen -t rsa –b 4096
that it is the only command I saw in the comments but IIRC this exact command is the one I use so I would not replicate the issue. I will provide the test environment in a while after my tests.
It seems like it might be related to the message digests allowed. SHA-1 has been deprecated in openssh, so maybe it's possible that trilead doesn't support SHA-256 or something?
SHA-256 is supported https://github.com/jenkinsci/trilead-ssh2/blob/3322e54a48f500487ddb6862be42e289e7d87e08/src/com/trilead/ssh2/transport/KexManager.java#L435-L439
This is my test environment https://github.com/kuisathaverat/jenkins-issues/tree/master/JENKINS-61356 and the key is generated with this ssh-keygen version
ssh -V OpenSSH_8.1p1, LibreSSL 2.7.3
The key generated with this command
ssh-keygen -t rsa –b 4096 -N ""
I use JCasC to configure a credential with the key
https://github.com/kuisathaverat/jenkins-issues/blob/master/JENKINS-61356/jenkins/jenkins_home/jenkins.yaml#L11-L59
and I've added the public key to the authorized key in the agent,
https://github.com/kuisathaverat/jenkins-issues/blob/master/JENKINS-61356/ssh-agent/ssh/authorized_keys
I cannot replicate the issue on the latest Jenkins core (2.30) and the latest plugins
Thinking on common pitfalls I make a change on my configuration to create the credential with the PUBLIC key (totally wrong thing) and guess what the same error
SSHLauncher{host='ssh-agent', port=22, credentialsId='12187fc5-6f2d-4343-ad41-7a4a8c683b85', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.NonVerifyingKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true} [04/16/20 19:32:34] [SSH] Opening SSH connection to ssh-agent:22. [04/16/20 19:32:34] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:12187fc5-6f2d-4343-ad41-7a4a8c683b85/method:publickey) ERROR: Failed to authenticate as jenkins with credential=12187fc5-6f2d-4343-ad41-7a4a8c683b85 java.io.IOException: Publickey authentication failed. at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:358) at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:472) at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:109) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:436) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:473) at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:863) at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:435) at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:422) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: PEM problem: it is of unknown type at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:500) at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:292) ... 11 more [04/16/20 19:32:34] [SSH] Authentication failed. Authentication failed. [04/16/20 19:32:34] Launch failed - cleaning up connection [04/16/20 19:32:34] [SSH] Connection closed.
clsmith4 Are you sure you are creating the Jenkins credential with the PRIVATE key(something like this one https://github.com/kuisathaverat/jenkins-issues/blob/master/JENKINS-61356/ssh-agent/ssh/rsa-key) and the authorized_keys with the PUBLIC key (https://github.com/kuisathaverat/jenkins-issues/blob/master/JENKINS-61356/ssh-agent/ssh/rsa-key.pub)
Other common pitfalls with the same result are to copy and paste the key in an editor that does not support Unix line breaks and save the key with windows linebreaks, add spaces on any part of the key, in general, anything that modifies the original ARMOR format of the key break the key.
To have a key pair that is confirmed that fails could help to be sure that the problem is not in the plugin or Jenkins
Thank you for all your updates on this issue.
Yes, I'm 1000% sure that it's the PRIVATE key(not PUBLIC) that I'm copying into the SSH credential field that asks for the PRIVATE key. I've created another keypair(61356-key & 61356-key.pub) and able to ssh without issue from the Master to the Build(slave) system using this test keypair. However, the same keypair isn't working when used by the shh credential plugin to launch the build agent. I've attached the keypair for your testing/visibility.
If this is an issue with copying/pasting the keypair with added Unix line breaks, then what is a workaround for this? I'm a Unix guy so I copy the PRIVATE key from my Putty terminal directly into the SSH credential field. Is there another way of doing this to test this theory?
Other than that, we're currently running RHEL7.8:
$ ssh -V OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
We also run security McAfee software that may potentially be an issue when creating the keypair:
# systemctl status mfeespd.service; systemctl status mfetpd.service ● mfeespd.service - McAfee Endpoint Security Platform for Linux Loaded: loaded (/usr/lib/systemd/system/mfeespd.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2020-04-15 14:13:48 CDT; 5 days ago Docs: man:mfeespd(8) Main PID: 4574 (mfeespd) Tasks: 9 Memory: 2.9M CGroup: /system.slice/mfeespd.service └─4574 /opt/McAfee/ens/esp/bin/mfeespdApr 15 14:13:46 ansible-baseline-rhel7-test.uscis.dhs.gov systemd[1]: Starting M... Apr 15 14:13:46 ansible-baseline-rhel7-test.uscis.dhs.gov prepare-mfeesp.sh[4561]: ... Apr 15 14:13:48 ansible-baseline-rhel7-test.uscis.dhs.gov systemd[1]: Started Mc... Hint: Some lines were ellipsized, use -l to show in full. ● mfetpd.service - McAfee Endpoint Security for Linux Threat Prevention Loaded: loaded (/usr/lib/systemd/system/mfetpd.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2020-04-15 14:13:49 CDT; 5 days ago Docs: man:mfetpd(8) Main PID: 4732 (mfetpd) Tasks: 49 Memory: 1.4G CGroup: /system.slice/mfetpd.service ├─ 4732 /opt/McAfee/ens/tp/bin/mfetpd ├─ 4777 /opt/McAfee/ens/tp/bin/mfetpd ├─20076 /opt/McAfee/ens/tp/bin/mfetpd └─20084 /opt/McAfee/ens/tp/bin/mfetpdApr 15 14:13:49 ansible-baseline-rhel7-test.uscis.dhs.gov systemd[1]: Starting M... Apr 15 14:13:49 ansible-baseline-rhel7-test.uscis.dhs.gov systemd[1]: Started Mc... Apr 15 14:13:50 ansible-baseline-rhel7-test.uscis.dhs.gov mfetpd[4732]: error: F... Hint: Some lines were ellipsized, use -l to show in full.
However, the keypair works when I use it to ssh from the shell.
Hello All - Just checking for any more responses or updates on this? Where to go from here? How to test if erroneous characters are being added to the priv key section of that plugin? Any more information would be appreciated. Thanks.
Can you try generating a key using openssl genrsa -out private.pem 4096 and see if that works? I wonder if there's been some sort of key format change in ssh-keygen. I really wish this were using OpenSSH, but it's not.
In fact, I wonder if that's the issue. It says the PEM is of an unknown type, but an SSH private key is usually encoded via RFC4716 rather than PEM (which ssh-keygen(1) says is the "legacy" format). There's also PCKS8.
Version: OpenSSH_8.1p1, LibreSSL 2.7.3
Based on version differences, it seems like your fips-patched version of OpenSSH is likely what's causing the issue. There might be a missing SSH key format being supported by trilead?
Thanks for the updates. How do you want me to extract the public key from the pem file?
Using ssh-keygen:
ssh-keygen-y -f key.pem > key.pub
Or using openssl:
openssl rsa -in key.pem -pubout -out pubkey.pem
Note that when using openssl, the public key doesn't have the standard ssh-rsa heading that normal ssh public keys use.
The fips-patched version of OpenSSH makes sense. I've attached a test keypair earlier that you can try and test with to see if the issue can be replicated. I'm not using that keypair.
You can convert an existing key with ssh-keygen -p -f private_key_file -m pem
And I meant to try and see if the RSA keys generated from openssl are decoded properly by Jenkins compared to the ones being generated via ssh-keygen.
Alright, I've done some digging, and here's what I've found. The key you attached (61356) is encoded in PKCS#8 which can be seen by the "BEGIN PRIVATE KEY" header. What is usually used in SSH is PKCS#1 (i.e., no special encoding apparently) which has the header "BEGIN RSA PRIVATE KEY". The former format encodes the algorithm in the base64 data while the latter is more of a naked format with the algorithm in the header. I've also seen "BEGIN OPENSSH PRIVATE KEY" which appears to be the newer format which is used by ed25519 and other ciphers.
One workaround you could try (in production, too!) would be to convert your private key to the old PEM format. You can use openssl rsa -in private.key -out private.pem -outform PEM to convert the key encoding format which should be supported by trilead. If that doesn't work, then we may have an interesting problem.
Looking at trilead, I see that RSAKeyAlgorithm only supports SHA1, and it doesn't even support aes-128/256-gcm, so it seems somewhat dated in algorithm support as it is. I'm kind of wondering why this library is the default rather than Apache SSHD (which was used for some other SSH functionality in Jenkins).
I see there's support for the old pem file format and the new openssh (RFC 4716) format, but no explicit support for PKCS#8. Could you try out the other formats for the same key to see if it works? If it doesn't, then I think we have some missing crypto primitives here.
Thanks for the new information.
Just to confirm with you:
1.) Do you want me to use the ssh-keygen or openssl command to generate the public key? I've already created the key.pem file with the openssl genrsa command?
2.) Do you want me to run openssl rsa -in private.key -out private.pem -outform PEM to change the private key format to pem and then extract the public key from the converted pem file?
Sorry I want to make sure that I do this right.
Take your original private key (or any of the private keys you've created through ssh-keygen) and use the openssl command to convert its format to PEM. Then try that PEM key in Jenkins.
I'm unable to get this to work using the 61356-key private key.
To replicate:
I ran the following to convert the key to PEM format(61356-key.pem):
jenkins@dbis-admin:/var/lib/jenkins/.ssh/test $ openssl rsa -in 61356-key -out 61356-key.pem -outform PEM writing RSA key jenkins@dbis-admin:/var/lib/jenkins/.ssh/test $ ls -l total 8 -rw-------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key -rw-------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key.pem
Then, copied(selected) the new key in vi format and pasted it into the SSH credential key box. I launch the build agent and get the same PEM error. Both the original private key and pem file are the same exact (i.e. no changes to the BEGIN PRIVATE KEY headers.)
Well I'll be damned. Seems like this fips version just doesn't let you use the old format. Could you try exporting it in the RFC4716 format? ssh-keygen -f 61356-key -p -m RFC4716 and save without a password. Check the updated key file and make sure it has a "BEGIN OPENSSH PRIVATE KEY" header instead of "BEGIN PRIVATE KEY". If it still has the BEGIN PRIVATE KEY, then it really seems locked down to a single key format, period.
Yes it's confirmed. The 61356-key still has the "BEGIN PRIVATE KEY" header after attempting to change the format to RFC4716:
$ ls -l
total 8
rw------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key
rw------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key.pem
$ grep "BEGIN\ PRIVATE\ KEY" 61356-key
----BEGIN PRIVATE KEY----
$ date
Thu Apr 23 22:58:12 EDT 2020
$ ssh-keygen -f 61356-key -p -m RFC4716
Enter new passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved with the new passphrase.
$ ls -l
total 8
rw------. 1 jenkins jenkins 3272 Apr 23 22:58 61356-key
rw------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key.pem
$ grep "BEGIN\ PRIVATE\ KEY" 61356-key
----BEGIN PRIVATE KEY----
${code}
I make a test with the original keys, then I've converted the original key with openssl and the issue is resolved.
Here is the environment I made for the tests https://github.com/kuisathaverat/jenkins-issues/tree/master/JENKINS-61356
This are the steps to start the environment
git clone https://github.com/kuisathaverat/jenkins-issues.git
cd jenkins-issues/JENKINS-61356
make clean start
The original key is in https://github.com/kuisathaverat/jenkins-issues/blob/master/JENKINS-61356/ssh-agent/ssh/rsa-key.org
I've converted it with the following commands, then I've updated my JCasC configuration and everything works as expected.
openssl version LibreSSL 2.8.3 openssl rsa -in rsa-key.org -out rsa-key
The PKCS#8 (BEGIN PRIVATE KEY) is not supported. There are other formats supported see https://github.com/jenkinsci/trilead-ssh2/tree/master/test/com/trilead/ssh2/signature
That's the problem the key never gets converted when I run those openssl commands. I noticed that you haven't tested yet with the attached 61356-key keypairs that I have attached to this ticket. Try your test on that keypair as I cannot get it to convert.
# openssl version OpenSSL 1.0.2k-fips 26 Jan 2017
I've attached the 61356 key converted into the ssh2 format. It's the newer version of the same file.
I tested the files uploaded 3 days ago https://issues.jenkins-ci.org/secure/attachment/51008/61356-key.pub.txt and https://issues.jenkins-ci.org/secure/attachment/51007/61356-key.txt
I just updated the test repo with three agents using the three different formats we have in the issue:
- "----
BEGIN PRIVATE KEY----" failure - "----
BEGIN OPENSSH PRIVATE KEY----" success - "----
BEGIN RSA PRIVATE KEY----" success
Alright, then I think we've isolated the issue. trilead doesn't support PKCS#8 private keys.
When I use your converted 61356 key to launch the build agent, I now receive a different error:
SSHLauncher{host='prod-slave-01.tt.com', port=22, credentialsId='jenkins-test-ssh-key', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=60, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true} [04/24/20 11:13:58] [SSH] Opening SSH connection to dbis-slave-01.uscis.dhs.gov:22. [04/24/20 11:13:58] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed. ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:jenkins-test-ssh-key/method:publickey) [04/24/20 11:13:59] [SSH] Authentication failed. Authentication failed. [04/24/20 11:13:59] Launch failed - cleaning up connection [04/24/20 11:13:59] [SSH] Connection closed.
However, I'm able to successfully SSH into the build agent from the jenkins master shell using the same converted key. Which openssl command did you use to confirm that the privkeys are PKCS#8?
I converted the key using the command I pasted earlier:
ssh-keygen -f 61356-key.txt -p -m RFC4716
When prompted, I used an empty password. I tried using some other flags besides -p to convert the private key, but the other flags seem to only output the public key.
And to confirm that the key is PKCS#8, you can tell because it has the header "BEGIN PRIVATE KEY" with no specific type of key mentioned in the header. That data is encoded in the base64 contents.
Ok that command doesn't work on my system when trying to convert the key. Is this because of the fips enabled versions of SSH and SSL that's being used?
As a last ditch effort for me, can you please create a new test ssh keypair that isn't PKCS#8? (i.e. - not converted key)?
I would like to test one last time with the 2 keypairs:
- "----
BEGIN OPENSSH PRIVATE KEY----" success - "----
BEGIN RSA PRIVATE KEY----" success
I can't convert the key on my system to either of those formats.
I've uploaded a tar.gz file with two newly generated RSA 4096 key pairs in both the legacy PEM and RFC4716 format from ssh-keygen.
I created a new ssh credential and used the private key with "----BEGIN RSA PRIVATE KEY----" and now the build agent successfully starts online:
SSHLauncher{host='dbis-slave-01.uscis.dhs.gov', port=22, credentialsId='jenkins-rsa-test-ssh-key', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=60, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true} [04/24/20 13:47:11] [SSH] Opening SSH connection to dbis-slave-01.uscis.dhs.gov:22. [04/24/20 13:47:11] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed. [04/24/20 13:47:12] [SSH] Authentication successful. [04/24/20 13:47:12] [SSH] The remote user's environment is: BASH=/usr/bin/bash BASHOPTS=cmdhist:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath BASH_ALIASES=() BASH_ARGC=() BASH_ARGV=() BASH_CMDS=() BASH_EXECUTION_STRING=set BASH_LINENO=() BASH_SOURCE=() BASH_VERSINFO=([0]="4" [1]="2" [2]="46" [3]="2" [4]="release" [5]="x86_64-redhat-linux-gnu") BASH_VERSION='4.2.46(2)-release' DIRSTACK=() EUID=987 GROUPS=() HOME=/var/lib/jenkins HOSTNAME=dbis-slave-01.uscis.dhs.gov HOSTTYPE=x86_64 IFS=$' \t\n' LOGNAME=jenkins MACHTYPE=x86_64-redhat-linux-gnu MAIL=/var/mail/jenkins OPTERR=1 OPTIND=1 OSTYPE=linux-gnu PATH=/usr/local/bin:/usr/bin PPID=27318 PS4='+ ' PWD=/var/lib/jenkins SELINUX_LEVEL_REQUESTED= SELINUX_ROLE_REQUESTED= SELINUX_USE_CURRENT_RANGE= SHELL=/bin/bash SHELLOPTS=braceexpand:hashall:interactive-comments SHLVL=1 SSH_CLIENT='10.193.177.209 43519 22' SSH_CONNECTION='10.193.177.209 43519 10.193.177.232 22' TERM=dumb UID=987 USER=jenkins XDG_RUNTIME_DIR=/run/user/987 XDG_SESSION_ID=9633 _=bash [04/24/20 13:47:12] [SSH] Checking java version of /var/lib/jenkins/jdk/bin/java Couldn't figure out the Java version of /var/lib/jenkins/jdk/bin/java bash: /var/lib/jenkins/jdk/bin/java: No such file or directory[04/24/20 13:47:12] [SSH] Checking java version of java [04/24/20 13:47:12] [SSH] java -version returned 1.8.0_252. [04/24/20 13:47:12] [SSH] Starting sftp client. [04/24/20 13:47:12] [SSH] Copying latest remoting.jar... [04/24/20 13:47:12] [SSH] Copied 1,522,914 bytes. Expanded the channel window size to 4MB [04/24/20 13:47:12] [SSH] Starting agent process: cd "/var/lib/jenkins" && java -jar remoting.jar -workDir /var/lib/jenkins -jar-cache /var/lib/jenkins/remoting/jarCache Apr 24, 2020 12:47:13 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /var/lib/jenkins/remoting as a remoting work directory Apr 24, 2020 12:47:13 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /var/lib/jenkins/remoting <===[JENKINS REMOTING CAPACITY]===>channel started Remoting version: 4.2 This is a Unix agent Apr 24, 2020 12:47:15 PM hudson.remoting.UserRequest perform WARNING: LinkageError while performing UserRequest:jenkins.slaves.StandardOutputSwapper$ChannelSwapper@16267446 java.lang.UnsatisfiedLinkError: /tmp/resource-2287829487335364078/com/sun/jna/linux-x86-64/libjnidispatch.so: /tmp/resource-2287829487335364078/com/sun/jna/linux-x86-64/libjnidispatch.so: failed to map segment from shared object: Operation not permitted at java.lang.ClassLoader$NativeLibrary.load(Native Method) at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1934) at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1817) at java.lang.Runtime.load0(Runtime.java:809) at java.lang.System.load(System.java:1088) at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1018) at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988) at com.sun.jna.Native.<clinit>(Native.java:195) at hudson.util.jna.GNUCLibrary.<clinit>(GNUCLibrary.java:115) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.swap(StandardOutputSwapper.java:60) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:45) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:39) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)Agent successfully connected and online
So the key with the "----BEGIN RSA PRIVATE KEY----" seems to work, but it does give a java error:
java.lang.UnsatisfiedLinkError: /tmp/resource-2287829487335364078/com/sun/jna/linux-x86-64/libjnidispatch.so: /tmp/resource-2287829487335364078/com/sun/jna/linux-x86-64/libjnidispatch.so: failed to map segment from shared object: Operation not permitted
Can you tell me what command was used to create that key if different than previously specified? I'll need to somehow figure out how to generate this type of keypair on my system.
Commands I used to generate keys:
ssh-keygen -f pemkey -m PEM -t rsa -b 4096 ssh-keygen -f ssh2key -m RFC4716 -t rsa -b 4096
As for the linkage error you got, that looks like a seccomp error I think? Never seen that before.
I've updated the original ticket to better reflect our understanding of the underlying issue.
For me is not a critical issue, there are several formats supported, those formats can be generated with any modern version of OpenSSL or OpenSSH. The issue of the user is thigh attached to the restrictions and versions they use in their environment, so the overall impact on users is minimum and has a solution in most of the cases.
I've documented the issue on the troubleshooting guide and in the configuration documents.
Makes sense. I've updated it to minor. I've also started talking with some other Jenkins developers to try to find out whether or not we can migrate to Apache MINA SSH as our SSH client library as it's still actively developed. That may or may not be relevant in the future as default OpenSSH ciphers change over time. For example, there's no AES/GCM support, and I don't remember if ChaCha20-Poly1305 is already supported or not. Then there will be updated key exchanges and all the other supporting primitives. Not sure how much of that we want to continue maintaining if we can use an OSS dependency with a community around it.
Adding this to the Mina SSH epic as it supports more key types, so this should get swept up as part of the migration.
As workaround is possible to convert PKCS#8 format to PKCS#1, the key is stored encrypted in the Jenkins Controller so input format is not really relevant.
openssl rsa -in private_pkcs8.pem -out private_pkcs1.pem -traditional
jvz
GM Matt - I'm running out of ideas on why this issue continues to occur with slave node connections via SSH. I found in the following links below that the Java Cryptography Extension (JCE) may need to be installed in the JVM that Jenkins uses; the RSA key size is limited without installing this extension. Do you by chance have directions on how to install this extension and/or how to confirm that it is currently installed?
Links:
https://stackoverflow.com/questions/30285122/jenkins-master-fails-to-connect-to-the-slave-over-ssh?rq=1
https://issues.jenkins-ci.org/browse/JENKINS-26495