-
Improvement
-
Resolution: Unresolved
-
Minor
The Jenkins SSH agent plugin uses trilead-ssh which only supports the legacy PEM and new SSH2 private key file formats. In certain hardened environments, SSH keys are mandated to be encoded in PKCS#8 (presumably because they're more universal and allow for password protection and encrypting the key) to the point where OpenSSH and ssh-keygen are patched to only support PKCS#8 private keys.
Attached to this ticket are various encodings of private keys as we troubleshooted the initial issue described below. Note that private keys that have a header line with BEGIN PRIVATE KEY are indicative of the file being encoded in PKCS#8. If the file header contains BEGIN RSA PRIVATE KEY (or EC or whatever other algorithm), that is the legacy PEM format. If the file header contains BEGIN OPENSSH PRIVATE KEY, then that is the new SSH2 format. The latter two encoding formats are already supported in trilead-ssh.
Original Details
- My Jenkins exists on an Amazon EC2 instance.
- I'm trying to add Red Hat 7.7 slave via ssh.
- I can ssh directly using terminal from my master to slave and vice versa using that key file
jenkins@master:/var/lib/jenkins $ ssh -i .ssh/id_rsa -q 10.193.177.232 jenkins@slave:/var/lib/jenkins $ ssh -i .ssh/id_rsa -q 10.193.177.209
I’m also able to make a successful SSH connection when I perform a SSH connection test from the Manage Jenkins -> Configure System section of the Master (FYI - see attached screenshot).
But when I try to configure my slave via launch agent method, it gives me the following error:
caused by: java.io.IOException: PEM problem: it is of unknown type at com.trilead.ssh2.crypto.PEMDecoder.decpdeKeyPair(PEMDecoder.java:500)
- How I've configured the slave (one of the method)?
- I've saved slave machine credentials as "ssh key username with private key"
- Launch method: Launch agents via SSH
- Hostname: slave machine private ip address.
- Host Key Verification Strategy: Manually trusted key Verification Strategy
Note:
- I cannot regenerate ssh keys as I'll not be able to access my machines again and the infrastructure is very strict.
- All SSH key are 4096 bits in length and RSA.
- I cannot use credentials as username/password as either there is no password.
- I've tried everything whatever mentioned in other jenkins tickets for same issue or on internet, but my issue is not getting resolved.
Please see the following attachments for log details and screenshots.
This is a total blocker. Please advise. Thank you.
[JENKINS-61356] Trilead SSH does not support PKCS#8 encoded private keys
Did this issue appear in a particular version of the plugin? Or do you have a test that demonstrates the issue?
Charles Smith
added a comment - Hi Matt, This is the first build agent that has been added to our Jenkins instance. We are running the latest plugins and Jenkins version. As far as a test, all you have to do is create a new build agent and attempt to launch it and it produces the PEM error. Please let me know if you need anymore information apart from what has already been provided.
Charles Smith
added a comment - jvz
Hi Matt, This is the first build agent that has been added to our Jenkins instance. We are running the latest plugins and Jenkins version. As far as a test, all you have to do is create a new build agent and attempt to launch it and it produces the PEM error. Please let me know if you need anymore information apart from what has already been provided. Take a look at this test: https://github.com/jenkinsci/ssh-credentials-plugin/blob/master/src/test/java/com/cloudbees/jenkins/plugins/sshcredentials/impl/TrileadSSHPublicKeyAuthenticatorTest.java
If you run that test locally but instead put the contents of your private key in the getPrivateKey() method, does the test still pass?
Charles Smith
added a comment - My apologies Matt, but I'm having a lot of trouble trying to compile that java program before running it 'TrileadSSHPublicKeyAuthenticatorTest.java'. Every time I attempt to compile the program it errors out with the following:
javac TrileadSSHPublicKeyAuthenticatorTest.java symbol: method assertNotNull(Object) location: class TrileadSSHPublicKeyAuthenticatorTest TrileadSSHPublicKeyAuthenticatorTest.java:343: error: cannot find symbol assertNotNull(factory); ^ symbol: method assertNotNull(Object) location: class TrileadSSHPublicKeyAuthenticatorTest Note: TrileadSSHPublicKeyAuthenticatorTest.java uses unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. 68 errors 1 warning
Can you let me know if I'm doing this correctly? Or is there an easier way that I can test this program and the getPrivateKey() method? I don't have access to an IDE so will need to run this test from the server manually with javac and java commands.
Charles Smith
added a comment - jvz
My apologies Matt, but I'm having a lot of trouble trying to compile that java program before running it ' TrileadSSHPublicKeyAuthenticatorTest.java '. Every time I attempt to compile the program it errors out with the following:
javac TrileadSSHPublicKeyAuthenticatorTest.java
symbol: method assertNotNull( Object )
location: class TrileadSSHPublicKeyAuthenticatorTest
TrileadSSHPublicKeyAuthenticatorTest.java:343: error: cannot find symbol
assertNotNull(factory);
^
symbol: method assertNotNull( Object )
location: class TrileadSSHPublicKeyAuthenticatorTest
Note: TrileadSSHPublicKeyAuthenticatorTest.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
68 errors
1 warning
Can you let me know if I'm doing this correctly? Or is there an easier way that I can test this program and the getPrivateKey() method? I don't have access to an IDE so will need to run this test from the server manually with javac and java commands. You'll need to git clone the repo, edit that file to change the key to your key, then you can run mvn test -Dtest=TrileadSSHPublicKeyAuthenticatorTest to run just that test.
Charles Smith
added a comment - Thank you for providing the information and direction. I'll work on cloning the repo and try running the maven test.
Charles Smith
added a comment - jvz
Thank you for providing the information and direction. I'll work on cloning the repo and try running the maven test. The test is failing after adding my private key to the TrileadSSHPublicKeyAuthenticatorTest.java file. I'm not sure if anything else needs tp be added in order for the build to become successful or if there's a limitation on how big the private key must be? The private key that is found in this file is only 20 line long whereas my private key is 50 lines long. When I run the test with the private key that's already found in the file the test build is successful. I've also attached the test txt log of the test of my private key to this ticket. Below is the output from the maven test:
$ mvn -e test -Dtest=TrileadSSHPublicKeyAuthenticatorTest [INFO] Error stacktraces are turned on. [INFO] Scanning for projects... [WARNING] The POM for org.jenkins-ci.tools:maven-hpi-plugin:jar:3.11 is missing, no dependency information available [WARNING] Failed to build parent project for org.jenkins-ci.plugins:ssh-credentials:hpi:1.18.2-SNAPSHOT [INFO] [INFO] ---------------< org.jenkins-ci.plugins:ssh-credentials >--------------- [INFO] Building SSH Credentials Plugin 1.18.2-SNAPSHOT [INFO] --------------------------------[ hpi ]--------------------------------- [INFO] [INFO] --- maven-hpi-plugin:3.11:validate (default-validate) @ ssh-credentials --- [INFO] [INFO] --- maven-enforcer-plugin:3.0.0-M3:display-info (display-info) @ ssh-credentials --- [INFO] Maven Version: 3.6.3 [INFO] JDK Version: 1.8.0_242 normalized as: 1.8.0-242 [INFO] OS Info: Arch: amd64 Family: unix Name: linux Version: 3.10.0-1062.12.1.el7.x86_64 [INFO] [INFO] --- maven-enforcer-plugin:3.0.0-M3:enforce (display-info) @ ssh-credentials --- [INFO] Adding ignore: module-info [INFO] Ignoring requireUpperBoundDeps in com.google.guava:guava [INFO] [INFO] --- maven-localizer-plugin:1.26:generate (default) @ ssh-credentials --- [INFO] [INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ ssh-credentials --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 10 resources [INFO] [INFO] --- flatten-maven-plugin:1.1.0:flatten (flatten) @ ssh-credentials --- [INFO] Generating flattened POM of project org.jenkins-ci.plugins:ssh-credentials:hpi:1.18.2-SNAPSHOT... [INFO] [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ ssh-credentials --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- access-modifier-checker:1.16:enforce (default-enforce) @ ssh-credentials --- [INFO] [INFO] --- maven-hpi-plugin:3.11:insert-test (default-insert-test) @ ssh-credentials --- [INFO] [INFO] --- gmaven-plugin:1.5-jenkins-3:generateTestStubs (test-in-groovy) @ ssh-credentials --- [INFO] No sources found for Java stub generation [INFO] [INFO] --- maven-antrun-plugin:1.8:run (createTempDir) @ ssh-credentials --- [INFO] Executing tasksmain: [INFO] Executed tasks [INFO] [INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ ssh-credentials --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 4 resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ ssh-credentials --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-hpi-plugin:3.11:test-hpl (default-test-hpl) @ ssh-credentials --- [INFO] Generating /home/clsmith4/test/ssh-credentials-plugin-master/target/test-classes/the.hpl [INFO] [INFO] --- maven-hpi-plugin:3.11:resolve-test-dependencies (default-resolve-test-dependencies) @ ssh-credentials --- [INFO] [INFO] --- gmaven-plugin:1.5-jenkins-3:testCompile (test-in-groovy) @ ssh-credentials --- [INFO] No sources found to compile [INFO] [INFO] --- maven-surefire-plugin:3.0.0-M4:test (default-test) @ ssh-credentials --- [INFO] Surefire report directory: /home/clsmith4/test/ssh-credentials-plugin-master/target/surefire-reports [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest === Starting testAuthenticate(com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest) 0.098 [id=12] INFO o.jvnet.hudson.test.WarExploder#explode: Picking up existing exploded jenkins.war at /home/clsmith4/test/ssh-credentials-plugin-master/target/jenkins-for-test 0.307 [id=12] INFO o.jvnet.hudson.test.JenkinsRule#createWebServer: Running on http://localhost:15058/jenkins/ 1.851 [id=20] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization 2.099 [id=20] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /home/clsmith4/test/ssh-credentials-plugin-master/target/tmp/jenkins3113231465058299879/command-launcher.jpi 2.156 [id=20] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /home/clsmith4/test/ssh-credentials-plugin-master/target/tmp/jenkins3113231465058299879/jdk-tool.jpi 2.203 [id=20] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /home/clsmith4/test/ssh-credentials-plugin-master/target/tmp/jenkins3113231465058299879/script-security.jpi 2.493 [id=18] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /home/clsmith4/test/ssh-credentials-plugin-master/target/tmp/jenkins3113231465058299879/bouncycastle-api.jpi 2.679 [id=19] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins 2.737 [id=18] INFO j.b.a.SecurityProviderInitializer#addSecurityProvider: Initializing Bouncy Castle security provider. 2.880 [id=18] INFO j.b.a.SecurityProviderInitializer#addSecurityProvider: Bouncy Castle security provider initialized. 4.126 [id=21] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins 4.131 [id=19] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins 4.133 [id=20] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions 4.778 [id=20] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs 4.976 [id=21] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization ERROR: Server rejected the 1 private key(s) for foobar (credentialId:/method:publickey) ERROR: Failed to authenticate as foobar with credential= java.io.IOException: Publickey authentication failed. at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:302) at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:451) at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:109) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:436) at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.testAuthenticate(TrileadSSHPublicKeyAuthenticatorTest.java:200) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:596) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: PEM problem: it is of unknown type at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:500) at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:236) ... 19 more 7.035 [id=12] INFO jenkins.model.Jenkins#cleanUp: Stopping Jenkins 7.315 [id=12] INFO jenkins.model.Jenkins#cleanUp: Jenkins stopped === Starting testFactory(com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest) 0.017 [id=57] INFO o.jvnet.hudson.test.JenkinsRule#createWebServer: Running on http://localhost:20104/jenkins/ 0.048 [id=64] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization 0.053 [id=63] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins 0.159 [id=64] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins 0.162 [id=63] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins 0.165 [id=66] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions 1.462 [id=65] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs 1.478 [id=64] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization ERROR: Server rejected the 1 private key(s) for foobar (credentialId:/method:publickey) ERROR: Failed to authenticate as foobar with credential= java.io.IOException: Publickey authentication failed. at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:302) at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:451) at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:109) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:436) at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.testFactory(TrileadSSHPublicKeyAuthenticatorTest.java:239) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:596) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: PEM problem: it is of unknown type at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:500) at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:236) ... 19 more 2.518 [id=57] INFO jenkins.model.Jenkins#cleanUp: Stopping Jenkins 2.600 [id=57] INFO jenkins.model.Jenkins#cleanUp: Jenkins stopped === Starting testAltUsername(com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest) 0.022 [id=91] INFO o.jvnet.hudson.test.JenkinsRule#createWebServer: Running on http://localhost:19917/jenkins/ 0.048 [id=98] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization 0.056 [id=99] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins 0.135 [id=98] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins 0.137 [id=99] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins 0.138 [id=97] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions 0.626 [id=100] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs 0.645 [id=98] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization ERROR: Server rejected the 1 private key(s) for foobar (credentialId:/method:publickey) ERROR: Failed to authenticate as foobar with credential= java.io.IOException: Publickey authentication failed. at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:302) at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:451) at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:109) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:436) at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.testAltUsername(TrileadSSHPublicKeyAuthenticatorTest.java:277) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:596) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: PEM problem: it is of unknown type at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:500) at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:236) ... 19 more ERROR: Server rejected the 1 private key(s) for bill (credentialId:/method:publickey) ERROR: Failed to authenticate as bill with credential= java.io.IOException: Publickey authentication failed. at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:302) at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:451) at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:109) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:436) at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.testAltUsername(TrileadSSHPublicKeyAuthenticatorTest.java:290) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:596) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: PEM problem: it is of unknown type at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:500) at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:236) ... 19 more 2.252 [id=91] INFO jenkins.model.Jenkins#cleanUp: Stopping Jenkins 2.321 [id=91] INFO jenkins.model.Jenkins#cleanUp: Jenkins stopped [ERROR] Tests run: 3, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 13.285 s <<< FAILURE! - in com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest [ERROR] com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.testAuthenticate Time elapsed: 7.489 s <<< FAILURE! java.lang.AssertionError:Expected: is <true> but: was <false> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) at org.junit.Assert.assertThat(Assert.java:956) at org.junit.Assert.assertThat(Assert.java:923) at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.testAuthenticate(TrileadSSHPublicKeyAuthenticatorTest.java:200) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:596) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748)[ERROR] com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.testFactory Time elapsed: 2.752 s <<< FAILURE! java.lang.AssertionError:Expected: is <true> but: was <false> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) at org.junit.Assert.assertThat(Assert.java:956) at org.junit.Assert.assertThat(Assert.java:923) at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.testFactory(TrileadSSHPublicKeyAuthenticatorTest.java:239) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:596) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748)[ERROR] com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.testAltUsername Time elapsed: 2.446 s <<< FAILURE! java.lang.AssertionError:Expected: is <true> but: was <false> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) at org.junit.Assert.assertThat(Assert.java:956) at org.junit.Assert.assertThat(Assert.java:923) at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.testAltUsername(TrileadSSHPublicKeyAuthenticatorTest.java:290) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:596) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748)[INFO] [INFO] Results: [INFO] [ERROR] Failures: [ERROR] TrileadSSHPublicKeyAuthenticatorTest.testAltUsername:290 Expected: is <true> but: was <false> [ERROR] TrileadSSHPublicKeyAuthenticatorTest.testAuthenticate:200 Expected: is <true> but: was <false> [ERROR] TrileadSSHPublicKeyAuthenticatorTest.testFactory:239 Expected: is <true> but: was <false> [INFO] [ERROR] Tests run: 3, Failures: 3, Errors: 0, Skipped: 0 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 24.609 s [INFO] Finished at: 2020-03-12T14:16:20-05:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M4:test (default-test) on project ssh-credentials: There are test failures. [ERROR] [ERROR] Please refer to /home/clsmith4/test/ssh-credentials-plugin-master/target/surefire-reports for the individual test results. [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream. [ERROR] -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M4:test (default-test) on project ssh-credentials: There are test failures.Please refer to /home/clsmith4/test/ssh-credentials-plugin-master/target/surefire-reports for the individual test results. Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream. at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289) at org.apache.maven.cli.MavenCli.main (MavenCli.java:193) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) Caused by: org.apache.maven.plugin.MojoFailureException: There are test failures.Please refer to /home/clsmith4/test/ssh-credentials-plugin-master/target/surefire-reports for the individual test results. Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream. at org.apache.maven.plugin.surefire.SurefireHelper.throwException (SurefireHelper.java:271) at org.apache.maven.plugin.surefire.SurefireHelper.reportExecution (SurefireHelper.java:159) at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary (SurefirePlugin.java:435) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked (AbstractSurefireMojo.java:1079) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute (AbstractSurefireMojo.java:889) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289) at org.apache.maven.cli.MavenCli.main (MavenCli.java:193) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) [ERROR] [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Charles Smith
added a comment - - edited jvz
The test is failing after adding my private key to the TrileadSSHPublicKeyAuthenticatorTest.java file. I'm not sure if anything else needs tp be added in order for the build to become successful or if there's a limitation on how big the private key must be? The private key that is found in this file is only 20 line long whereas my private key is 50 lines long. When I run the test with the private key that's already found in the file the test build is successful. I've also attached the test txt log of the test of my private key to this ticket. Below is the output from the maven test:
$ mvn -e test -Dtest=TrileadSSHPublicKeyAuthenticatorTest [INFO] Error stacktraces are turned on.
[INFO] Scanning for projects...
[WARNING] The POM for org.jenkins-ci.tools:maven-hpi-plugin:jar:3.11 is missing, no dependency information available
[WARNING] Failed to build parent project for org.jenkins-ci.plugins:ssh-credentials:hpi:1.18.2-SNAPSHOT
[INFO]
[INFO] ---------------< org.jenkins-ci.plugins:ssh-credentials >---------------
[INFO] Building SSH Credentials Plugin 1.18.2-SNAPSHOT
[INFO] --------------------------------[ hpi ]---------------------------------
[INFO]
[INFO] --- maven-hpi-plugin:3.11:validate ( default -validate) @ ssh-credentials ---
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M3:display-info (display-info) @ ssh-credentials ---
[INFO] Maven Version: 3.6.3
[INFO] JDK Version: 1.8.0_242 normalized as: 1.8.0-242
[INFO] OS Info: Arch: amd64 Family: unix Name: linux Version: 3.10.0-1062.12.1.el7.x86_64
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M3:enforce (display-info) @ ssh-credentials ---
[INFO] Adding ignore: module-info
[INFO] Ignoring requireUpperBoundDeps in com.google.guava:guava
[INFO]
[INFO] --- maven-localizer-plugin:1.26:generate ( default ) @ ssh-credentials ---
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:resources ( default -resources) @ ssh-credentials ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 10 resources
[INFO]
[INFO] --- flatten-maven-plugin:1.1.0:flatten (flatten) @ ssh-credentials ---
[INFO] Generating flattened POM of project org.jenkins-ci.plugins:ssh-credentials:hpi:1.18.2-SNAPSHOT...
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile ( default -compile) @ ssh-credentials ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- access-modifier-checker:1.16:enforce ( default -enforce) @ ssh-credentials ---
[INFO]
[INFO] --- maven-hpi-plugin:3.11:insert-test ( default -insert-test) @ ssh-credentials ---
[INFO]
[INFO] --- gmaven-plugin:1.5-jenkins-3:generateTestStubs (test-in-groovy) @ ssh-credentials ---
[INFO] No sources found for Java stub generation
[INFO]
[INFO] --- maven-antrun-plugin:1.8:run (createTempDir) @ ssh-credentials ---
[INFO] Executing tasksmain:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:testResources ( default -testResources) @ ssh-credentials ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 4 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile ( default -testCompile) @ ssh-credentials ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-hpi-plugin:3.11:test-hpl ( default -test-hpl) @ ssh-credentials ---
[INFO] Generating /home/clsmith4/test/ssh-credentials-plugin-master/target/test-classes/the.hpl
[INFO]
[INFO] --- maven-hpi-plugin:3.11:resolve-test-dependencies ( default -resolve-test-dependencies) @ ssh-credentials ---
[INFO]
[INFO] --- gmaven-plugin:1.5-jenkins-3:testCompile (test-in-groovy) @ ssh-credentials ---
[INFO] No sources found to compile
[INFO]
[INFO] --- maven-surefire-plugin:3.0.0-M4:test ( default -test) @ ssh-credentials ---
[INFO] Surefire report directory: /home/clsmith4/test/ssh-credentials-plugin-master/target/surefire-reports
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest
=== Starting testAuthenticate(com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest)
0.098 [id=12] INFO o.jvnet.hudson.test.WarExploder#explode: Picking up existing exploded jenkins.war at /home/clsmith4/test/ssh-credentials-plugin-master/target/jenkins- for -test
0.307 [id=12] INFO o.jvnet.hudson.test.JenkinsRule#createWebServer: Running on http: //localhost:15058/jenkins/
1.851 [id=20] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization
2.099 [id=20] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /home/clsmith4/test/ssh-credentials-plugin-master/target/tmp/jenkins3113231465058299879/command-launcher.jpi
2.156 [id=20] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /home/clsmith4/test/ssh-credentials-plugin-master/target/tmp/jenkins3113231465058299879/jdk-tool.jpi
2.203 [id=20] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /home/clsmith4/test/ssh-credentials-plugin-master/target/tmp/jenkins3113231465058299879/script-security.jpi
2.493 [id=18] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /home/clsmith4/test/ssh-credentials-plugin-master/target/tmp/jenkins3113231465058299879/bouncycastle-api.jpi
2.679 [id=19] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins
2.737 [id=18] INFO j.b.a.SecurityProviderInitializer#addSecurityProvider: Initializing Bouncy Castle security provider.
2.880 [id=18] INFO j.b.a.SecurityProviderInitializer#addSecurityProvider: Bouncy Castle security provider initialized.
4.126 [id=21] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
4.131 [id=19] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins
4.133 [id=20] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
4.778 [id=20] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
4.976 [id=21] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization
ERROR: Server rejected the 1 private key(s) for foobar (credentialId:/method:publickey)
ERROR: Failed to authenticate as foobar with credential=
java.io.IOException: Publickey authentication failed.
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:302)
at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:451)
at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:109)
at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:436)
at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.testAuthenticate(TrileadSSHPublicKeyAuthenticatorTest.java:200)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:596)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang. Thread .run( Thread .java:748)
Caused by: java.io.IOException: PEM problem: it is of unknown type
at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:500)
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:236)
... 19 more
7.035 [id=12] INFO jenkins.model.Jenkins#cleanUp: Stopping Jenkins
7.315 [id=12] INFO jenkins.model.Jenkins#cleanUp: Jenkins stopped
=== Starting testFactory(com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest)
0.017 [id=57] INFO o.jvnet.hudson.test.JenkinsRule#createWebServer: Running on http: //localhost:20104/jenkins/
0.048 [id=64] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization
0.053 [id=63] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins
0.159 [id=64] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
0.162 [id=63] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins
0.165 [id=66] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
1.462 [id=65] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
1.478 [id=64] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization
ERROR: Server rejected the 1 private key(s) for foobar (credentialId:/method:publickey)
ERROR: Failed to authenticate as foobar with credential=
java.io.IOException: Publickey authentication failed.
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:302)
at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:451)
at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:109)
at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:436)
at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.testFactory(TrileadSSHPublicKeyAuthenticatorTest.java:239)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:596)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang. Thread .run( Thread .java:748)
Caused by: java.io.IOException: PEM problem: it is of unknown type
at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:500)
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:236)
... 19 more
2.518 [id=57] INFO jenkins.model.Jenkins#cleanUp: Stopping Jenkins
2.600 [id=57] INFO jenkins.model.Jenkins#cleanUp: Jenkins stopped
=== Starting testAltUsername(com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest)
0.022 [id=91] INFO o.jvnet.hudson.test.JenkinsRule#createWebServer: Running on http: //localhost:19917/jenkins/
0.048 [id=98] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization
0.056 [id=99] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins
0.135 [id=98] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
0.137 [id=99] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins
0.138 [id=97] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
0.626 [id=100] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
0.645 [id=98] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization
ERROR: Server rejected the 1 private key(s) for foobar (credentialId:/method:publickey)
ERROR: Failed to authenticate as foobar with credential=
java.io.IOException: Publickey authentication failed.
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:302)
at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:451)
at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:109)
at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:436)
at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.testAltUsername(TrileadSSHPublicKeyAuthenticatorTest.java:277)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:596)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang. Thread .run( Thread .java:748)
Caused by: java.io.IOException: PEM problem: it is of unknown type
at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:500)
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:236)
... 19 more
ERROR: Server rejected the 1 private key(s) for bill (credentialId:/method:publickey)
ERROR: Failed to authenticate as bill with credential=
java.io.IOException: Publickey authentication failed.
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:302)
at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:451)
at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:109)
at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:436)
at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.testAltUsername(TrileadSSHPublicKeyAuthenticatorTest.java:290)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:596)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang. Thread .run( Thread .java:748)
Caused by: java.io.IOException: PEM problem: it is of unknown type
at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:500)
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:236)
... 19 more
2.252 [id=91] INFO jenkins.model.Jenkins#cleanUp: Stopping Jenkins
2.321 [id=91] INFO jenkins.model.Jenkins#cleanUp: Jenkins stopped
[ERROR] Tests run: 3, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 13.285 s <<< FAILURE! - in com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest
[ERROR] com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.testAuthenticate Time elapsed: 7.489 s <<< FAILURE!
java.lang.AssertionError:Expected: is < true >
but: was < false >
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.testAuthenticate(TrileadSSHPublicKeyAuthenticatorTest.java:200)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:596)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang. Thread .run( Thread .java:748)[ERROR] com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.testFactory Time elapsed: 2.752 s <<< FAILURE!
java.lang.AssertionError:Expected: is < true >
but: was < false >
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.testFactory(TrileadSSHPublicKeyAuthenticatorTest.java:239)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:596)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang. Thread .run( Thread .java:748)[ERROR] com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.testAltUsername Time elapsed: 2.446 s <<< FAILURE!
java.lang.AssertionError:Expected: is < true >
but: was < false >
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticatorTest.testAltUsername(TrileadSSHPublicKeyAuthenticatorTest.java:290)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:596)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang. Thread .run( Thread .java:748)[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TrileadSSHPublicKeyAuthenticatorTest.testAltUsername:290
Expected: is < true >
but: was < false >
[ERROR] TrileadSSHPublicKeyAuthenticatorTest.testAuthenticate:200
Expected: is < true >
but: was < false >
[ERROR] TrileadSSHPublicKeyAuthenticatorTest.testFactory:239
Expected: is < true >
but: was < false >
[INFO]
[ERROR] Tests run: 3, Failures: 3, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 24.609 s
[INFO] Finished at: 2020-03-12T14:16:20-05:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M4:test ( default -test) on project ssh-credentials: There are test failures.
[ERROR]
[ERROR] Please refer to /home/clsmith4/test/ssh-credentials-plugin-master/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files ( if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M4:test ( default -test) on project ssh-credentials: There are test failures.Please refer to /home/clsmith4/test/ssh-credentials-plugin-master/target/surefire-reports for the individual test results.
Please refer to dump files ( if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.maven.plugin.MojoFailureException: There are test failures.Please refer to /home/clsmith4/test/ssh-credentials-plugin-master/target/surefire-reports for the individual test results.
Please refer to dump files ( if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
at org.apache.maven.plugin.surefire.SurefireHelper.throwException (SurefireHelper.java:271)
at org.apache.maven.plugin.surefire.SurefireHelper.reportExecution (SurefireHelper.java:159)
at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary (SurefirePlugin.java:435)
at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked (AbstractSurefireMojo.java:1079)
at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute (AbstractSurefireMojo.java:889)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
[ERROR]
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http: //cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
So it looks like whatever key file format you're using isn't supported potentially? What encoding format are you using? I'm not very familiar with what's supported upstream at the moment besides what I've been using.
My private key being used is a 4096 bit RSA key. The command that was used to create the keypair was:
ssh-keygen -t rsa –b 4096
I'm pretty sure RSA should be a supported format, do you know if there's a limit on the bit size? I'm able to SSH between systems from the shell with this same keypair.
$ file .ssh/id_rsa .ssh/id_rsa: ASCII text
Charles Smith
added a comment - - edited jvz
My private key being used is a 4096 bit RSA key. The command that was used to create the keypair was:
ssh-keygen -t rsa –b 4096
I'm pretty sure RSA should be a supported format, do you know if there's a limit on the bit size? I'm able to SSH between systems from the shell with this same keypair.
$ file .ssh/id_rsa
.ssh/id_rsa: ASCII text
That's how I've used it, too. Could you try using an older version of the plugin to see if this is a regression? Previous releases are available here: https://updates.jenkins-ci.org/download/plugins/ssh-credentials/
Charles Smith
added a comment - Ok. Just to confirm, you would like me to try downgrading the ssh-credential plugin to a previous version from the Plugin Manager section of my Jenkins' master?
Charles Smith
added a comment - jvz
Ok. Just to confirm, you would like me to try downgrading the ssh-credential plugin to a previous version from the Plugin Manager section of my Jenkins' master? Charles Smith
added a comment - After downgrading the ssh-credentials plugin from version 1.18.1 to 1.18, I'm still getting the same PEM error. I also generated a new RSA keypair but this time of 2048 bits. When I try with the new keypair it still errors out. I was only able to downgrade from 1 previous version from the Plugin Manager page(FYI - It didn't give me an option to downgrade lower than version 1.18). If you can provide me with some directions on how to manually downgrade the plugin to an even lower version then I can try that test as well. Also, what private key type are you currently using when launching your slave?
Charles Smith
added a comment - jvz
After downgrading the ssh-credentials plugin from version 1.18.1 to 1.18 , I'm still getting the same PEM error. I also generated a new RSA keypair but this time of 2048 bits. When I try with the new keypair it still errors out. I was only able to downgrade from 1 previous version from the Plugin Manager page( FYI - It didn't give me an option to downgrade lower than version 1.18 ). If you can provide me with some directions on how to manually downgrade the plugin to an even lower version then I can try that test as well. Also, what private key type are you currently using when launching your slave? I've been using ssh-credentials for use with GitHub myself. As I'm not a Jenkins admin, I haven't had to configure an SSH agent in quite some time. Perhaps it would be useful to try and ping the maintainer(s) of the ssh-agent plugin?
From a crypto point of view, my guess would be that you're using some sort of insecure crypto setting due to using an old version of ssh with poor default settings (e.g., use of SHA-1 somewhere), though that's typically an issue with the SSH connection itself, not the key file. What version of openssh are you using?
I recall seeing issues in the past reported by users around trailing newlines or other formatting issues, though a lot of those have been patched.
Charles Smith
added a comment - We are using the following OpenSSH version:
$ ssh -V OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
I'm also not sure if there's a limitation with regard to using a Red Hat system instead of a Centos operating system? Is there anything OS wide I should look into? Also, should we try an earlier version of the SSH Credential plugin?
Charles Smith
added a comment - jvz
We are using the following OpenSSH version:
$ ssh -V
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
I'm also not sure if there's a limitation with regard to using a Red Hat system instead of a Centos operating system? Is there anything OS wide I should look into? Also, should we try an earlier version of the SSH Credential plugin? I'd suggest trying an earlier version of SSH Credentials, yes. Your version of ssh is a few releases behind what I have (OpenSSH_7.9p1, LibreSSL 2.7.3), though I can't think of any reason offhand that would cause a problem. A few years ago, many SSH keys had to be revoked due to a bug in Debian distribution of openssh, but that didn't affect RHEL/CentOS from what I recall.
Does this issue still occur with newly generated keys? If so, could you generate a new key and attach it to this ticket?
Yes I generated a new keypair of 2048 bits and still getting this error. I've attached the new key to this ticket.
Can you also send me some directions on how to manually downgrade the ssh credentials plugin? I'm only able to downgrade to version 1.18 from the Plugins Manager page and would like to know the correct steps on downgrading this plugin manually.
This also is the latest SSH version from RHEL7.7:
# rpm -qa|grep ssh
sshpass-1.06-1.el7.x86_64
libssh2-1.8.0-3.el7.x86_64
openssh-7.4p1-21.el7.x86_64
openssh-clients-7.4p1-21.el7.x86_64
openssh-server-7.4p1-21.el7.x86_64root@dbis-admin:/root
# yum update openssh
Loaded plugins: amazon-id, product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
No packages marked for update
Charles Smith
added a comment - - edited jvz
Yes I generated a new keypair of 2048 bits and still getting this error. I've attached the new key to this ticket.
Can you also send me some directions on how to manually downgrade the ssh credentials plugin? I'm only able to downgrade to version 1.18 from the Plugins Manager page and would like to know the correct steps on downgrading this plugin manually.
This also is the latest SSH version from RHEL7.7:
# rpm -qa|grep ssh
sshpass-1.06-1.el7.x86_64
libssh2-1.8.0-3.el7.x86_64
openssh-7.4p1-21.el7.x86_64
openssh-clients-7.4p1-21.el7.x86_64
openssh-server-7.4p1-21.el7.x86_64root@dbis-admin:/root
# yum update openssh
Loaded plugins: amazon-id, product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
No packages marked for update
Thanks for the additional info. I've added this to my team's backlog to investigate.
As for manually downgrading a plugin, I don't know how to do that from the update center. However, using the link I provided, you can download the old HPI file and use the plugin manager in Jenkins to upload a plugin to install manually.
Charles Smith
added a comment - Thanks for the update and escalating with your team. Can you possibly send me some directions on how to install the old HPI file from the plugin manager page?
Charles Smith
added a comment - jvz
Thanks for the update and escalating with your team. Can you possibly send me some directions on how to install the old HPI file from the plugin manager page? I've already provided all the directions I know about how to downgrade. You have to manually download the old version of the plugin from the link provided in my earlier comment, and then you can upload the file following these directions: https://jenkins.io/doc/book/managing/plugins/#from-the-web-ui-2
Hey Matt - So I can't downgrade the ssh-credential-plugin any lower than version 1.18. When I attempt to downgrade to version 1.17.4 or lower, the Build Agent gives an error and removes the Launch Agent button from launching via SSH(see new attachments).
It looks like SSH is completely removed from the Master, do I need to reset something under MANAGE JENKINS -> CONFIGURE SECURITY?
Charles Smith
added a comment - - edited jvz
Hey Matt - So I can't downgrade the ssh-credential-plugin any lower than version 1.18. When I attempt to downgrade to version 1.17.4 or lower, the Build Agent gives an error and removes the Launch Agent button from launching via SSH(see new attachments).
It looks like SSH is completely removed from the Master, do I need to reset something under MANAGE JENKINS -> CONFIGURE SECURITY? GM Matt - I'm running out of ideas on why this issue continues to occur with slave node connections via SSH. I found in the following links below that the Java Cryptography Extension (JCE) may need to be installed in the JVM that Jenkins uses; the RSA key size is limited without installing this extension. Do you by chance have directions on how to install this extension and/or how to confirm that it is currently installed?
Links:
Charles Smith
added a comment - - edited jvz
GM Matt - I'm running out of ideas on why this issue continues to occur with slave node connections via SSH. I found in the following links below that the Java Cryptography Extension (JCE) may need to be installed in the JVM that Jenkins uses; the RSA key size is limited without installing this extension. Do you by chance have directions on how to install this extension and/or how to confirm that it is currently installed?
Links:
https://stackoverflow.com/questions/30285122/jenkins-master-fails-to-connect-to-the-slave-over-ssh?rq=1
https://issues.jenkins-ci.org/browse/JENKINS-26495 Starting in Java 8u151, unlimited strength cryptography is supported without additional configuration. Make sure you have the latest updates there.
If you can supply a test to reproduce the issue, that will help us address the issue sooner. It is still in our backlog.
Charles Smith
added a comment - Which other test would you suggest me trying? I've already tried multiple different SSH keys, downgrading the ssh-credential plugin, and the maven test of the ssh-credential plugin without success. Is there anything else that we can try at this juncture as this is a major blocker for us?
$ java -version openjdk version "1.8.0_242" OpenJDK Runtime Environment (build 1.8.0_242-b08) OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)
Charles Smith
added a comment - jvz
Which other test would you suggest me trying? I've already tried multiple different SSH keys, downgrading the ssh-credential plugin, and the maven test of the ssh-credential plugin without success. Is there anything else that we can try at this juncture as this is a major blocker for us?
$ java -version
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-b08)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)
Oh I'm sorry, I confused this ticket with someone else. We already tested this together in the earlier comments. It looks like you should have an appropriate JVM for handling "large" RSA keys, so that likely isn't the issue.
I'll check if any of my co-workers know anything about this issue. It's possible that this issue is fairly new and hasn't been noticed by anyone else yet.
I just noticed that you may need to run Security.setProperty("crypto.policy", "unlimited"); before startup somewhere. Alternatively, the policy file is still available on Oracle's site: https://www.oracle.com/java/technologies/javase-jce8-downloads.html
I'll also note that you can set that crypto.policy property in the java.security file which is in JAVA_HOME/jre/lib/security/
jvz Something we should do for our Docker images if we don't already…?
Seems like openjdk doesn't disable strong crypto; it's oraclejdk that did. Some info here: https://support.cloudbees.com/hc/en-us/articles/115003785531-SSL-handshake-failures-due-to-incompatible-Cipher-Suite
Looking at the code, though, it seems like trilead-ssh is using its own PEM parsing code rather than the standard Java API. I'll note the method in question: https://github.com/jenkinsci/trilead-ssh2/blob/master/src/com/trilead/ssh2/crypto/PEMDecoder.java#L478
Could you try executing the following in a script console: com.trilead.ssh2.signature.KeyAlgorithmManager.getSupportedAlgorithms().dump()
Adding trilead as an affected component since the stacktrace relates to it. The linked file hasn't been updated in three years, so perhaps you're using a newer key algorithm?
Charles Smith
added a comment - Thank you for all of the updated information.
Do I need to rerun a similar Maven test on com.trilead.ssh2.signature.KeyAlgorithmManager.getSupportedAlgorithms().dump() like I did previously, or is there another way to execute this?
Charles Smith
added a comment - jvz
Thank you for all of the updated information.
Do I need to rerun a similar Maven test on com.trilead.ssh2.signature.KeyAlgorithmManager.getSupportedAlgorithms().dump() like I did previously, or is there another way to execute this? You can run that from the Jenkins script console: https://jenkins.io/doc/book/managing/script-console/
GM Matt - Below is the result after running the com.trilead.ssh2.signature.KeyAlgorithmManager.getSupportedAlgorithms().dump() from the script console:
Result Result: <java.util.Collections$UnmodifiableCollection@42327252 c=[com.trilead.ssh2.signature.ED25519KeyAlgorithm@4eff3ac, com.trilead.ssh2.signature.ECDSAKeyAlgorithm$ECDSASha2Nistp521@551eb181, com.trilead.ssh2.signature.ECDSAKeyAlgorithm$ECDSASha2Nistp384@4ca06dc6, com.trilead.ssh2.signature.ECDSAKeyAlgorithm$ECDSASha2Nistp256@796bd6b5, com.trilead.ssh2.signature.RSAKeyAlgorithm@26238a7, com.trilead.ssh2.signature.DSAKeyAlgorithm@225013d3]>
Is this the expected result? I cannot tell from the output?
Charles Smith
added a comment - - edited jvz
GM Matt - Below is the result after running the com.trilead.ssh2.signature.KeyAlgorithmManager.getSupportedAlgorithms().dump() from the script console:
Result
Result: <java.util.Collections$UnmodifiableCollection@42327252 c=[com.trilead.ssh2.signature.ED25519KeyAlgorithm@4eff3ac, com.trilead.ssh2.signature.ECDSAKeyAlgorithm$ECDSASha2Nistp521@551eb181, com.trilead.ssh2.signature.ECDSAKeyAlgorithm$ECDSASha2Nistp384@4ca06dc6, com.trilead.ssh2.signature.ECDSAKeyAlgorithm$ECDSASha2Nistp256@796bd6b5, com.trilead.ssh2.signature.RSAKeyAlgorithm@26238a7, com.trilead.ssh2.signature.DSAKeyAlgorithm@225013d3]>
Is this the expected result? I cannot tell from the output? That looks correct to me. I tried to reproduce this issue, but I'm not getting any PEM errors. Are you including the
-----BEGIN RSA PRIVATE KEY-----
and
-----END RSA PRIVATE KEY-----
lines in the key when you save it as an SSH credential? That's about the only thing I can think of that would cause the log message you're seeing to be output.
Edit: my reproduction attempt was with a master running on macOS and using an outbound SSH agent to connect to an Ubuntu 18.04 server. I tried using your provided key as well, though that just gives me a public key refused connection error as expected.
Oh here's a hypothesis: try adding "RSA" to your BEGIN/END PRIVATE KEY lines and see if you get the same error. If that works, then I believe we've isolated the issue to trilead's PEM decoder being overly strict.
Charles Smith
added a comment - I have just tried adding the "RSA" to the privkey with:
-----BEGIN RSA PRIVATE KEY-----
and
-----END RSA PRIVATE KEY-----
and still receiving the PEM error:
ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:jenkins-test-ssh-key/method:publickey) ERROR: Failed to authenticate as jenkins with credential=jenkins-test-ssh-key java.io.IOException: Publickey authentication failed. at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:358) at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:472) at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:109) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:436) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:473) at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:863) at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:435) at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:422) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: PEM problem: it is of unknown type at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:500) at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:292) ... 11 more [04/15/20 14:07:48] [SSH] Authentication failed. Authentication failed. [04/15/20 14:07:48] Launch failed - cleaning up connection
I noticed that in one of your earlier replies you were using a "noformat" option:
{ { noformat} -----BEGIN RSA PRIVATE KEY----- {noformat } } and { { noformat} -----END RSA PRIVATE KEY----- {noformat } }
Do I need to add the 'noformat" option to the privkey?
Charles Smith
added a comment - jvz
I have just tried adding the "RSA" to the privkey with:
-----BEGIN RSA PRIVATE KEY-----
and
-----END RSA PRIVATE KEY-----
and still receiving the PEM error:
ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:jenkins-test-ssh-key/method:publickey)
ERROR: Failed to authenticate as jenkins with credential=jenkins-test-ssh-key
java.io.IOException: Publickey authentication failed.
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:358)
at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:472)
at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:109)
at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:436)
at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:473)
at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:863)
at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:435)
at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:422)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang. Thread .run( Thread .java:748)
Caused by: java.io.IOException: PEM problem: it is of unknown type
at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:500)
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:292)
... 11 more
[04/15/20 14:07:48] [SSH] Authentication failed.
Authentication failed.
[04/15/20 14:07:48] Launch failed - cleaning up connection
I noticed that in one of your earlier replies you were using a " noformat " option:
{ { noformat} -----BEGIN RSA PRIVATE KEY----- {noformat } } and { { noformat} -----END RSA PRIVATE KEY----- {noformat } }
Do I need to add the ' noformat " option to the privkey? "noformat" is the formatting code for
this
in this issue tracker. Matt formatted his message after it looked bad with just basic text.
I did the noformat thing because otherwise jira was interpreting some of the dashes as strikeout text.
Oh ok thanks for the confirmation. I'm not using the private key that I provided earlier. If you want to test with the public key then I can send it to you?
I just tried a new separate 2048bit and 4096bit key with the RSA suggestion and still receive the PEM error message. I'm n ot sure if they're may be some unseen characters being added when I copy the new priv keys into the credential field.
Charles Smith
added a comment - - edited jvz danielbeck
Oh ok thanks for the confirmation. I'm not using the private key that I provided earlier. If you want to test with the public key then I can send it to you?
I just tried a new separate 2048bit and 4096bit key with the RSA suggestion and still receive the PEM error message. I'm n ot sure if they're may be some unseen characters being added when I copy the new priv keys into the credential field. I think we've isolated this to being a problem in trilead, or at least in our fork of it. The last active developer there is ifernandezcalvo. Perhaps this is something supported upstream that's not merged into our fork?
Reading the issue I do not know which version of trilead-ssh2 or trilead-api plugins are installed. I am gonna try with an RSA key generated with the following command
ssh-keygen -t rsa –b 4096
that it is the only command I saw in the comments but IIRC this exact command is the one I use so I would not replicate the issue. I will provide the test environment in a while after my tests.
It seems like it might be related to the message digests allowed. SHA-1 has been deprecated in openssh, so maybe it's possible that trilead doesn't support SHA-256 or something?
SHA-256 is supported https://github.com/jenkinsci/trilead-ssh2/blob/3322e54a48f500487ddb6862be42e289e7d87e08/src/com/trilead/ssh2/transport/KexManager.java#L435-L439
This is my test environment https://github.com/kuisathaverat/jenkins-issues/tree/master/JENKINS-61356 and the key is generated with this ssh-keygen version
ssh -V OpenSSH_8.1p1, LibreSSL 2.7.3
The key generated with this command
ssh-keygen -t rsa –b 4096 -N ""
I use JCasC to configure a credential with the key
https://github.com/kuisathaverat/jenkins-issues/blob/master/JENKINS-61356/jenkins/jenkins_home/jenkins.yaml#L11-L59
and I've added the public key to the authorized key in the agent,
https://github.com/kuisathaverat/jenkins-issues/blob/master/JENKINS-61356/ssh-agent/ssh/authorized_keys
I cannot replicate the issue on the latest Jenkins core (2.30) and the latest plugins
Thinking on common pitfalls I make a change on my configuration to create the credential with the PUBLIC key (totally wrong thing) and guess what the same error
SSHLauncher{host='ssh-agent', port=22, credentialsId='12187fc5-6f2d-4343-ad41-7a4a8c683b85', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.NonVerifyingKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
[04/16/20 19:32:34] [SSH] Opening SSH connection to ssh-agent:22.
[04/16/20 19:32:34] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:12187fc5-6f2d-4343-ad41-7a4a8c683b85/method:publickey)
ERROR: Failed to authenticate as jenkins with credential=12187fc5-6f2d-4343-ad41-7a4a8c683b85
java.io.IOException: Publickey authentication failed.
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:358)
at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:472)
at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:109)
at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:436)
at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:473)
at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:863)
at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:435)
at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:422)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: PEM problem: it is of unknown type
at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:500)
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:292)
... 11 more
[04/16/20 19:32:34] [SSH] Authentication failed.
Authentication failed.
[04/16/20 19:32:34] Launch failed - cleaning up connection
[04/16/20 19:32:34] [SSH] Connection closed.
clsmith4 Are you sure you are creating the Jenkins credential with the PRIVATE key(something like this one https://github.com/kuisathaverat/jenkins-issues/blob/master/JENKINS-61356/ssh-agent/ssh/rsa-key) and the authorized_keys with the PUBLIC key (https://github.com/kuisathaverat/jenkins-issues/blob/master/JENKINS-61356/ssh-agent/ssh/rsa-key.pub)
Other common pitfalls with the same result are to copy and paste the key in an editor that does not support Unix line breaks and save the key with windows linebreaks, add spaces on any part of the key, in general, anything that modifies the original ARMOR format of the key break the key.
To have a key pair that is confirmed that fails could help to be sure that the problem is not in the plugin or Jenkins
Thank you for all your updates on this issue.
Yes, I'm 1000% sure that it's the PRIVATE key(not PUBLIC) that I'm copying into the SSH credential field that asks for the PRIVATE key. I've created another keypair(61356-key & 61356-key.pub) and able to ssh without issue from the Master to the Build(slave) system using this test keypair. However, the same keypair isn't working when used by the shh credential plugin to launch the build agent. I've attached the keypair for your testing/visibility.
If this is an issue with copying/pasting the keypair with added Unix line breaks, then what is a workaround for this? I'm a Unix guy so I copy the PRIVATE key from my Putty terminal directly into the SSH credential field. Is there another way of doing this to test this theory?
Other than that, we're currently running RHEL7.8:
$ ssh -V OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
We also run security McAfee software that may potentially be an issue when creating the keypair:
# systemctl status mfeespd.service; systemctl status mfetpd.service ● mfeespd.service - McAfee Endpoint Security Platform for Linux Loaded: loaded (/usr/lib/systemd/system/mfeespd.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2020-04-15 14:13:48 CDT; 5 days ago Docs: man:mfeespd(8) Main PID: 4574 (mfeespd) Tasks: 9 Memory: 2.9M CGroup: /system.slice/mfeespd.service └─4574 /opt/McAfee/ens/esp/bin/mfeespdApr 15 14:13:46 ansible-baseline-rhel7-test.uscis.dhs.gov systemd[1]: Starting M... Apr 15 14:13:46 ansible-baseline-rhel7-test.uscis.dhs.gov prepare-mfeesp.sh[4561]: ... Apr 15 14:13:48 ansible-baseline-rhel7-test.uscis.dhs.gov systemd[1]: Started Mc... Hint: Some lines were ellipsized, use -l to show in full. ● mfetpd.service - McAfee Endpoint Security for Linux Threat Prevention Loaded: loaded (/usr/lib/systemd/system/mfetpd.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2020-04-15 14:13:49 CDT; 5 days ago Docs: man:mfetpd(8) Main PID: 4732 (mfetpd) Tasks: 49 Memory: 1.4G CGroup: /system.slice/mfetpd.service ├─ 4732 /opt/McAfee/ens/tp/bin/mfetpd ├─ 4777 /opt/McAfee/ens/tp/bin/mfetpd ├─20076 /opt/McAfee/ens/tp/bin/mfetpd └─20084 /opt/McAfee/ens/tp/bin/mfetpdApr 15 14:13:49 ansible-baseline-rhel7-test.uscis.dhs.gov systemd[1]: Starting M... Apr 15 14:13:49 ansible-baseline-rhel7-test.uscis.dhs.gov systemd[1]: Started Mc... Apr 15 14:13:50 ansible-baseline-rhel7-test.uscis.dhs.gov mfetpd[4732]: error: F... Hint: Some lines were ellipsized, use -l to show in full.
However, the keypair works when I use it to ssh from the shell.
Charles Smith
added a comment - - edited ifernandezcalvo
Thank you for all your updates on this issue.
Yes, I'm 1000% sure that it's the PRIVATE key(not PUBLIC) that I'm copying into the SSH credential field that asks for the PRIVATE key. I've created another keypair( 61356-key & 61356-key.pub ) and able to ssh without issue from the Master to the Build(slave) system using this test keypair. However, the same keypair isn't working when used by the shh credential plugin to launch the build agent. I've attached the keypair for your testing/visibility.
If this is an issue with copying/pasting the keypair with added Unix line breaks, then what is a workaround for this? I'm a Unix guy so I copy the PRIVATE key from my Putty terminal directly into the SSH credential field. Is there another way of doing this to test this theory?
Other than that, we're currently running RHEL7.8:
$ ssh -V
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
We also run security McAfee software that may potentially be an issue when creating the keypair:
# systemctl status mfeespd.service; systemctl status mfetpd.service
● mfeespd.service - McAfee Endpoint Security Platform for Linux
Loaded: loaded (/usr/lib/systemd/system/mfeespd.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2020-04-15 14:13:48 CDT; 5 days ago
Docs: man:mfeespd(8)
Main PID: 4574 (mfeespd)
Tasks: 9
Memory: 2.9M
CGroup: /system.slice/mfeespd.service
└─4574 /opt/McAfee/ens/esp/bin/mfeespdApr 15 14:13:46 ansible-baseline-rhel7-test.uscis.dhs.gov systemd[1]: Starting M...
Apr 15 14:13:46 ansible-baseline-rhel7-test.uscis.dhs.gov prepare-mfeesp.sh[4561]: ...
Apr 15 14:13:48 ansible-baseline-rhel7-test.uscis.dhs.gov systemd[1]: Started Mc...
Hint: Some lines were ellipsized, use -l to show in full.
● mfetpd.service - McAfee Endpoint Security for Linux Threat Prevention
Loaded: loaded (/usr/lib/systemd/system/mfetpd.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2020-04-15 14:13:49 CDT; 5 days ago
Docs: man:mfetpd(8)
Main PID: 4732 (mfetpd)
Tasks: 49
Memory: 1.4G
CGroup: /system.slice/mfetpd.service
├─ 4732 /opt/McAfee/ens/tp/bin/mfetpd
├─ 4777 /opt/McAfee/ens/tp/bin/mfetpd
├─20076 /opt/McAfee/ens/tp/bin/mfetpd
└─20084 /opt/McAfee/ens/tp/bin/mfetpdApr 15 14:13:49 ansible-baseline-rhel7-test.uscis.dhs.gov systemd[1]: Starting M...
Apr 15 14:13:49 ansible-baseline-rhel7-test.uscis.dhs.gov systemd[1]: Started Mc...
Apr 15 14:13:50 ansible-baseline-rhel7-test.uscis.dhs.gov mfetpd[4732]: error: F...
Hint: Some lines were ellipsized, use -l to show in full.
However, the keypair works when I use it to ssh from the shell.
Charles Smith
added a comment - Hello All - Just checking for any more responses or updates on this? Where to go from here? How to test if erroneous characters are being added to the priv key section of that plugin? Any more information would be appreciated. Thanks.
Charles Smith
added a comment - jvz ifernandezcalvo
Hello All - Just checking for any more responses or updates on this? Where to go from here? How to test if erroneous characters are being added to the priv key section of that plugin? Any more information would be appreciated. Thanks. Can you try generating a key using openssl genrsa -out private.pem 4096 and see if that works? I wonder if there's been some sort of key format change in ssh-keygen. I really wish this were using OpenSSH, but it's not.
In fact, I wonder if that's the issue. It says the PEM is of an unknown type, but an SSH private key is usually encoded via RFC4716 rather than PEM (which ssh-keygen(1) says is the "legacy" format). There's also PCKS8.
Version: OpenSSH_8.1p1, LibreSSL 2.7.3
Based on version differences, it seems like your fips-patched version of OpenSSH is likely what's causing the issue. There might be a missing SSH key format being supported by trilead?
Charles Smith
added a comment - Thanks for the updates. How do you want me to extract the public key from the pem file?
Using ssh-keygen:
ssh-keygen-y -f key.pem > key.pub
Or using openssl:
openssl rsa -in key.pem -pubout -out pubkey.pem
Note that when using openssl, the public key doesn't have the standard ssh-rsa heading that normal ssh public keys use.
The fips-patched version of OpenSSH makes sense. I've attached a test keypair earlier that you can try and test with to see if the issue can be replicated. I'm not using that keypair.
Charles Smith
added a comment - jvz
Thanks for the updates. How do you want me to extract the public key from the pem file?
Using ssh-keygen :
ssh-keygen-y -f key.pem > key.pub
Or using openssl :
openssl rsa -in key.pem -pubout -out pubkey.pem
Note that when using openssl , the public key doesn't have the standard ssh-rsa heading that normal ssh public keys use.
The fips-patched version of OpenSSH makes sense. I've attached a test keypair earlier that you can try and test with to see if the issue can be replicated. I'm not using that keypair. You can convert an existing key with ssh-keygen -p -f private_key_file -m pem
And I meant to try and see if the RSA keys generated from openssl are decoded properly by Jenkins compared to the ones being generated via ssh-keygen.
Alright, I've done some digging, and here's what I've found. The key you attached (61356) is encoded in PKCS#8 which can be seen by the "BEGIN PRIVATE KEY" header. What is usually used in SSH is PKCS#1 (i.e., no special encoding apparently) which has the header "BEGIN RSA PRIVATE KEY". The former format encodes the algorithm in the base64 data while the latter is more of a naked format with the algorithm in the header. I've also seen "BEGIN OPENSSH PRIVATE KEY" which appears to be the newer format which is used by ed25519 and other ciphers.
One workaround you could try (in production, too!) would be to convert your private key to the old PEM format. You can use openssl rsa -in private.key -out private.pem -outform PEM to convert the key encoding format which should be supported by trilead. If that doesn't work, then we may have an interesting problem.
Looking at trilead, I see that RSAKeyAlgorithm only supports SHA1, and it doesn't even support aes-128/256-gcm, so it seems somewhat dated in algorithm support as it is. I'm kind of wondering why this library is the default rather than Apache SSHD (which was used for some other SSH functionality in Jenkins).
I see there's support for the old pem file format and the new openssh (RFC 4716) format, but no explicit support for PKCS#8. Could you try out the other formats for the same key to see if it works? If it doesn't, then I think we have some missing crypto primitives here.
Charles Smith
added a comment - Thanks for the new information.
Just to confirm with you:
1.) Do you want me to use the ssh-keygen or openssl command to generate the public key? I've already created the key.pem file with the openssl genrsa command?
2.) Do you want me to run openssl rsa -in private.key -out private.pem -outform PEM to change the private key format to pem and then extract the public key from the converted pem file?
Sorry I want to make sure that I do this right.
Charles Smith
added a comment - jvz
Thanks for the new information.
Just to confirm with you:
1.) Do you want me to use the ssh-keygen or openssl command to generate the public key? I've already created the key.pem file with the openssl genrsa command?
2.) Do you want me to run openssl rsa -in private.key -out private.pem -outform PEM to change the private key format to pem and then extract the public key from the converted pem file?
Sorry I want to make sure that I do this right. Take your original private key (or any of the private keys you've created through ssh-keygen) and use the openssl command to convert its format to PEM. Then try that PEM key in Jenkins.
Charles Smith
added a comment - I'm unable to get this to work using the 61356-key private key.
To replicate:
I ran the following to convert the key to PEM format(61356-key.pem):
jenkins@dbis-admin:/var/lib/jenkins/.ssh/test $ openssl rsa -in 61356-key -out 61356-key.pem -outform PEM writing RSA key jenkins@dbis-admin:/var/lib/jenkins/.ssh/test $ ls -l total 8 -rw-------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key -rw-------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key.pem
Then, copied(selected) the new key in vi format and pasted it into the SSH credential key box. I launch the build agent and get the same PEM error. Both the original private key and pem file are the same exact (i.e. no changes to the BEGIN PRIVATE KEY headers.)
Charles Smith
added a comment - jvz
I'm unable to get this to work using the 61356-key private key.
To replicate:
I ran the following to convert the key to PEM format( 61356-key.pem ):
jenkins@dbis-admin:/ var /lib/jenkins/.ssh/test
$ openssl rsa -in 61356-key -out 61356-key.pem -outform PEM
writing RSA key
jenkins@dbis-admin:/ var /lib/jenkins/.ssh/test
$ ls -l
total 8
-rw-------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key
-rw-------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key.pem
Then, copied(selected) the new key in vi format and pasted it into the SSH credential key box. I launch the build agent and get the same PEM error. Both the original private key and pem file are the same exact ( i.e. no changes to the BEGIN PRIVATE KEY headers. ) Well I'll be damned. Seems like this fips version just doesn't let you use the old format. Could you try exporting it in the RFC4716 format? ssh-keygen -f 61356-key -p -m RFC4716 and save without a password. Check the updated key file and make sure it has a "BEGIN OPENSSH PRIVATE KEY" header instead of "BEGIN PRIVATE KEY". If it still has the BEGIN PRIVATE KEY, then it really seems locked down to a single key format, period.
Charles Smith
added a comment - Yes it's confirmed. The 61356-key still has the "BEGIN PRIVATE KEY" header after attempting to change the format to RFC4716:
$ ls -l
total 8
rw------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key
rw------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key.pem
$ grep "BEGIN\ PRIVATE\ KEY" 61356-key
----BEGIN PRIVATE KEY----
$ date
Thu Apr 23 22:58:12 EDT 2020
$ ssh-keygen -f 61356-key -p -m RFC4716
Enter new passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved with the new passphrase.
$ ls -l
total 8
rw------. 1 jenkins jenkins 3272 Apr 23 22:58 61356-key
rw------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key.pem
$ grep "BEGIN\ PRIVATE\ KEY" 61356-key
----BEGIN PRIVATE KEY----
${code}
Charles Smith
added a comment - jvz
Yes it's confirmed. The 61356-key still has the " BEGIN PRIVATE KEY " header after attempting to change the format to RFC4716:
$ ls -l
total 8
rw ------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key
rw ------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key.pem
$ grep "BEGIN\ PRIVATE\ KEY" 61356-key
---- BEGIN PRIVATE KEY ----
$ date
Thu Apr 23 22:58:12 EDT 2020
$ ssh-keygen -f 61356-key -p -m RFC4716
Enter new passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved with the new passphrase.
$ ls -l
total 8
rw ------. 1 jenkins jenkins 3272 Apr 23 22:58 61356-key
rw ------. 1 jenkins jenkins 3272 Apr 23 18:33 61356-key.pem
$ grep "BEGIN\ PRIVATE\ KEY" 61356-key
---- BEGIN PRIVATE KEY ----
${code}
I make a test with the original keys, then I've converted the original key with openssl and the issue is resolved.
Here is the environment I made for the tests https://github.com/kuisathaverat/jenkins-issues/tree/master/JENKINS-61356
This are the steps to start the environment
git clone https://github.com/kuisathaverat/jenkins-issues.git
cd jenkins-issues/JENKINS-61356
make clean start
The original key is in https://github.com/kuisathaverat/jenkins-issues/blob/master/JENKINS-61356/ssh-agent/ssh/rsa-key.org
I've converted it with the following commands, then I've updated my JCasC configuration and everything works as expected.
openssl version LibreSSL 2.8.3 openssl rsa -in rsa-key.org -out rsa-key
The PKCS#8 (BEGIN PRIVATE KEY) is not supported. There are other formats supported see https://github.com/jenkinsci/trilead-ssh2/tree/master/test/com/trilead/ssh2/signature
Charles Smith
added a comment - That's the problem the key never gets converted when I run those openssl commands. I noticed that you haven't tested yet with the attached 61356-key keypairs that I have attached to this ticket. Try your test on that keypair as I cannot get it to convert.
# openssl version OpenSSL 1.0.2k-fips 26 Jan 2017
Charles Smith
added a comment - ifernandezcalvo
That's the problem the key never gets converted when I run those openssl commands. I noticed that you haven't tested yet with the attached 61356-key keypairs that I have attached to this ticket. Try your test on that keypair as I cannot get it to convert.
# openssl version
OpenSSL 1.0.2k-fips 26 Jan 2017
I've attached the 61356 key converted into the ssh2 format. It's the newer version of the same file.
I tested the files uploaded 3 days ago https://issues.jenkins-ci.org/secure/attachment/51008/61356-key.pub.txt and https://issues.jenkins-ci.org/secure/attachment/51007/61356-key.txt
I just updated the test repo with three agents using the three different formats we have in the issue:
- "----
BEGIN PRIVATE KEY----" failure - "----
BEGIN OPENSSH PRIVATE KEY----" success - "----
BEGIN RSA PRIVATE KEY----" success
Alright, then I think we've isolated the issue. trilead doesn't support PKCS#8 private keys.
Charles Smith
added a comment - When I use your converted 61356 key to launch the build agent, I now receive a different error:
SSHLauncher{host='prod-slave-01.tt.com', port=22, credentialsId='jenkins-test-ssh-key', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=60, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
[04/24/20 11:13:58] [SSH] Opening SSH connection to dbis-slave-01.uscis.dhs.gov:22.
[04/24/20 11:13:58] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:jenkins-test-ssh-key/method:publickey)
[04/24/20 11:13:59] [SSH] Authentication failed.
Authentication failed.
[04/24/20 11:13:59] Launch failed - cleaning up connection
[04/24/20 11:13:59] [SSH] Connection closed.
However, I'm able to successfully SSH into the build agent from the jenkins master shell using the same converted key. Which openssl command did you use to confirm that the privkeys are PKCS#8?
Charles Smith
added a comment - When I use your converted 61356 key to launch the build agent, I now receive a different error:
SSHLauncher{host= 'prod-slave-01.tt.com' , port=22, credentialsId= 'jenkins-test-ssh-key' , jvmOptions= '', javaPath=' ', prefixStartSlaveCmd=' ', suffixStartSlaveCmd=' ', launchTimeoutSeconds=60, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay= true , trackCredentials= true }
[04/24/20 11:13:58] [SSH] Opening SSH connection to dbis-slave-01.uscis.dhs.gov:22.
[04/24/20 11:13:58] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:jenkins-test-ssh-key/method:publickey)
[04/24/20 11:13:59] [SSH] Authentication failed.
Authentication failed.
[04/24/20 11:13:59] Launch failed - cleaning up connection
[04/24/20 11:13:59] [SSH] Connection closed.
However, I'm able to successfully SSH into the build agent from the jenkins master shell using the same converted key. Which openssl command did you use to confirm that the privkeys are PKCS#8? I converted the key using the command I pasted earlier:
ssh-keygen -f 61356-key.txt -p -m RFC4716
When prompted, I used an empty password. I tried using some other flags besides -p to convert the private key, but the other flags seem to only output the public key.
And to confirm that the key is PKCS#8, you can tell because it has the header "BEGIN PRIVATE KEY" with no specific type of key mentioned in the header. That data is encoded in the base64 contents.
Charles Smith
added a comment - Ok that command doesn't work on my system when trying to convert the key. Is this because of the fips enabled versions of SSH and SSL that's being used?
Charles Smith
added a comment - jvz
Ok that command doesn't work on my system when trying to convert the key. Is this because of the fips enabled versions of SSH and SSL that's being used? 
jvz
Hello Matt, can you give me some type of update on this issue? I've tried all the recommendations that I could find on this site and on google with no success. If you require anymore info please let me know. Thanks.