Status: Resolved (View Workflow)
Jenkins 1.640 running on docker from onesysadmin/jenkins-docker-executors
All plugins up to date
When running a local build (on the master), the build fails to load the ssh-agent:
Started by an SCM change Building in workspace /var/jenkins_home/workspace/production - client (prod3) [ssh-agent] Looking for ssh-agent implementation... [ssh-agent] Java/JNR ssh-agent [ssh-agent] Started. [ssh-agent] Using credentials git (Bitbucket checkout) [ssh-agent] Unable to read key: Unable to create OpenSSL PBDKF: PBKDF-OpenSSL SecretKeyFactory not available org.bouncycastle.openssl.PEMException: Unable to create OpenSSL PBDKF: PBKDF-OpenSSL SecretKeyFactory not available at org.bouncycastle.openssl.jcajce.PEMUtilities.getKey(Unknown Source) at org.bouncycastle.openssl.jcajce.PEMUtilities.getKey(Unknown Source) at org.bouncycastle.openssl.jcajce.PEMUtilities.crypt(Unknown Source) at org.bouncycastle.openssl.jcajce.JcePEMDecryptorProviderBuilder$1$1.decrypt(Unknown Source) at org.bouncycastle.openssl.PEMEncryptedKeyPair.decryptKeyPair(Unknown Source) at com.cloudbees.jenkins.plugins.sshagent.jna.JNRRemoteAgent.addIdentity(JNRRemoteAgent.java:98) at com.cloudbees.jenkins.plugins.sshagent.SSHAgentBuildWrapper$SSHAgentEnvironment.add(SSHAgentBuildWrapper.java:388) at com.cloudbees.jenkins.plugins.sshagent.SSHAgentBuildWrapper.preCheckout(SSHAgentBuildWrapper.java:227) at jenkins.scm.SCMCheckoutStrategy.preCheckout(SCMCheckoutStrategy.java:76) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528) at hudson.model.Run.execute(Run.java:1738) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:410) Caused by: java.security.NoSuchAlgorithmException: PBKDF-OpenSSL SecretKeyFactory not available at javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:121) at javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:159) at org.bouncycastle.jcajce.util.DefaultJcaJceHelper.createSecretKeyFactory(Unknown Source) ... 14 more [ssh-agent] Using credentials deploy (Deployment key to work with Minion Master)
Then the build looks like its continuing normally, but the ssh key was not actually loaded and as a result the build fails:
+ ssh -o StrictHostKeyChecking=no firstname.lastname@example.org some command Permission denied (publickey). + ... + exit 5 + exit 5 Build step 'Execute shell' marked build as failure [ssh-agent] Stopped.
Running the same job on a slave works fine:
Started by user Oded Arbel Building remotely on trusty (i-5827fe82) (trusty docker) in workspace /srv/build/workspace/production - client (prod3) [ssh-agent] Looking for ssh-agent implementation... [ssh-agent] Java/JNR ssh-agent [ssh-agent] Started. [ssh-agent] Using credentials git (Bitbucket checkout) [ssh-agent] Using credentials deploy (Deployment key to work with Minion Master)
Downgrading the ssh-agent plugin to version 1.8 worked around the problem. After downgrade:
Started by user Oded Arbel Building in workspace /var/jenkins_home/workspace/production - client (prodhk) [ssh-agent] Using credentials git (Bitbucket checkout) [ssh-agent] Using credentials deploy (Deployment key to work with Minion Master) [ssh-agent] Looking for ssh-agent implementation... [ssh-agent] Java/JNR ssh-agent [ssh-agent] Started.
Also see here: https://www.botbot.me/freenode/jenkins/2015-12-16/?msg=56268453&page=4
- links to
According to Freenode channel log the problem happens, when the SSH key protected with Passphrase is being used: https://botbot.me/freenode/jenkins/2015-12-16/?page=3
Having the same problems and can confirm, that it is only when your key is passphrase protected. Without passphrase everything is working as expected.
Jenkins v1.625.3, Ubuntu 14.04, java version 1.7.0_91 OpenJDK Runtime Environment
Hi. A fix, but very strange things going on for me.
JM=jenkins master, BS=linux bld slave, somehost=ssh to this host
BldUser=jenkins build user SshUser=ssh user
I think I fixed or found a way passed this problem.
with SshUser private key I did: openssl rsa -in id_rsa -check > id_rsa.stan
my guess is while this checks my private key it spits out a different version/type of the key??
If I use that private key for my ssh job my error goes away and it works.
I added a new private key credential for SshUser with passphrase, leaving other one still there.
But here is the weirdness or what I do not understand.
BS node connects JM with old version SshUser key.
on JM I have another ssh key for BldUser for the buildjob.
In BuildJob i changed ssh-agent from BldUser to SshUser new priv key. This made it work!
Job does ssh -q somehost "hostname; pwd; id"
id is for SshUser as I expect.
but if I change the buildjob ssh-agent back to BldUser - it still works and the ssh user it uses is
from the BS ssh-agent setup (SshUser)!
17:51:15 [ssh-agent] Looking for ssh-agent implementation...
17:51:15 [ssh-agent] Java/JNR ssh-agent
17:51:17 [ssh-agent] Started.
17:51:17 [ssh-agent] Using credentials BldUser <- from buildjob
17:51:17 [my-box] $ /bin/sh -x /tmp/hudson2794482700022152346.sh
17:51:18 ++ hostname
17:51:18 + ssh -q somehost 'hostname; pwd; id'
17:51:18 /home/SshUser <- why from BS config (SshUser) and not from buildjob(BldUser)??
17:51:18 uid=910(SshUser) gid=25(xxx) groups=25(xxx)
17:51:18 + rc=0
Is there some jenkins key caching and using going on here?
If I take out using ssh-agent key from build job it will not work, so it does need something in the buildjob.
p.s. earlier I also added this line to java.security per plugin page but it did not fix it then:
ssh agent plugin 1.9
ssh credentials plugin 1.11
so, it looks like what I did to get around the problem is create another version of our private key that stripped out the passphrase. It's a workaround but does not fix the problem of passphrase not working in jenkins credential plugin.
I have the same issue and can help with troubleshooting if useful. The problem that I've encountered seems to be that the SSH Agent plugin interferes with the Git plugin. With "SSH Agent" checked in the project definition, the ssh agent loads first, and fails to load keys because of the SecretKeyFactory error. The Git plugin then looks like it tries to use the ssh-agent configuration instead of the SSH credentials directly, and since there are no credentials, the whole job fails.
How can we resolve the issue with SecretKeyFactory not being available within the environment? That seems like the place to start.
[ssh-agent] Using credentials myUser (with private key and passphrase)
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent] Java/JNR ssh-agent
[ssh-agent] Unable to read key: exception using cipher - please check password and data.
org.bouncycastle.openssl.EncryptionException: exception using cipher - please check password and data.
at org.bouncycastle.openssl.PEMUtilities.crypt(Unknown Source)
at org.bouncycastle.openssl.PEMUtilities.crypt(Unknown Source)
at org.bouncycastle.openssl.PEMReader$KeyPairParser.readKeyPair(Unknown Source)
at org.bouncycastle.openssl.PEMReader$RSAKeyPairParser.parseObject(Unknown Source)
at org.bouncycastle.openssl.PEMReader.readObject(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Caused by: java.lang.SecurityException: JCE cannot authenticate the provider BC
... 21 more
Caused by: java.util.jar.JarException: Class is on the bootclasspath
These are are related as far as I can tell. A teammate figured out Bouncy Castle problem has to be done on a build slave if you are using those. I'm not sure if it needs to be done on the Jenkins master too, but I did it there first already.
I think java 6 does not work with newer Jenkins? I did not test fix with java 7. I used Java 8
add these 4 jar files here:
rw-r r- 1 root root 2070477 Feb 25 16:46 bcprov-ext-jdk15on-147.jar
rw-r r- 1 root root 1997327 Feb 25 16:46 bcprov-jdk15on-147.jar
rw-r r- 1 root root 3023 Feb 25 16:46 US_export_policy.jar
rw-r r- 1 root root 3035 Feb 25 16:46 local_policy.jar
add this line after lines like it:
A build slave may connect without these files in place on the build slave java, BUT
a build job using ssh-agent will not work !
after making these changes, you need to disconnect and reconnect the build slave.
Then build jobs using ssh-agent with a credential with a passphrase and the private key will work for things like:
ssh myUser@hostwhatever "hostname; pwd; id"
stephenconnolly made various changes in 1.9 including to BouncyCastle (I think).
Best to do builds on slaves anyway.
So the issue here is that https://issues.jenkins-ci.org/browse/JENKINS-30110 required a lot of plugins upgrade the agreedupon bouncycastle implementation version... and bouncycastle is notorious for breaking compatibility.
So you really need to either stay all below or all above the bouncycastle version change
These are great news. If only I can do "yum update jenkins", that currently doesn't work due INFRA-685.
aik099 take care that the upgrade of Jenkins core >= 1.648 upgrades the library Bouncycastle which fixes this issue BUT as mentioned by stephenconnolly in
JENKINS-30110 it may/will create incompatibilities with all plugins that may use directly bouncycastle and the recent APIs they broke (again).
Various jenkins developers are trying to identify all impacted plugins and to propose a fix which may be better in long term.
I've asked DevOps team to do the upgrade and what they did is downloaded RPM package by hand and installed it. Now I can confirm, that issue (with ssh-agent) is indeed solved for me as well.
Not happening for me. Not sure if it was fixed for other people participating in discussion.
I get this issue with Java 8 but not with Java 7 with 1.651.2 LTS
Me too, still having this issue.
[Pipeline] sshagent [ssh-agent] Using credentials prova [ssh-agent] Looking for ssh-agent implementation... [ssh-agent] Java/JNR ssh-agent [ssh-agent] Unable to read key: Unable to create OpenSSL PBDKF: PBKDF-OpenSSL SecretKeyFactory not available org.bouncycastle.openssl.PEMException: Unable to create OpenSSL PBDKF: PBKDF-OpenSSL SecretKeyFactory not available at org.bouncycastle.openssl.jcajce.PEMUtilities.getKey(Unknown Source)
SSH Agent Plugin 1.11
SSH Credentials Plugin 1.12
Running on slave
The problems are related with the Bouncy Castle version conflicts, the solution is to use Bouncy Castle API plugin to do all the Bouncy Castle related stuff. See Bouncy Castle API Plugin.
Some of the problems are related with BC not being correctly registered on build agents, I've submitted a PR hopefully fixing those cases: PR-14
Because I'm really a newbie, this means that from Update Center in Jenkins it will be possible to download a new release?
This means that the code for the fix has been submitted. It has to be reviewed and after that a released can be cut so you have it available on the Update Center.
The issue is present with a combination of ssh credentials plugin 1.11 + ssh agent plugin 1.9. Downgrading both to 1.10 / 1.8 respectively, restored expected functionality on clean installation with rpm jenkins-1.625.3-1.1.noarch, centos 67, oracle jdk 1.8.0_60.