Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-32120

With ssh-agent 1.9, local builds fail with "Unable to create OpenSSL PBDKF"

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • Jenkins 1.640 running on docker from onesysadmin/jenkins-docker-executors
      All plugins up to date

      When running a local build (on the master), the build fails to load the ssh-agent:

      Started by an SCM change
      Building in workspace /var/jenkins_home/workspace/production - client (prod3)
      [ssh-agent] Looking for ssh-agent implementation...
      [ssh-agent]   Java/JNR ssh-agent
      [ssh-agent] Started.
      [ssh-agent] Using credentials git (Bitbucket checkout)
      [ssh-agent] Unable to read key: Unable to create OpenSSL PBDKF: PBKDF-OpenSSL SecretKeyFactory not available
      org.bouncycastle.openssl.PEMException: Unable to create OpenSSL PBDKF: PBKDF-OpenSSL SecretKeyFactory not available
      	at org.bouncycastle.openssl.jcajce.PEMUtilities.getKey(Unknown Source)
      	at org.bouncycastle.openssl.jcajce.PEMUtilities.getKey(Unknown Source)
      	at org.bouncycastle.openssl.jcajce.PEMUtilities.crypt(Unknown Source)
      	at org.bouncycastle.openssl.jcajce.JcePEMDecryptorProviderBuilder$1$1.decrypt(Unknown Source)
      	at org.bouncycastle.openssl.PEMEncryptedKeyPair.decryptKeyPair(Unknown Source)
      	at com.cloudbees.jenkins.plugins.sshagent.jna.JNRRemoteAgent.addIdentity(JNRRemoteAgent.java:98)
      	at com.cloudbees.jenkins.plugins.sshagent.SSHAgentBuildWrapper$SSHAgentEnvironment.add(SSHAgentBuildWrapper.java:388)
      	at com.cloudbees.jenkins.plugins.sshagent.SSHAgentBuildWrapper.preCheckout(SSHAgentBuildWrapper.java:227)
      	at jenkins.scm.SCMCheckoutStrategy.preCheckout(SCMCheckoutStrategy.java:76)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528)
      	at hudson.model.Run.execute(Run.java:1738)
      	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      	at hudson.model.ResourceController.execute(ResourceController.java:98)
      	at hudson.model.Executor.run(Executor.java:410)
      Caused by: java.security.NoSuchAlgorithmException: PBKDF-OpenSSL SecretKeyFactory not available
      	at javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:121)
      	at javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:159)
      	at org.bouncycastle.jcajce.util.DefaultJcaJceHelper.createSecretKeyFactory(Unknown Source)
      	... 14 more
      [ssh-agent] Using credentials deploy (Deployment key to work with Minion Master)
      

      Then the build looks like its continuing normally, but the ssh key was not actually loaded and as a result the build fails:

      + ssh -o StrictHostKeyChecking=no deploy@xxx.xxx.xxx some command
      Permission denied (publickey).
      + ...
      + exit 5
      + exit 5
      Build step 'Execute shell' marked build as failure
      [ssh-agent] Stopped.
      

      Running the same job on a slave works fine:

      Started by user Oded Arbel
      Building remotely on trusty (i-5827fe82) (trusty docker) in workspace /srv/build/workspace/production - client (prod3)
      [ssh-agent] Looking for ssh-agent implementation...
      [ssh-agent]   Java/JNR ssh-agent
      [ssh-agent] Started.
      [ssh-agent] Using credentials git (Bitbucket checkout)
      [ssh-agent] Using credentials deploy (Deployment key to work with Minion Master)
      

      Downgrading the ssh-agent plugin to version 1.8 worked around the problem. After downgrade:

      Started by user Oded Arbel
      Building in workspace /var/jenkins_home/workspace/production - client (prodhk)
      [ssh-agent] Using credentials git (Bitbucket checkout)
      [ssh-agent] Using credentials deploy (Deployment key to work with Minion Master)
      [ssh-agent] Looking for ssh-agent implementation...
      [ssh-agent]   Java/JNR ssh-agent
      [ssh-agent] Started.
      

      Also see here: https://www.botbot.me/freenode/jenkins/2015-12-16/?msg=56268453&page=4

          [JENKINS-32120] With ssh-agent 1.9, local builds fail with "Unable to create OpenSSL PBDKF"

          Oded Arbel created issue -

          Jesper Markenstam added a comment - - edited

          The issue is present with a combination of ssh credentials plugin 1.11 + ssh agent plugin 1.9. Downgrading both to 1.10 / 1.8 respectively, restored expected functionality on clean installation with rpm jenkins-1.625.3-1.1.noarch, centos 67, oracle jdk 1.8.0_60.

          Jesper Markenstam added a comment - - edited The issue is present with a combination of ssh credentials plugin 1.11 + ssh agent plugin 1.9. Downgrading both to 1.10 / 1.8 respectively, restored expected functionality on clean installation with rpm jenkins-1.625.3-1.1.noarch, centos 67, oracle jdk 1.8.0_60.
          twolfart made changes -
          Comment [ Downgrading to ssh agent plugin 1.8 worked for me, ssh credentials plugin is still at 1.11 (latest). ]

          bfhzog6i added a comment -

          According to Freenode channel log the problem happens, when the SSH key protected with Passphrase is being used: https://botbot.me/freenode/jenkins/2015-12-16/?page=3

          bfhzog6i added a comment - According to Freenode channel log the problem happens, when the SSH key protected with Passphrase is being used: https://botbot.me/freenode/jenkins/2015-12-16/?page=3

          Philipp Kyeck added a comment -

          Having the same problems and can confirm, that it is only when your key is passphrase protected. Without passphrase everything is working as expected.

          Jenkins v1.625.3, Ubuntu 14.04, java version 1.7.0_91 OpenJDK Runtime Environment

          Philipp Kyeck added a comment - Having the same problems and can confirm, that it is only when your key is passphrase protected. Without passphrase everything is working as expected. Jenkins v1.625.3, Ubuntu 14.04, java version 1.7.0_91 OpenJDK Runtime Environment

          Stan Towianski added a comment - - edited

          Hi. A fix, but very strange things going on for me.
          JM=jenkins master, BS=linux bld slave, somehost=ssh to this host
          BldUser=jenkins build user SshUser=ssh user

          I think I fixed or found a way passed this problem.
          with SshUser private key I did: openssl rsa -in id_rsa -check > id_rsa.stan

          my guess is while this checks my private key it spits out a different version/type of the key??
          If I use that private key for my ssh job my error goes away and it works.
          I added a new private key credential for SshUser with passphrase, leaving other one still there.

          But here is the weirdness or what I do not understand.
          BS node connects JM with old version SshUser key.
          on JM I have another ssh key for BldUser for the buildjob.
          In BuildJob i changed ssh-agent from BldUser to SshUser new priv key. This made it work!
          Job does ssh -q somehost "hostname; pwd; id"
          id is for SshUser as I expect.

          but if I change the buildjob ssh-agent back to BldUser - it still works and the ssh user it uses is
          from the BS ssh-agent setup (SshUser)!

          17:51:15 [ssh-agent] Looking for ssh-agent implementation...
          17:51:15 [ssh-agent] Java/JNR ssh-agent
          17:51:17 [ssh-agent] Started.
          17:51:17 [ssh-agent] Using credentials BldUser <- from buildjob
          17:51:17 [my-box] $ /bin/sh -x /tmp/hudson2794482700022152346.sh
          17:51:18 ++ hostname
          17:51:18 + ssh -q somehost 'hostname; pwd; id'
          17:51:18 somehost
          17:51:18 /home/SshUser <- why from BS config (SshUser) and not from buildjob(BldUser)??
          17:51:18 uid=910(SshUser) gid=25(xxx) groups=25(xxx)
          17:51:18 + rc=0

          Is there some jenkins key caching and using going on here?
          If I take out using ssh-agent key from build job it will not work, so it does need something in the buildjob.

          p.s. earlier I also added this line to java.security per plugin page but it did not fix it then:
          security.provider.10=org.bouncycastle.jce.provider.BouncyCastleProvider

          jenkins 1.625.3
          ssh agent plugin 1.9
          ssh credentials plugin 1.11
          java 8

          Stan Towianski added a comment - - edited Hi. A fix, but very strange things going on for me. JM=jenkins master, BS=linux bld slave, somehost=ssh to this host BldUser=jenkins build user SshUser=ssh user I think I fixed or found a way passed this problem. with SshUser private key I did: openssl rsa -in id_rsa -check > id_rsa.stan my guess is while this checks my private key it spits out a different version/type of the key?? If I use that private key for my ssh job my error goes away and it works. I added a new private key credential for SshUser with passphrase, leaving other one still there. But here is the weirdness or what I do not understand. BS node connects JM with old version SshUser key. on JM I have another ssh key for BldUser for the buildjob. In BuildJob i changed ssh-agent from BldUser to SshUser new priv key. This made it work! Job does ssh -q somehost "hostname; pwd; id" id is for SshUser as I expect. but if I change the buildjob ssh-agent back to BldUser - it still works and the ssh user it uses is from the BS ssh-agent setup (SshUser)! 17:51:15 [ssh-agent] Looking for ssh-agent implementation... 17:51:15 [ssh-agent] Java/JNR ssh-agent 17:51:17 [ssh-agent] Started. 17:51:17 [ssh-agent] Using credentials BldUser <- from buildjob 17:51:17 [my-box] $ /bin/sh -x /tmp/hudson2794482700022152346.sh 17:51:18 ++ hostname 17:51:18 + ssh -q somehost 'hostname; pwd; id' 17:51:18 somehost 17:51:18 /home/SshUser <- why from BS config (SshUser) and not from buildjob(BldUser)?? 17:51:18 uid=910(SshUser) gid=25(xxx) groups=25(xxx) 17:51:18 + rc=0 Is there some jenkins key caching and using going on here? If I take out using ssh-agent key from build job it will not work, so it does need something in the buildjob. p.s. earlier I also added this line to java.security per plugin page but it did not fix it then: security.provider.10=org.bouncycastle.jce.provider.BouncyCastleProvider jenkins 1.625.3 ssh agent plugin 1.9 ssh credentials plugin 1.11 java 8

          so, it looks like what I did to get around the problem is create another version of our private key that stripped out the passphrase. It's a workaround but does not fix the problem of passphrase not working in jenkins credential plugin.

          Stan Towianski added a comment - so, it looks like what I did to get around the problem is create another version of our private key that stripped out the passphrase. It's a workaround but does not fix the problem of passphrase not working in jenkins credential plugin.

          Adrian G added a comment -

          I have the same issue and can help with troubleshooting if useful. The problem that I've encountered seems to be that the SSH Agent plugin interferes with the Git plugin. With "SSH Agent" checked in the project definition, the ssh agent loads first, and fails to load keys because of the SecretKeyFactory error. The Git plugin then looks like it tries to use the ssh-agent configuration instead of the SSH credentials directly, and since there are no credentials, the whole job fails.

          How can we resolve the issue with SecretKeyFactory not being available within the environment? That seems like the place to start.

          Adrian G added a comment - I have the same issue and can help with troubleshooting if useful. The problem that I've encountered seems to be that the SSH Agent plugin interferes with the Git plugin. With "SSH Agent" checked in the project definition, the ssh agent loads first, and fails to load keys because of the SecretKeyFactory error. The Git plugin then looks like it tries to use the ssh-agent configuration instead of the SSH credentials directly, and since there are no credentials, the whole job fails. How can we resolve the issue with SecretKeyFactory not being available within the environment? That seems like the place to start.

          [ssh-agent] Using credentials myUser (with private key and passphrase)
          [ssh-agent] Looking for ssh-agent implementation...
          [ssh-agent] Java/JNR ssh-agent
          [ssh-agent] Unable to read key: exception using cipher - please check password and data.
          org.bouncycastle.openssl.EncryptionException: exception using cipher - please check password and data.
          at org.bouncycastle.openssl.PEMUtilities.crypt(Unknown Source)
          at org.bouncycastle.openssl.PEMUtilities.crypt(Unknown Source)
          at org.bouncycastle.openssl.PEMReader$KeyPairParser.readKeyPair(Unknown Source)
          at org.bouncycastle.openssl.PEMReader$RSAKeyPairParser.parseObject(Unknown Source)
          at org.bouncycastle.openssl.PEMReader.readObject(Unknown Source)
          at com.cloudbees.jenkins.plugins.sshagent.jna.JNRRemoteAgent.addIdentity(JNRRemoteAgent.java:92)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:497)
          at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
          at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
          at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
          at hudson.remoting.UserRequest.perform(UserRequest.java:120)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
          Caused by: java.lang.SecurityException: JCE cannot authenticate the provider BC
          at javax.crypto.Cipher.getInstance(Cipher.java:657)
          ... 21 more
          Caused by: java.util.jar.JarException: Class is on the bootclasspath

          These are are related as far as I can tell. A teammate figured out Bouncy Castle problem has to be done on a build slave if you are using those. I'm not sure if it needs to be done on the Jenkins master too, but I did it there first already.

          I think java 6 does not work with newer Jenkins? I did not test fix with java 7. I used Java 8

          add these 4 jar files here:

          /…./java/jdk/jdk-1.8u66/jre/lib/ext
          rw-rr- 1 root root 2070477 Feb 25 16:46 bcprov-ext-jdk15on-147.jar
          rw-rr- 1 root root 1997327 Feb 25 16:46 bcprov-jdk15on-147.jar

          /…./java/jdk/jdk-1.8u66/jre/lib/security
          rw-rr- 1 root root 3023 Feb 25 16:46 US_export_policy.jar
          rw-rr- 1 root root 3035 Feb 25 16:46 local_policy.jar

          vi /…./java/jdk/jdk-1.8u66/jre/lib/security/java.security
          add this line after lines like it:
          security.provider.10=org.bouncycastle.jce.provider.BouncyCastleProvider

          A build slave may connect without these files in place on the build slave java, BUT
          a build job using ssh-agent will not work !

          after making these changes, you need to disconnect and reconnect the build slave.
          Then build jobs using ssh-agent with a credential with a passphrase and the private key will work for things like:
          ssh myUser@hostwhatever "hostname; pwd; id"

          Stan Towianski added a comment - [ssh-agent] Using credentials myUser (with private key and passphrase) [ssh-agent] Looking for ssh-agent implementation... [ssh-agent] Java/JNR ssh-agent [ssh-agent] Unable to read key: exception using cipher - please check password and data. org.bouncycastle.openssl.EncryptionException: exception using cipher - please check password and data. at org.bouncycastle.openssl.PEMUtilities.crypt(Unknown Source) at org.bouncycastle.openssl.PEMUtilities.crypt(Unknown Source) at org.bouncycastle.openssl.PEMReader$KeyPairParser.readKeyPair(Unknown Source) at org.bouncycastle.openssl.PEMReader$RSAKeyPairParser.parseObject(Unknown Source) at org.bouncycastle.openssl.PEMReader.readObject(Unknown Source) at com.cloudbees.jenkins.plugins.sshagent.jna.JNRRemoteAgent.addIdentity(JNRRemoteAgent.java:92) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542) at hudson.remoting.UserRequest.perform(UserRequest.java:120) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.SecurityException: JCE cannot authenticate the provider BC at javax.crypto.Cipher.getInstance(Cipher.java:657) ... 21 more Caused by: java.util.jar.JarException: Class is on the bootclasspath These are are related as far as I can tell. A teammate figured out Bouncy Castle problem has to be done on a build slave if you are using those. I'm not sure if it needs to be done on the Jenkins master too, but I did it there first already. I think java 6 does not work with newer Jenkins? I did not test fix with java 7. I used Java 8 add these 4 jar files here: /…./java/jdk/jdk-1.8u66/jre/lib/ext rw-r r - 1 root root 2070477 Feb 25 16:46 bcprov-ext-jdk15on-147.jar rw-r r - 1 root root 1997327 Feb 25 16:46 bcprov-jdk15on-147.jar /…./java/jdk/jdk-1.8u66/jre/lib/security rw-r r - 1 root root 3023 Feb 25 16:46 US_export_policy.jar rw-r r - 1 root root 3035 Feb 25 16:46 local_policy.jar vi /…./java/jdk/jdk-1.8u66/jre/lib/security/java.security add this line after lines like it: security.provider.10=org.bouncycastle.jce.provider.BouncyCastleProvider A build slave may connect without these files in place on the build slave java, BUT a build job using ssh-agent will not work ! after making these changes, you need to disconnect and reconnect the build slave. Then build jobs using ssh-agent with a credential with a passphrase and the private key will work for things like: ssh myUser@hostwhatever "hostname; pwd; id"

          Jesse Glick added a comment -

          stephenconnolly made various changes in 1.9 including to BouncyCastle (I think).

          Best to do builds on slaves anyway.

          Jesse Glick added a comment - stephenconnolly made various changes in 1.9 including to BouncyCastle (I think). Best to do builds on slaves anyway.
          Jesse Glick made changes -
          Assignee New: Stephen Connolly [ stephenconnolly ]

            alobato Alvaro Lobato
            guss77 Oded Arbel
            Votes:
            16 Vote for this issue
            Watchers:
            27 Start watching this issue

              Created:
              Updated:
              Resolved: