Details
-
Bug
-
Status: Open (View Workflow)
-
Minor
-
Resolution: Unresolved
-
None
-
OS: centos
Jenkins: 2.223
OpenJDK 1.8.0_242
credentials-binding: 1.21
ssh-steps: 2.0.0
Description
Hi.
I am currently working on a Jenkins pipeline and trying to move all credential handling in the domain of the jenkins instance.
To be able to execute an ssh command on another machine we use private key authentication. I tried to accomplish this by combining credentials-binding and ssh-steps but ran into a problem I couldn't find a solution to.
I basicly do this:
node { def remote = [:] remote.name = "integration_server" remote.host = integration_server remote.allowAnyHosts = true withCredentials([sshUserPrivateKey(credentialsId: 'myCredentialId', keyFileVariable: 'identity', passphraseVariable: '', usernameVariable: 'userName')]) { remote.user = userName remote.identityFile = identity sshCommand remote: remote, command: "do stuff" } ... withCredentials([sshUserPrivateKey(credentialsId: 'myCredentialId', keyFileVariable: 'identity', passphraseVariable: '', usernameVariable: 'userName')]) { remote.user = userName remote.identityFile = identity sshCommand remote: remote, command: "do some other stuff" } }
The first block runs fine. The second block fails with a FileNotFoundException looking for a temporary "secretFile".
My guess is, that credential-binding reuses the reference to the temporary file from the first block during execution of the second block while simultaniously deleting it after the first block ends.
I would like to be able to define multiple indipendent withCredential steps to omit wrapping the whole script in one large block.
Attachments
Activity
I am getting the same error. I am using Jenkins 2.235.3. Running Jenkins under windows 10 and agent under Centos 7.6.
Running my jobs all under the Centos agent, here is the log.
java.io.FileNotFoundException: C:/Jenkins/workspace/pipeline_test@tmp/secretFiles/a338eb6c-9e3e-4c8d-8adc-5d30f3be374a/ssh-key-keyFileName (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at java.io.FileInputStream.<init>(FileInputStream.java:93) at com.jcraft.jsch.Util.fromFile(Util.java:508) at com.jcraft.jsch.KeyPair.load(KeyPair.java:540) Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to Test_agent at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1788) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356) at hudson.remoting.Channel.call(Channel.java:998) at org.jenkinsci.plugins.sshsteps.steps.CommandStep$Execution.run(CommandStep.java:72) at org.jenkinsci.plugins.sshsteps.util.SSHStepExecution.lambda$start$0(SSHStepExecution.java:84) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused: com.jcraft.jsch.JSchException at com.jcraft.jsch.KeyPair.load(KeyPair.java:543) at com.jcraft.jsch.IdentityFile.newInstance(IdentityFile.java:40) at com.jcraft.jsch.JSch.addIdentity(JSch.java:406) at com.jcraft.jsch.JSch.addIdentity(JSch.java:387) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSite.invoke(PojoMetaMethodSite.java:192) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133) at org.hidetake.groovy.ssh.connection.UserAuthentication$Trait$Helper.configureUserAuthentication(UserAuthentication.groovy:36) at org.hidetake.groovy.ssh.connection.UserAuthentication$Trait$Helper$configureUserAuthentication$2.call(Unknown Source) at org.hidetake.groovy.ssh.connection.ConnectionManager.configureUserAuthentication(ConnectionManager.groovy) at org.hidetake.groovy.ssh.connection.UserAuthentication$configureUserAuthentication$1.callCurrent(Unknown Source) at org.hidetake.groovy.ssh.connection.ConnectionManager.connectInternal(ConnectionManager.groovy:104) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:384) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:69) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:190) at org.hidetake.groovy.ssh.connection.ConnectionManager$_connectInternal_closure1.doCall(ConnectionManager.groovy:85) at org.hidetake.groovy.ssh.connection.ConnectionManager$_connectInternal_closure1.doCall(ConnectionManager.groovy) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117) at org.hidetake.groovy.ssh.util.Utility.retry(Utility.groovy:52) at org.hidetake.groovy.ssh.util.Utility$retry.callStatic(Unknown Source) at org.hidetake.groovy.ssh.connection.ConnectionManager.connectInternal(ConnectionManager.groovy:83) at org.hidetake.groovy.ssh.connection.ConnectionManager.connectInternal(ConnectionManager.groovy) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210) at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:166) at org.hidetake.groovy.ssh.connection.ConnectionManager.connect(ConnectionManager.groovy:59) at org.hidetake.groovy.ssh.connection.ConnectionManager$connect$0.call(Unknown Source) at org.hidetake.groovy.ssh.session.SessionTask.wetRun(SessionTask.groovy:61) at org.hidetake.groovy.ssh.session.SessionTask.call(SessionTask.groovy:48) at java_util_concurrent_Callable$call$0.call(Unknown Source) at org.hidetake.groovy.ssh.core.Service.run(Service.groovy:81) at org.hidetake.groovy.ssh.core.Service$run$1.call(Unknown Source) at org.jenkinsci.plugins.sshsteps.SSHService.executeCommand(SSHService.groovy:177) at org.jenkinsci.plugins.sshsteps.steps.CommandStep$Execution$CommandCallable.execute(CommandStep.java:84) at org.jenkinsci.plugins.sshsteps.util.SSHMasterToSlaveCallable.call(SSHMasterToSlaveCallable.java:32) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Finished: FAILURE
Let me know what can I do to help diagnose this issue. It is 100% reproducible.
Seems that I found the reason of the error. Every time that a new block withCredentials is invoked, a new temporal folder is created. But when a sshCommand or any other option from ssh-steps-plugin is used it locks the original temporal directory/file to be used, but as after every withCredentials run that file is deleted, it will not find it, ssh-steps-plugin should use the new temporal directory created and not be fixed with the first temporal path from the first withCredentials block.
I tested this code and also check the files, and checking the error that's what is happening.
Jenkinsfile code:
def fileName def remote = [:] remote.name = "integration_server" remote.user = "ubuntu" remote.host = "10.20.156.167" remote.allowAnyHosts = truepipeline { agent any stages { stage('APP_01') { steps { withCredentials([sshUserPrivateKey(credentialsId: 'shared-dev-key', keyFileVariable: 'identityFileName', passphraseVariable: 'password', usernameVariable: 'username')]) { script { fileName = identityFileName def data = readFile(file: fileName) println data // <--- This print the secret ssh key with no issues remote.identityFile = identityFileName sshCommand remote: remote, command: "echo first-app01-ok" } echo 'after first withCredentials block' sh 'sleep 30' } withCredentials([sshUserPrivateKey(credentialsId: 'shared-dev-key', keyFileVariable: 'identityFileName', passphraseVariable: 'password', usernameVariable: 'username')]) { script { fileName = identityFileName def data = readFile(file: fileName) println data // <--- This print the secret ssh key again with no issues remote.identityFile = identityFileName //sshCommand remote: remote, command: "echo this-fails" } echo 'after second withCredentials block' sh 'sleep 30' } } } stage('APP_01_AGAIN') { //this is never run, but if you comment the second withCredentials above it will run an fail steps { withCredentials([sshUserPrivateKey(credentialsId: 'shared-dev-key', keyFileVariable: 'identityFileName', passphraseVariable: 'password', usernameVariable: 'username')]) { script { fileName = identityFileName def data = readFile(file: fileName) println data remote.identityFile = identityFileName sshCommand remote: remote, command: "echo this-one-fails-too" } } } } } }
Jenkins Error (Look, it's trying to find the 6e2a9038-339c-4988-a5a5-4468bacea4c8/ssh-key-identityFileName but it doesn't exists anymore):
16:49:37 [Pipeline] sshCommand 16:49:37 Executing command on integration_server[10.20.156.167]: echo this-one-fails-too sudo: false 16:49:37 [Pipeline] } 16:49:37 [Pipeline] // script 16:49:37 [Pipeline] } 16:49:37 [Pipeline] // withCredentials 16:49:37 [Pipeline] } 16:49:37 [Pipeline] // stage 16:49:37 [Pipeline] } 16:49:37 [Pipeline] // node 16:49:37 [Pipeline] End of Pipeline 16:49:37 java.io.FileNotFoundException: /var/lib/jenkins/workspace/test-bug-JENKINS-61341@tmp/secretFiles/6e2a9038-339c-4988-a5a5-4468bacea4c8/ssh-key-identityFileName (No such file or directory) 16:49:37 at java.base/java.io.FileInputStream.open0(Native Method) 16:49:37 at java.base/java.io.FileInputStream.open(FileInputStream.java:219) 16:49:37 at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157) 16:49:37 at java.base/java.io.FileInputStream.<init>(FileInputStream.java:112) 16:49:37 at com.jcraft.jsch.Util.fromFile(Util.java:508) 16:49:37 at com.jcraft.jsch.KeyPair.load(KeyPair.java:540) 16:49:37 Caused: com.jcraft.jsch.JSchException 16:49:37 at com.jcraft.jsch.KeyPair.load(KeyPair.java:543) 16:49:37 at com.jcraft.jsch.IdentityFile.newInstance(IdentityFile.java:40) 16:49:37 at com.jcraft.jsch.JSch.addIdentity(JSch.java:406) 16:49:37 at com.jcraft.jsch.JSch.addIdentity(JSch.java:387) 16:49:37 at jdk.internal.reflect.GeneratedMethodAccessor4870.invoke(Unknown Source) 16:49:37 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Checking the real files in the server, the files with the key exists but every time than a withCredentials block is used it deletes the files and recreates them, with another tmp name.
jenkins@ip-10-20-150-196:~/workspace/test-bug-JENKINS-61341@tmp$ cat secretFiles/6e2a9038-339c-4988-a5a5-4468bacea4c8/ssh-key-identityFileName -----BEGIN RSA PRIVATE KEY----- MY_SSH_KEY . . . -----END RSA PRIVATE KEY----- jenkins@ip-10-20-150-196:~/workspace/test-bug-JENKINS-61341@tmp$ cat secretFiles/4b169093-f5a3-4316-b7e2-e940149b361f/ssh-key-identityFileName -----BEGIN RSA PRIVATE KEY----- MY_SSH_KEY . . . -----END RSA PRIVATE KEY-----
Can you take a look nrayapati?
That looks like a good find julian_alarcon, please let me know if I can help in anything. I am using a scripted pipeline.
Hi, as I found the reason (not changing the value of the map value remote.identityFile), I was able to repeat the execution using two workarounds, you can find my code here:
server_01 = [:] server_01.name = "integration_server" server_01.host = "10.20.11.1" def remotename = "integration_server" def remotehost = "10.20.11.1" pipeline { agent any stages { stage('APP_01') { steps { withCredentials([sshUserPrivateKey(credentialsId: 'shared-dev-key', keyFileVariable: 'identityFileName', passphraseVariable: 'password', usernameVariable: 'username')]) { script { def remote = [ name: server_01.name, host: server_01.host, user: username, identityFile: identityFileName, allowAnyHosts: true] sshCommand remote: remote, command: "echo First workaround setting a new map for the withCredentials step It is possible to repeat the same map name remote as it is defined only by script block" } } withCredentials([sshUserPrivateKey(credentialsId: 'shared-dev-key', keyFileVariable: 'identityFileName', passphraseVariable: 'password', usernameVariable: 'username')]) { script { sshCommand remote: [ name: remotename, host: remotehost, allowAnyHosts: true, user: username, identityFile: identityFileName ], command: "echo works too, but seems ugly, as it is too large" } } withCredentials([sshUserPrivateKey(credentialsId: 'shared-dev-key', keyFileVariable: 'identityFileName', passphraseVariable: 'password', usernameVariable: 'username')]) { script { def remote = [ name: server_01.name, host: server_01.host, user: username, identityFile: identityFileName, allowAnyHosts: true] sshCommand remote: remote, command: "echo First workaround. It keeps working" } } withCredentials([sshUserPrivateKey(credentialsId: 'shared-dev-key', keyFileVariable: 'identityFileName', passphraseVariable: 'password', usernameVariable: 'username')]) { script { sshCommand remote: [ name: remotename, host: remotehost, allowAnyHosts: true, user: username, identityFile: identityFileName ], command: "echo Second workaround. It keeps working too" } } } } } }
Job Log:
Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Running on Jenkins in /var/lib/jenkins/workspace/test-bug-JENKINS-61341 [Pipeline] { [Pipeline] stage [Pipeline] { (APP_01) [Pipeline] withCredentials Masking supported pattern matches of $identityFileName or $password or $username [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] sshCommand Executing command on integration_server[10.20.11.1]: echo First workaround setting a new map for the withCredentials step It is possible to repeat the same map name remote as it is defined only by script block sudo: false First workaround setting a new map for the withCredentials step It is possible to repeat the same map name remote as it is defined only by script block [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // withCredentials [Pipeline] withCredentials Masking supported pattern matches of $identityFileName or $password or $username [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] sshCommand Executing command on integration_server[10.20.156.167]: echo works too, but seems ugly, as it is too large sudo: false works too, but seems ugly, as it is too large [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // withCredentials [Pipeline] withCredentials Masking supported pattern matches of $identityFileName or $password or $username [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] sshCommand Executing command on integration_server[10.20.11.1]: echo First workaround. It keeps working sudo: false First workaround. It keeps working [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // withCredentials [Pipeline] withCredentials Masking supported pattern matches of $identityFileName or $password or $username [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] sshCommand Executing command on integration_server[10.20.156.167]: echo Second workaround. It keeps working too sudo: false Second workaround. It keeps working too [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS
I updated my Jenkins to 2.330, same error. I'm out of ideas.
This is my list of installed plugins: