Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-36997

sshAgent {} inside docker.image().inside {} does not work with long project name

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • ssh-agent-plugin
    • None
    • Jenkins: 2.7.1 LTS
      docker-workflow-plugin: 1.7
      ssh-agent-plugin: 1.13

      If the socket can not be created in the tmp directory of the job, which happens if the socket path would get longer than 108, it will create the socket directly under /tmp. If I am using docker.image().inside this does not work. If I am using node {} it does work as expected.

      When using the GitHub Organisation Plugin I can not avoid long job names.

          [JENKINS-36997] sshAgent {} inside docker.image().inside {} does not work with long project name

          Jesse Glick added a comment -

          Yes the SSH Agent plugin falls back to the system temporary directory when the path is going to be longer than 108 characters, the maximum typically supported by Linux kernels. Unfortunately this breaks the assumption of Docker Pipeline that the container and (agent) host share filesystems only in the workspace and its associated temporary directory.

          Might work to use ws with a (shortish) absolute path inside node but outside inside. Or use withCredentials to access the SSH private key explicitly rather than using sshAgent.

          Jesse Glick added a comment - Yes the SSH Agent plugin falls back to the system temporary directory when the path is going to be longer than 108 characters, the maximum typically supported by Linux kernels. Unfortunately this breaks the assumption of Docker Pipeline that the container and (agent) host share filesystems only in the workspace and its associated temporary directory. Might work to use ws with a (shortish) absolute path inside node but outside inside . Or use withCredentials to access the SSH private key explicitly rather than using sshAgent .

          Shane McDonald added a comment - - edited

          I ran into this as well. I got around the issue by checking "Use custom child workspace" in my job config, and setting "Child Directory" to ${SHORT_COMBINATION}

          Shane McDonald added a comment - - edited I ran into this as well. I got around the issue by checking "Use custom child workspace" in my job config, and setting "Child Directory" to ${SHORT_COMBINATION}

          Jesse Glick added a comment -

          Avoid that workaround; use ws instead.

          Jesse Glick added a comment - Avoid that workaround; use ws instead.

          Interesting... Couple of questions:

          • Why should I avoid this?
          • What is ws?

          Shane McDonald added a comment - Interesting... Couple of questions: Why should I avoid this? What is ws ?

          Jesse Glick added a comment -

          Look at the documentation.

          Jesse Glick added a comment - Look at the documentation.

          Thanks for being so helpful.

          Shane McDonald added a comment - Thanks for being so helpful.

          Jesse Glick added a comment -

          Would perhaps be bypassed by PR 2, TBD.

          Jesse Glick added a comment - Would perhaps be bypassed by PR 2, TBD.

          mor lajb added a comment -

          HI , I have the same problem - I am using jenkins version 2.32.1
          Docker Pipeline - 1.9.1
          Pipeline - 2.4
          Pipeline: Basic Steps -2.3
          Pipeline: Job - 2.9
          SSH Agent Plugin 1.13

          I couldn't workaround it , any idea ?

          here is my pipeline

          _node ('ubuntu-aws'){
          timestamps
          {
          docker.image('node').inside {
          sshagent(['XXX'])

          { stage "git checkout" checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [], gitTool: 'Default', submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'YYY', url: 'git@xxx:yyy.git']]]) stage "npm install" sh "npm install" }

          } // ssh agent
          }
          }

          _

          mor lajb added a comment - HI , I have the same problem - I am using jenkins version 2.32.1 Docker Pipeline - 1.9.1 Pipeline - 2.4 Pipeline: Basic Steps -2.3 Pipeline: Job - 2.9 SSH Agent Plugin 1.13 I couldn't workaround it , any idea ? here is my pipeline _node ('ubuntu-aws'){ timestamps { docker.image('node').inside { sshagent( ['XXX'] ) { stage "git checkout" checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [], gitTool: 'Default', submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'YYY', url: 'git@xxx:yyy.git']]]) stage "npm install" sh "npm install" } } // ssh agent } } _

          Jesse Glick added a comment -

          Again: only known workaround is to use the ws Pipeline step with a short absolute path, taking care to avoid collisions with other builds somehow (not generally easy). Proposed alternate plugin implementation would probably solve this issue; needs review and testing.

          Jesse Glick added a comment - Again: only known workaround is to use the ws Pipeline step with a short absolute path, taking care to avoid collisions with other builds somehow (not generally easy). Proposed alternate plugin implementation would probably solve this issue; needs review and testing.

          Jesse Glick added a comment -

          PR 17 is the new version.

          Jesse Glick added a comment - PR 17 is the new version.

          Jesse Glick added a comment -

          For those using multibranch projects, read release notes. Since the default workspace directory name will be up to 80 characters, and you must stay within 108 for sshagent to work, that means you must limit your “remote FS root” to well under 28 characters.

          Jesse Glick added a comment - For those using multibranch projects, read release notes . Since the default workspace directory name will be up to 80 characters, and you must stay within 108 for sshagent to work, that means you must limit your “remote FS root” to well under 28 characters.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          src/test/java/plugins/WorkflowPluginTest.java
          http://jenkins-ci.org/commit/acceptance-test-harness/7c815e3bcdb83c68dbfa1d8778ce6465fe7b3517
          Log:
          JENKINS-36997 Noting that weird script is a workaround for a PATH_MAX problem.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/test/java/plugins/WorkflowPluginTest.java http://jenkins-ci.org/commit/acceptance-test-harness/7c815e3bcdb83c68dbfa1d8778ce6465fe7b3517 Log: JENKINS-36997 Noting that weird script is a workaround for a PATH_MAX problem.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          pom.xml
          src/main/java/com/cloudbees/jenkins/plugins/sshagent/RemoteAgent.java
          src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBuildWrapper.java
          src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentStepExecution.java
          src/main/java/com/cloudbees/jenkins/plugins/sshagent/exec/ExecRemoteAgent.java
          src/main/java/com/cloudbees/jenkins/plugins/sshagent/exec/ExecRemoteAgentFactory.java
          src/test/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBase.java
          src/test/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBuildWrapperTest.java
          src/test/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentStepWorkflowTest.java
          http://jenkins-ci.org/commit/ssh-agent-plugin/8d02c6ca20ce514e8737a15b24bbfa7557930273
          Log:
          Merge pull request #18 from jglick/ExecRemoteAgent-JENKINS-36997

          JENKINS-36997 CLI implementation of RemoteAgent

          Compare: https://github.com/jenkinsci/ssh-agent-plugin/compare/f38bb5f22fea...8d02c6ca20ce

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/com/cloudbees/jenkins/plugins/sshagent/RemoteAgent.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBuildWrapper.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentStepExecution.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/exec/ExecRemoteAgent.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/exec/ExecRemoteAgentFactory.java src/test/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBase.java src/test/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBuildWrapperTest.java src/test/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentStepWorkflowTest.java http://jenkins-ci.org/commit/ssh-agent-plugin/8d02c6ca20ce514e8737a15b24bbfa7557930273 Log: Merge pull request #18 from jglick/ExecRemoteAgent- JENKINS-36997 JENKINS-36997 CLI implementation of RemoteAgent Compare: https://github.com/jenkinsci/ssh-agent-plugin/compare/f38bb5f22fea...8d02c6ca20ce

          Mike Kobit added a comment -

          We tried this out with:

          sshagent(['CredId']) {
            docker.inside('image') {
              // git operations
            }
          }

          and it did not work.

           

          Using the following did work:

          docker.inside('image') {
            sshagent(['CredId']) {
              // git operations
            }
          }

          That makes sense to me based on how the execution works now with using a CLI implementation.

          Thanks for fixing this!

          Mike Kobit added a comment - We tried this out with: sshagent([ 'CredId' ]) { docker.inside( 'image' ) { // git operations } } and it did not work.   Using the following did work: docker.inside( 'image' ) { sshagent([ 'CredId' ]) { // git operations } } That makes sense to me based on how the execution works now with using a CLI implementation. Thanks for fixing this!

          Jesse Glick added a comment -

          Right, the ssh-agent needs to be run inside the container so its socket is in the same kernel namespace as the commands which try to access it.

          Jesse Glick added a comment - Right, the ssh-agent needs to be run inside the container so its socket is in the same kernel namespace as the commands which try to access it.

          Hermann Schweizer added a comment - - edited

          Why is this marked as resolved? I use a multibranch pipeline and still suffer from this issue.
          I have a master slave setup. Which workspaces now need to be shorter than 108? master or slave or both?
          Simply doing ws("test") before the image.inside didn't help although the resulting ws path is really short.

          Outside of the container git cloning with inside sshagent(..){ works. Inside container it doesn't and the log says:
          docker exec 9859e94c20b84efd81e1752417c1f5144fa198ba76b14e1670f8993512af7d60 ssh-agent
          SSH_AUTH_SOCK=/tmp/ssh-GoP5qbl4iakN/agent.13
          SSH_AGENT_PID=20

          ```
          $ docker exec --env SSH_AGENT_PID=20 --env SSH_AUTH_SOCK=/tmp/ssh-GoP5qbl4iakN/agent.13 9859e94c20b84efd81e1752417c1f5144fa198ba76b14e1670f8993512af7d60 ssh-add /home/INT/jenkins/short@tmp/private_key_3688909095782238252.key
          Identity added: /home/INT/jenkins/short@tmp/private_key_3688909095782238252.key (/home/INT/jenkins/short@tmp/private_key_3688909095782238252.key)

          ```

          Hermann Schweizer added a comment - - edited Why is this marked as resolved? I use a multibranch pipeline and still suffer from this issue. I have a master slave setup. Which workspaces now need to be shorter than 108? master or slave or both? Simply doing ws("test") before the image.inside didn't help although the resulting ws path is really short. Outside of the container git cloning with inside sshagent(..){ works. Inside container it doesn't and the log says: docker exec 9859e94c20b84efd81e1752417c1f5144fa198ba76b14e1670f8993512af7d60 ssh-agent SSH_AUTH_SOCK=/tmp/ssh-GoP5qbl4iakN/agent.13 SSH_AGENT_PID=20 ``` $ docker exec --env SSH_AGENT_PID=20 --env SSH_AUTH_SOCK=/tmp/ssh-GoP5qbl4iakN/agent.13 9859e94c20b84efd81e1752417c1f5144fa198ba76b14e1670f8993512af7d60 ssh-add /home/INT/jenkins/short@tmp/private_key_3688909095782238252.key Identity added: /home/INT/jenkins/short@tmp/private_key_3688909095782238252.key (/home/INT/jenkins/short@tmp/private_key_3688909095782238252.key) ```

          Denys Digtiar added a comment - - edited

          hermain As Mike and Jesse alluded to, the new Agent implementation was added which is based on the CLI ssh-agent. If you have a CLI available inside the docker container and use the `sshagent` inside the docker.inside() closure, your git clone should work.

          Look for the message like "Exec ssh-agent (binary ssh-agent on a remote machine)" or any errors that mention ssh-agent

          Denys Digtiar added a comment - - edited hermain As Mike and Jesse alluded to, the new Agent implementation was added which is based on the CLI ssh-agent . If you have a CLI available inside the docker container and use the `sshagent` inside the docker.inside() closure, your git clone should work. Look for the message like "Exec ssh-agent (binary ssh-agent on a remote machine)" or any errors that mention ssh-agent

          devopsfido I managed to resolve the issue but this page was offline at the time so I forgot to mention it here:

          The problem in my case was that my git was not a known host inside the container:

          image.inside("-u root:root --network=host") {
                      sshagent(credentials: [config.gitKeyCredentialsId] ) {
                          sh "mkdir ~/.ssh"
                          sh "ssh-keyscan git.myCompany.com >> ~/.ssh/known_hosts"
                         // call scripts that do git clone...
                      }
                  }

           

          I hope that helps you and anyone else with a similar problem.

          Hermann Schweizer added a comment - devopsfido I managed to resolve the issue but this page was offline at the time so I forgot to mention it here: The problem in my case was that my git was not a known host inside the container: image.inside("-u root:root --network=host") {             sshagent(credentials: [config.gitKeyCredentialsId] ) {                 sh "mkdir ~/.ssh"                 sh "ssh-keyscan git.myCompany.com >> ~/.ssh/known_hosts"                // call scripts that do git clone...             }         }   I hope that helps you and anyone else with a similar problem.

          Evgeny Shepelyuk added a comment - - edited

          Hello, I'm using latest 1.17 version of SSH Agent Plugin and still expirience the same issue.
          Although docker image has ssh-agent cmd line command inside - the socket is created under {{/tmp} so it's inaccesisble

          Evgeny Shepelyuk added a comment - - edited Hello, I'm using latest 1.17 version of SSH Agent Plugin and still expirience the same issue. Although docker image has ssh-agent cmd line command inside - the socket is created under {{/tmp} so it's inaccesisble

          Jesse Glick added a comment -

          eshepelyuk maybe ssh-agent is not in $PATH or something.

          Jesse Glick added a comment - eshepelyuk maybe ssh-agent is not in $PATH or something.

          hermain Thank you!  This was exactly our issue.

           

          Works with SSH Agent Plugin v1.17 for us, using declarative pipeline and just a regular agent block:

          pipeline {
            agent { docker {
              image 'my_build_image'
              ...
            } }
            ...
            stage("foo") {
              sshagent(credentials: [config.gitKeyCredentialsId] ) {
                sh "mkdir ~/.ssh && ssh-keyscan git.myCompany.com >> ~/.ssh/known_hosts"
                // call scripts that do git clone...
              }
            }
          }
          
          

          Matt C. Wilson added a comment - hermain  Thank you!  This was exactly our issue.   Works with SSH Agent Plugin v1.17 for us, using declarative pipeline and just a regular agent block: pipeline { agent { docker { image 'my_build_image' ... } } ... stage( "foo" ) { sshagent(credentials: [config.gitKeyCredentialsId] ) {       sh "mkdir ~/.ssh && ssh-keyscan git.myCompany.com >> ~/.ssh/known_hosts"        // call scripts that do git clone...     } } }

            jglick Jesse Glick
            sdomula Stan Domula
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: