Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-32624

sshagent{} ignored when executed in docker.image().inside{...}

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major
    • None
    • CloudBees Docker Workflow 1.2
      Jenkins ver. 1.625.3.1 (CloudBees Jenkins Enterprise 15.11)

      sshagent('my-ssh-key'){...} is ignored in docker.image('cloudbees:java-build-tools').inside{...}. The same sshagent step works in 'standard' linux node.

      See attached logs

      sshagent(){...} ignored in docker.image(...).inside{...} 
      docker.image ('cloudbees/java-build-tools:0.0.7.1').inside {
          sh 'ls -al ~/.ssh/ || true'
          sshagent(['aws-cleclerc-ssh-key-ubuntu-cleclerc']) {
             sh 'echo SSH_AUTH_SOCK=$SSH_AUTH_SOCK'
             sh 'ls -al $SSH_AUTH_SOCK || true'
             sh "ssh -vvv -o StrictHostKeyChecking=no ubuntu@docker-agent.beesshop.org uname -a"
          }
      }
      
      sshagent(){...} WORKS in node{...} 
      node {
          sh 'ls -al ~/.ssh/ || true'
          sshagent(['aws-cleclerc-ssh-key-ubuntu-cleclerc']) {
             sh 'echo SSH_AUTH_SOCK=$SSH_AUTH_SOCK'
             sh 'ls -al $SSH_AUTH_SOCK || true'
             sh "ssh -vvv -o StrictHostKeyChecking=no ubuntu@docker-agent.beesshop.org uname -a"
          }
      }
      

          [JENKINS-32624] sshagent{} ignored when executed in docker.image().inside{...}

          Code changed in jenkins
          User: Jesse Glick
          Path:
          src/main/java/com/cloudbees/jenkins/plugins/sshagent/RemoteAgentFactory.java
          src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBuildWrapper.java
          src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentStepExecution.java
          src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/AgentServer.java
          src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgent.java
          src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgentFactory.java
          src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgentStarter.java
          src/main/java/com/cloudbees/jenkins/plugins/sshagent/mina/MinaRemoteAgentFactory.java
          http://jenkins-ci.org/commit/ssh-agent-plugin/d2e0f53b6eda56012c2bd5d9bc553b31667462a2
          Log:
          JENKINS-32624 JENKINS-27152 Use a standardized temporary directory when possible.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/com/cloudbees/jenkins/plugins/sshagent/RemoteAgentFactory.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBuildWrapper.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentStepExecution.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/AgentServer.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgent.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgentFactory.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgentStarter.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/mina/MinaRemoteAgentFactory.java http://jenkins-ci.org/commit/ssh-agent-plugin/d2e0f53b6eda56012c2bd5d9bc553b31667462a2 Log: JENKINS-32624 JENKINS-27152 Use a standardized temporary directory when possible.

          Cyrille Le Clerc added a comment - - edited

          Successfully tested.

          Pipeline Script

          node {
              docker.image ('cloudbees/java-build-tools:0.0.7.1').inside {
                  sh 'whoami'
                  sh 'ls -al ~/.ssh/ || true'
                  sshagent(['aws-cleclerc-ssh-key-ubuntu-cleclerc']) {
                     sh 'echo SSH_AUTH_SOCK=$SSH_AUTH_SOCK'
                     sh 'ls -al $SSH_AUTH_SOCK || true'
                     sh "ssh -o StrictHostKeyChecking=no ubuntu@jenkins-agent.beesshop.org uname -a"
                  }
              }
          }
          

          build logs

          Started by user admin
          [Pipeline] Allocate node : Start
          Running on jenkins-agent.example.com in /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent
          [Pipeline] node {
          [Pipeline] sh
          [test-ssh-agent] Running shell script
          + docker inspect -f . cloudbees/java-build-tools:0.0.7.1
          .
          [Pipeline] Run build steps inside a Docker container : Start
          $ docker run -t -d -u 1000:1000 -w /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent -v /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent:/home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent:rw -v /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp:/home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** cloudbees/java-build-tools:0.0.7.1 cat
          [Pipeline] withDockerContainer {
          [Pipeline] sh
          [test-ssh-agent] Running shell script
          + whoami
          jenkins
          [Pipeline] sh
          [test-ssh-agent] Running shell script
          + ls -al /home/jenkins/.ssh/
          ls: cannot access /home/jenkins/.ssh/: No such file or directory
          + true
          [Pipeline] SSH Agent : Start
          [ssh-agent] Using credentials ubuntu (cleclerc-us-east.pem)
          [ssh-agent] Looking for ssh-agent implementation...
          [ssh-agent]   Java/JNR ssh-agent
          [ssh-agent] Started.
          [Pipeline] sshagent {
          [Pipeline] sh
          [test-ssh-agent] Running shell script
          + echo SSH_AUTH_SOCK=/home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp/ssh330183201329153851
          SSH_AUTH_SOCK=/home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp/ssh330183201329153851
          [Pipeline] sh
          [test-ssh-agent] Running shell script
          + ls -al /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp/ssh330183201329153851
          srw------- 1 jenkins jenkins 0 Mar  4 10:03 /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp/ssh330183201329153851
          [Pipeline] sh
          [test-ssh-agent] Running shell script
          + ssh -o StrictHostKeyChecking=no ubuntu@jenkins-agent.example.com uname -a
          Warning: Permanently added 'jenkins-agent.example.com,52.90.106.69' (ECDSA) to the list of known hosts.
          Linux ip-10-0-0-151 3.13.0-76-generic #120-Ubuntu SMP Mon Jan 18 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          [Pipeline] } //sshagent
          [Pipeline] SSH Agent : End
          [Pipeline] } //withDockerContainer
          $ docker stop e24c314f6d82ac7d15033a996a56bf33e25096b4b47a0138f1c205559c4adc32
          $ docker rm -f e24c314f6d82ac7d15033a996a56bf33e25096b4b47a0138f1c205559c4adc32
          [Pipeline] Run build steps inside a Docker container : End
          [Pipeline] } //node
          [Pipeline] Allocate node : End
          [Pipeline] End of Pipeline
          Finished: SUCCESS
          

          Cyrille Le Clerc added a comment - - edited Successfully tested. Pipeline Script node { docker.image ( 'cloudbees/java-build-tools:0.0.7.1' ).inside { sh 'whoami' sh 'ls -al ~/.ssh/ || true ' sshagent([ 'aws-cleclerc-ssh-key-ubuntu-cleclerc' ]) { sh 'echo SSH_AUTH_SOCK=$SSH_AUTH_SOCK' sh 'ls -al $SSH_AUTH_SOCK || true ' sh "ssh -o StrictHostKeyChecking=no ubuntu@jenkins-agent.beesshop.org uname -a" } } } build logs Started by user admin [Pipeline] Allocate node : Start Running on jenkins-agent.example.com in /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent [Pipeline] node { [Pipeline] sh [test-ssh-agent] Running shell script + docker inspect -f . cloudbees/java-build-tools:0.0.7.1 . [Pipeline] Run build steps inside a Docker container : Start $ docker run -t -d -u 1000:1000 -w /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent -v /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent:/home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent:rw -v /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp:/home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** cloudbees/java-build-tools:0.0.7.1 cat [Pipeline] withDockerContainer { [Pipeline] sh [test-ssh-agent] Running shell script + whoami jenkins [Pipeline] sh [test-ssh-agent] Running shell script + ls -al /home/jenkins/.ssh/ ls: cannot access /home/jenkins/.ssh/: No such file or directory + true [Pipeline] SSH Agent : Start [ssh-agent] Using credentials ubuntu (cleclerc-us-east.pem) [ssh-agent] Looking for ssh-agent implementation... [ssh-agent] Java/JNR ssh-agent [ssh-agent] Started. [Pipeline] sshagent { [Pipeline] sh [test-ssh-agent] Running shell script + echo SSH_AUTH_SOCK=/home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp/ssh330183201329153851 SSH_AUTH_SOCK=/home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp/ssh330183201329153851 [Pipeline] sh [test-ssh-agent] Running shell script + ls -al /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp/ssh330183201329153851 srw------- 1 jenkins jenkins 0 Mar 4 10:03 /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp/ssh330183201329153851 [Pipeline] sh [test-ssh-agent] Running shell script + ssh -o StrictHostKeyChecking=no ubuntu@jenkins-agent.example.com uname -a Warning: Permanently added 'jenkins-agent.example.com,52.90.106.69' (ECDSA) to the list of known hosts. Linux ip-10-0-0-151 3.13.0-76-generic #120-Ubuntu SMP Mon Jan 18 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux [Pipeline] } //sshagent [Pipeline] SSH Agent : End [Pipeline] } //withDockerContainer $ docker stop e24c314f6d82ac7d15033a996a56bf33e25096b4b47a0138f1c205559c4adc32 $ docker rm -f e24c314f6d82ac7d15033a996a56bf33e25096b4b47a0138f1c205559c4adc32 [Pipeline] Run build steps inside a Docker container : End [Pipeline] } //node [Pipeline] Allocate node : End [Pipeline] End of Pipeline Finished: SUCCESS

          Stan Domula added a comment -

          Still not fully fixed.

          If the socket name gets longer than 108 chars it tries to create the socket in /tmp that doesn't work inside docker. So this is still an issue when using GitHub Organistaion Plugin. There it happens that paths are getting longer.

          Stan Domula added a comment - Still not fully fixed. If the socket name gets longer than 108 chars it tries to create the socket in /tmp that doesn't work inside docker. So this is still an issue when using GitHub Organistaion Plugin. There it happens that paths are getting longer.

          If one uses the standard location of `/var/lib/jenkins` as the remote root on the Docker daemon agent, for a multi-branch pipeline job the shortest possible path that minimize will generate is too long:

          echo "/var/lib/jenkins/workspace/a-R6R3BLSUWMBHF2ZTDGNA6MR4UMMWUNRCZRZ2356BQKJBSSLTLWYA@tmp/ssh9122165901071866621" | wc -c
          109
          

          Joshua Spiewak added a comment - If one uses the standard location of `/var/lib/jenkins` as the remote root on the Docker daemon agent, for a multi-branch pipeline job the shortest possible path that minimize will generate is too long: echo "/ var /lib/jenkins/workspace/a-R6R3BLSUWMBHF2ZTDGNA6MR4UMMWUNRCZRZ2356BQKJBSSLTLWYA@tmp/ssh9122165901071866621" | wc -c 109

          Jesse Glick added a comment -

          There is no “standard location” for Docker-based agent; depends on the image you use. So you need to use something shorter. The path name length limitation for sshagent sockets is an unfortunate artifact of a weird default in typical Linux kernels. I do not know of a satisfactory fix.

          Jesse Glick added a comment - There is no “standard location” for Docker-based agent; depends on the image you use. So you need to use something shorter. The path name length limitation for sshagent sockets is an unfortunate artifact of a weird default in typical Linux kernels. I do not know of a satisfactory fix.

          Sure, we don't have to call it a "standard location" for agents, but given that the Jenkins package on, say, Ubuntu, installs to /var/lib/jenkins, maintaining a similar location on the agents would seem to be a sensible way to keep systems consistent, and far more obvious than setting it to /j.

          Perhaps while this is being iterated upon there could be a feature toggle that could allow folks to fallback to the previous behavior if they need to?

          Joshua Spiewak added a comment - Sure, we don't have to call it a "standard location" for agents, but given that the Jenkins package on, say, Ubuntu, installs to /var/lib/jenkins, maintaining a similar location on the agents would seem to be a sensible way to keep systems consistent, and far more obvious than setting it to /j. Perhaps while this is being iterated upon there could be a feature toggle that could allow folks to fallback to the previous behavior if they need to?

          Jesse Glick added a comment -

          Which previous behavior are you referring to?

          Jesse Glick added a comment - Which previous behavior are you referring to?

          Stanislav Vetlovskiy added a comment - - edited

          I faced with the same issue, all was good on simple pipeline, but when i switched to multi-branch pipeline job, tasks in sshagent fall with error

          SSH_AUTH_SOCK=/tmp/ssh3569037635123687207
          + ls -al /tmp/ssh3569037635123687207
          ls: cannot access /tmp/ssh3569037635123687207: No such file or directory
          

          is there any advice to fix it or some workaround?

          Stanislav Vetlovskiy added a comment - - edited I faced with the same issue, all was good on simple pipeline, but when i switched to multi-branch pipeline job, tasks in sshagent fall with error SSH_AUTH_SOCK=/tmp/ssh3569037635123687207 + ls -al /tmp/ssh3569037635123687207 ls: cannot access /tmp/ssh3569037635123687207: No such file or directory is there any advice to fix it or some workaround?

          I am referring to the previous behavior that stored the workspace in sub-folders without the base32 of the hash that was introduced for JENKINS-34564.

          Joshua Spiewak added a comment - I am referring to the previous behavior that stored the workspace in sub-folders without the base32 of the hash that was introduced for JENKINS-34564 .

          Jesse Glick added a comment -

          The previous naming pattern for branch projects can be restored with a system property, see release notes.

          Jesse Glick added a comment - The previous naming pattern for branch projects can be restored with a system property, see release notes.

            jglick Jesse Glick
            cleclerc Cyrille Le Clerc
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: