Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-28821

Docker Workflow demos: sh step on Docker image hangs if workspace is not writable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Blocker Blocker
    • docker-workflow-plugin
    • remote docker server
      docker-workflow-1.0

      A sample demo workflow script hangs infinitely on the shell step.
      Logs are available below, stackdump is attached.

      Workflow:

      docker.image('maven:3.3.3-jdk-8').inside {
        git url: "https://github.com/oleg-nenashev/ircbot.git"
        sh 'echo hello'
        //Same thing with mvn -B clean install'
      }
      

      Build log

      Started by user anonymous
      Running: Allocate node : Start
      Running on master in /Users/nenashev/Documents/cloudbees/demo/docker-traceability/work/jobs/test-example-1/workspace
      Running: Allocate node : Body : Start
      Running: Shell Script
      [workspace] Running shell script
      + docker inspect -f . maven:3.3.3-jdk-8
      .
      Running: Run build steps inside a Docker container : Start
      $ docker run -t -d -u 502:20 -w /Users/nenashev/Documents/cloudbees/demo/docker-traceability/work/jobs/test-example-1/workspace -v /Users/nenashev/Documents/cloudbees/demo/docker-traceability/work/jobs/test-example-1/workspace:/Users/nenashev/Documents/cloudbees/demo/docker-traceability/work/jobs/test-example-1/workspace:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** maven:3.3.3-jdk-8 cat
      Running: Run build steps inside a Docker container : Body : Start
      Running: Git
       > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repository
       > git config remote.origin.url https://github.com/oleg-nenashev/ircbot.git # timeout=10
      Fetching upstream changes from https://github.com/oleg-nenashev/ircbot.git
       > git --version # timeout=10
       > git -c core.askpass=true fetch --tags --progress https://github.com/oleg-nenashev/ircbot.git +refs/heads/*:refs/remotes/origin/*
       > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
       > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
      Checking out Revision eaa8c349f12288276e5c27fcef78c245bb41517a (refs/remotes/origin/master)
       > git config core.sparsecheckout # timeout=10
       > git checkout -f eaa8c349f12288276e5c27fcef78c245bb41517a
       > git rev-list eaa8c349f12288276e5c27fcef78c245bb41517a # timeout=10
      Running: Shell Script
      [workspace] Running shell script
      <<< hangs here
      

          [JENKINS-28821] Docker Workflow demos: sh step on Docker image hangs if workspace is not writable

          Kris Musard added a comment - - edited

          gsymons How were you able to make the workspace writable for the shell command? Even with 777 permissions on the workspace I have the same issue. If I run the same docker command I see in the jenkins console log with "bash" instead of "cat" I can touch files in the workspace directory without a problem.

          UPDATE 11/16/2015:
          In my case I was able to get sh commands working inside the container by adding a user to the docker image with the uid/gid that was bieng specified on the docker run command executed by the plugin (the local jenkins userid and group).

          Kris Musard added a comment - - edited gsymons How were you able to make the workspace writable for the shell command? Even with 777 permissions on the workspace I have the same issue. If I run the same docker command I see in the jenkins console log with "bash" instead of "cat" I can touch files in the workspace directory without a problem. UPDATE 11/16/2015: In my case I was able to get sh commands working inside the container by adding a user to the docker image with the uid/gid that was bieng specified on the docker run command executed by the plugin (the local jenkins userid and group).

          In my case, I was running Jenkins in a container, and had mounted the host's Docker socket in the container. Since /var/jenkins_home was a data volume for the container, it didn't exist on the host, and so the workspace didn't exist either. I fixed it by running a second Docker process inside the container alongside Jenkins.

          Gregory Symons added a comment - In my case, I was running Jenkins in a container, and had mounted the host's Docker socket in the container. Since /var/jenkins_home was a data volume for the container, it didn't exist on the host, and so the workspace didn't exist either. I fixed it by running a second Docker process inside the container alongside Jenkins.

          Vlad Korolev added a comment -

          Not sure if mine is related to this issue. I experience the same symptoms when I try to run containers that have entry point defined. The thing tries to run 'cat' in the container before doing anything else. But the entry point program considers cat to be an argument and exits right away. Jenkins doesn't notice that and tries to continue with workflow and launch things on killed container which never runs and the whole thing hangs forever until it's manually killed.

          Here is relevant log:

          58d775c792e6: Pull complete
          9ba627a43b4f: Pull complete
          Digest: sha256:ea5101ed616f7a0bc59ac1bd7034c706e60b46d422d161b774f7b0b2ad547ed9
          Status: Downloaded newer image for sequenceiq/packer:v0.7.5.dev
          [Workflow] Run build steps inside a Docker container : Start
          $ docker run -t -d -u 500:500 -w /home/ec2-user/workspace/Boxes/boxes -v /home/ec2-user/workspace/Boxes/boxes:/home/ec2-user/workspace/Boxes/boxes:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** sequenceiq/packer:v0.7.5.dev cat
          [Workflow] withDockerContainer {
          [Workflow] sh
          [boxes] Running shell script
          Aborted by Vlad Korolev
          Aborted by Vlad Korolev
          Forcibly terminate running steps
          Terminating sh

          Vlad Korolev added a comment - Not sure if mine is related to this issue. I experience the same symptoms when I try to run containers that have entry point defined. The thing tries to run 'cat' in the container before doing anything else. But the entry point program considers cat to be an argument and exits right away. Jenkins doesn't notice that and tries to continue with workflow and launch things on killed container which never runs and the whole thing hangs forever until it's manually killed. Here is relevant log: 58d775c792e6: Pull complete 9ba627a43b4f: Pull complete Digest: sha256:ea5101ed616f7a0bc59ac1bd7034c706e60b46d422d161b774f7b0b2ad547ed9 Status: Downloaded newer image for sequenceiq/packer:v0.7.5.dev [Workflow] Run build steps inside a Docker container : Start $ docker run -t -d -u 500:500 -w /home/ec2-user/workspace/Boxes/boxes -v /home/ec2-user/workspace/Boxes/boxes:/home/ec2-user/workspace/Boxes/boxes:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** sequenceiq/packer:v0.7.5.dev cat [Workflow] withDockerContainer { [Workflow] sh [boxes] Running shell script Aborted by Vlad Korolev Aborted by Vlad Korolev Forcibly terminate running steps Terminating sh

          Jesse Glick added a comment - - edited

          vkorolev sounds like a distinct issue, file separately with steps to reproduce.

          Jesse Glick added a comment - - edited vkorolev sounds like a distinct issue, file separately with steps to reproduce.

          Kurt Madel added a comment -

          kmusard I have tried configuring the build node (with the mapped docker socket) to have a jenkins user with the same uid:gid as used by the plugin i(1000:1001) to no avail. What do you mean by:

          In my case I was able to get sh commands working inside the container by adding a user to the docker image with the uid/gid that was bieng specified on the docker run command executed by the plugin (the local jenkins userid and group).

          Kurt Madel added a comment - kmusard I have tried configuring the build node (with the mapped docker socket) to have a jenkins user with the same uid:gid as used by the plugin i(1000:1001) to no avail. What do you mean by: In my case I was able to get sh commands working inside the container by adding a user to the docker image with the uid/gid that was bieng specified on the docker run command executed by the plugin (the local jenkins userid and group).

          Kris Musard added a comment -

          kmadel In my setup I had a Docker server that was also running jenkins. Once I setup a jenkins user inside the container with matching uid:gid to the server running jenkins, the sh commands no longer hung.

          Kris Musard added a comment - kmadel In my setup I had a Docker server that was also running jenkins. Once I setup a jenkins user inside the container with matching uid:gid to the server running jenkins, the sh commands no longer hung.

          Oleg Nenashev added a comment -

          Stopped the progress since jglick didn't like https://github.com/jenkinsci/docker-workflow-plugin/pull/7 . I have not implemented anything else

          Oleg Nenashev added a comment - Stopped the progress since jglick didn't like https://github.com/jenkinsci/docker-workflow-plugin/pull/7 . I have not implemented anything else

          Keith Zantow added a comment -

          I was able to work around this without modifying the container by explicitly specifying the uid to match the mounted filesystem (uid: 1000, gid: 50).

          Like this:

          docker.image('maven:3.3.3-jdk-8').inside('-u 1000:50') {
            git '...'
            sh 'mvn -B clean install'
          }
          

          Less than ideal, as the uid is specified twice in the command line...

          Keith Zantow added a comment - I was able to work around this without modifying the container by explicitly specifying the uid to match the mounted filesystem (uid: 1000, gid: 50). Like this: docker.image( 'maven:3.3.3-jdk-8' ).inside( '-u 1000:50' ) { git '...' sh 'mvn -B clean install' } Less than ideal, as the uid is specified twice in the command line...

          Jesse Glick added a comment -

          Seems there are various things being mixed together here.

          First is the robustness issue: if the control directory cannot be created, we had better report that properly, and fail. JENKINS-28400 covers this.

          Then there is an issue reported with DinD, which is perhaps best avoided by using bind-mounts of the Docker socket; PR 31 purports to address this.

          Then there was a comment about containers with an entry point. If reproducible, clearly a distinct bug.

          Then there is some discussion of UID mismatches, which PR 25 might address, though there is no information given there about how to reproduce.

          Jesse Glick added a comment - Seems there are various things being mixed together here. First is the robustness issue: if the control directory cannot be created, we had better report that properly, and fail. JENKINS-28400 covers this. Then there is an issue reported with DinD, which is perhaps best avoided by using bind-mounts of the Docker socket; PR 31 purports to address this. Then there was a comment about containers with an entry point. If reproducible, clearly a distinct bug. Then there is some discussion of UID mismatches, which PR 25 might address, though there is no information given there about how to reproduce.

          Jesse Glick added a comment -

          Closing given the confused set of issues here. For any particular bug which causes the workspace to not be writable (or not mounted, etc.), please file a distinct report with steps to reproduce.

          Jesse Glick added a comment - Closing given the confused set of issues here. For any particular bug which causes the workspace to not be writable (or not mounted, etc.), please file a distinct report with steps to reproduce.

            Unassigned Unassigned
            oleg_nenashev Oleg Nenashev
            Votes:
            3 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: