Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37069

Permission denied on durable task directory when using docker.image.inside step on fresh install of jenkins

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • docker-workflow-plugin

      [Maybe related to issues JENKINS-28821, JENKINS-33632 and JENKINS-36842]

      Hello,

      I am trying to stablish a new CI environment with jobs running on docker but I am running into permission issues. I tried to create a minimal reproducible scenario without slave machines.

      Looking at some of the related issues, I wonder whether I was supposed to configure permissions or group membership for the jenkins user on the container (instead of just using the image as is), but I assume that is not the case.

      Starting on a clean Centos 7 vm, I installed jenkins 2.7.1 and docker, and then added the jenkins user to the docker group (ansible playbook follows). Then I only installed "Pipeline" and "CloudBees Docker Pipeline" plugins and its dependencies. Everything is updated as of today. Then I created a single pipeline job:

      node {
         docker.image('centos:7').inside {
            sh 'pwd'
         }
      }
      

      This job fails with permission issues:

      Started by user admin
      [Pipeline] node
      Running on master in /var/lib/jenkins/workspace/container-test
      [Pipeline] {
      [Pipeline] sh
      [container-test] Running shell script
      + docker inspect -f . centos:7
      .
      [Pipeline] withDockerContainer
      $ docker run -t -d -u 992:989 -w /var/lib/jenkins/workspace/container-test -v /var/lib/jenkins/workspace/container-test:/var/lib/jenkins/workspace/container-test:rw -v /var/lib/jenkins/workspace/container-test@tmp:/var/lib/jenkins/workspace/container-test@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** centos:7 cat
      [Pipeline] {
      [container-test] Running shell script
      [Pipeline] sh
      sh: /var/lib/jenkins/workspace/container-test@tmp/durable-890dccc6/pid: Permission denied
      sh: /var/lib/jenkins/workspace/container-test@tmp/durable-890dccc6/jenkins-log.txt: Permission denied
      sh: /var/lib/jenkins/workspace/container-test@tmp/durable-890dccc6/jenkins-result.txt: Permission denied
      [Pipeline] }
      $ docker stop c71c65555ff53a1bd87db33a9d240c6eb4ae14d9c61a0a0a348c7f72f82b7a50
      $ docker rm -f c71c65555ff53a1bd87db33a9d240c6eb4ae14d9c61a0a0a348c7f72f82b7a50
      [Pipeline] // withDockerContainer
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      ERROR: script returned exit code -2
      Finished: FAILURE
      

      Tools were installed using the following ansible recipe:

      ---
      - hosts: jenkins-minimal
      
        tasks:
        - yum: name={{ item }} state=installed
          with_items:
            - libselinux-python
            - dejavu-sans-fonts
            - fontconfig
            - java-1.8.0-openjdk-headless
            - docker
      
        - yum_repository:
            name: jenkins
            description: 'Jenkins-stable'
            baseurl: http://pkg.jenkins.io/redhat-stable
            gpgkey: http://pkg.jenkins.io/redhat-stable/jenkins.io.key
      
        - yum: name=jenkins state=installed
      
        - group: name=docker
        - user: name=jenkins groups=docker
      
        - firewalld: port=8080/tcp state=enabled permanent=true immediate=yes
      
        - service: name={{ item }} state=started enabled=yes
          with_items:
            - jenkins
            - docker
      

          [JENKINS-37069] Permission denied on durable task directory when using docker.image.inside step on fresh install of jenkins

          Just now I've ensured that /Users/mmccaskill/.jenkins has 777 permissions and still the same error.

          Michael McCaskill added a comment - Just now I've ensured that /Users/mmccaskill/.jenkins has 777 permissions and still the same error.

          Interestingly using fswatch reveals creating and deleting of other directories

          /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-803731eb/jenkins-log.txt
          /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-803731eb/jenkins-result.txt
          /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-803731eb/output.txt
          /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-803731eb/pid
          /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-803731eb/script.sh
          /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-803731eb
          /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-1839560a/jenkins-log.txt
          /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-1839560a/jenkins-result.txt
          /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-1839560a/pid
          /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-1839560a/script.sh
          /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-1839560a
          /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-c91c1312
          /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-c91c1312/script.sh
          /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-c91c1312/script.sh
          /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-c91c1312
          

          Michael McCaskill added a comment - Interestingly using fswatch reveals creating and deleting of other directories /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-803731eb/jenkins-log.txt /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-803731eb/jenkins-result.txt /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-803731eb/output.txt /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-803731eb/pid /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-803731eb/script.sh /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-803731eb /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-1839560a/jenkins-log.txt /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-1839560a/jenkins-result.txt /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-1839560a/pid /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-1839560a/script.sh /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-1839560a /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-c91c1312 /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-c91c1312/script.sh /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-c91c1312/script.sh /Users/mmccaskill/.jenkins/workspace/example_poc_master-GGCVXMNZOUBSWREPVP27YOR3DZTHKBWNU2ISCLF5PYNDINJX5K6Q@tmp/durable-c91c1312

          Jesse Glick added a comment -

          Generally this is an environmental issue rather than a product bug, though there may be ways the product could assist in automatic diagnosis.

          Jesse Glick added a comment - Generally this is an environmental issue rather than a product bug, though there may be ways the product could assist in automatic diagnosis.

          Craig Rodrigues added a comment - - edited

          I encountered exactly the same problem.

          I am using:

          • Docker 1.13.1
          • MacOS 10.12.3

          I am running Jenkins locally, under my own user ID, so I don't understand why I am getting a permission denied error.

          I used this:

          node {
              stage("Inside the container") {
                  docker.image('debian:stretch').inside {   
                     sh "whoami"
                     sh "echo Hi"
                     sh "dpkg -l"
                  }
              }
          }
          

          I looked at the output of my script and got this:

          [test3] Running shell script
          + docker inspect -f . debian:stretch
          .
          [Pipeline] withDockerContainer
          $ docker run -t -d -u 948935127:269013081 -w /Users/c-craigr/.jenkins/workspace/test3 -v /Users/c-craigr/.jenkins/workspace/test3:/Users/c-craigr/.jenkins/workspace/test3:rw -v /Users/c-craigr/.jenkins/workspace/test3@tmp:/Users/c-craigr/.jenkins/workspace/test3@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat debian:stretch
          [Pipeline] {
          [Pipeline] sh
          [test3] Running shell script
          sh: 1: cannot create /Users/c-craigr/.jenkins/workspace/test3@tmp/durable-56d7e69e/pid: Permission denied
          sh: 1: cannot create /Users/c-craigr/.jenkins/workspace/test3@tmp/durable-56d7e69e/jenkins-log.txt: Permission denied
          sh: 1: cannot create /Users/c-craigr/.jenkins/workspace/test3@tmp/durable-56d7e69e/jenkins-result.txt: Permission denied
          [Pipeline] }
          $ docker stop --time=1 186dc9f881b57c62d9ac59ff3533a259b70fd80670de2a0e6e1ff37a8129c90d
          $ docker rm -f 186dc9f881b57c62d9ac59ff3533a259b70fd80670de2a0e6e1ff37a8129c90d
          [Pipeline] // withDockerContainer
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] End of Pipeline
          ERROR: script returned exit code -2
          Finished: FAILURE
          

          Craig Rodrigues added a comment - - edited I encountered exactly the same problem. I am using: Docker 1.13.1 MacOS 10.12.3 I am running Jenkins locally, under my own user ID, so I don't understand why I am getting a permission denied error. I used this: node { stage("Inside the container") { docker.image('debian:stretch').inside { sh "whoami" sh "echo Hi" sh "dpkg -l" } } } I looked at the output of my script and got this: [test3] Running shell script + docker inspect -f . debian:stretch . [Pipeline] withDockerContainer $ docker run -t -d -u 948935127:269013081 -w /Users/c-craigr/.jenkins/workspace/test3 -v /Users/c-craigr/.jenkins/workspace/test3:/Users/c-craigr/.jenkins/workspace/test3:rw -v /Users/c-craigr/.jenkins/workspace/test3@tmp:/Users/c-craigr/.jenkins/workspace/test3@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat debian:stretch [Pipeline] { [Pipeline] sh [test3] Running shell script sh: 1: cannot create /Users/c-craigr/.jenkins/workspace/test3@tmp/durable-56d7e69e/pid: Permission denied sh: 1: cannot create /Users/c-craigr/.jenkins/workspace/test3@tmp/durable-56d7e69e/jenkins-log.txt: Permission denied sh: 1: cannot create /Users/c-craigr/.jenkins/workspace/test3@tmp/durable-56d7e69e/jenkins-result.txt: Permission denied [Pipeline] } $ docker stop --time=1 186dc9f881b57c62d9ac59ff3533a259b70fd80670de2a0e6e1ff37a8129c90d $ docker rm -f 186dc9f881b57c62d9ac59ff3533a259b70fd80670de2a0e6e1ff37a8129c90d [Pipeline] // withDockerContainer [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline ERROR: script returned exit code -2 Finished: FAILURE

          Jesse Glick added a comment -

          Using docker-workflow 1.9.1, durable-task 1.12, and workflow-durable-task-step 2.8 on Jenkins 2.32.2 on Linux / Java 8 / Docker 1.12.3, whoami fails (no such user) but the other commands succeed.

          Do not try to use Image.inside on a Jenkins master node running on a Mac. The tricks used by the nonnative Docker ports do not permit the container to share a filesystem with the Jenkins JVM. (You should be able to run a Jenkins agent in a container, if its filesystem root is a volume, and use Image.inside from that “remote” node.)

          Jesse Glick added a comment - Using docker-workflow 1.9.1, durable-task 1.12, and workflow-durable-task-step 2.8 on Jenkins 2.32.2 on Linux / Java 8 / Docker 1.12.3, whoami fails (no such user) but the other commands succeed. Do not try to use Image.inside on a Jenkins master node running on a Mac. The tricks used by the nonnative Docker ports do not permit the container to share a filesystem with the Jenkins JVM. (You should be able to run a Jenkins agent in a container, if its filesystem root is a volume, and use Image.inside from that “remote” node.)

          Craig Rodrigues added a comment - - edited

          jglick Ah! OK, that makes a lot of sense. I switched to an a completely Linux + Docker environment,
          and re-ran the pipeline and it worked.
          I don't know if it is possible, but it might be nice to put some error messages
          in the Docker plugin to indicate that this type of thing is not supported on platforms that are non-native Docker ports, like Mac.

          Craig Rodrigues added a comment - - edited jglick Ah! OK, that makes a lot of sense. I switched to an a completely Linux + Docker environment, and re-ran the pipeline and it worked. I don't know if it is possible, but it might be nice to put some error messages in the Docker plugin to indicate that this type of thing is not supported on platforms that are non-native Docker ports, like Mac.

          Jesse Glick added a comment -

          Well the documentation does mention this but if I can figure out a way for Jenkins to automatically detect this situation and report a nicer error I will do so.

          Jesse Glick added a comment - Well the documentation does mention this but if I can figure out a way for Jenkins to automatically detect this situation and report a nicer error I will do so.

          rodrigc - If it is helpful I was able to get it to work using the xhyve driver with experimental NFS share

          brew install docker-machine-driver-xhyve
          sudo chown root:wheel /usr/local/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
          sudo chmod u+s /usr/local/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
          docker-machine create -d xhyve --xhyve-experimental-nfs-share default2
          eval $(docker-machine env default2)
          jenkins
          

          Then it worked for me.

          Michael McCaskill added a comment - rodrigc - If it is helpful I was able to get it to work using the xhyve driver with experimental NFS share brew install docker-machine-driver-xhyve sudo chown root:wheel /usr/local/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve sudo chmod u+s /usr/local/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve docker-machine create -d xhyve --xhyve-experimental-nfs-share default2 eval $(docker-machine env default2) jenkins Then it worked for me.

          rodrigc - Another option I've used successfully recently was using vagrant.

          • vagrant init ubuntu/xenial64
          • vagrant up
          • install JDK and Docker
          • Copy appropriate SSH public key to /home/ubuntu/.ssh/authorized_keys
          • Setup this vagrant machine as a SSH Slave

          For my purposes I did mount my /Users -> /Users via the Vagrantfile and it works nicely. You may want to label the node as 'docker' and have the Jenkinsfile use that node.

          Michael McCaskill added a comment - rodrigc - Another option I've used successfully recently was using vagrant. vagrant init ubuntu/xenial64 vagrant up install JDK and Docker Copy appropriate SSH public key to /home/ubuntu/.ssh/authorized_keys Setup this vagrant machine as a SSH Slave For my purposes I did mount my /Users -> /Users via the Vagrantfile and it works nicely. You may want to label the node as 'docker' and have the Jenkinsfile use that node.

          rodrigc - Another option that works that's much easier to continue using the virtualbox driver and

          https://github.com/adlogix/docker-machine-nfs

          brew install docker-machine-nfs
          docker-machine-nfs <name of docker-machine>
          

          Michael McCaskill added a comment - rodrigc - Another option that works that's much easier to continue using the virtualbox driver and https://github.com/adlogix/docker-machine-nfs brew install docker-machine-nfs docker-machine-nfs <name of docker-machine>

            Unassigned Unassigned
            seuvitor Vitor Dantas
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: