Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-69504

rootless docker/podman workspace volume permissions

      Utilizing rootless podman (alias docker) with Jenkins Pipeline has workspace permission issues.

      A given agent "Remote root directory" is set to /home/jenkins/ssd  The idea is to provide a specific SSD for workspace speed purposes.

       

      jenkins@podman0 ~]$ ls -l /home/jenkins/ssd/
      total 1488
      drwx------. 3 jenkins jenkins      26 Aug 29 16:55 caches
      drwx------. 4 jenkins jenkins      34 Aug 29 15:57 remoting
      -rw-------. 1 jenkins jenkins 1522481 Aug 29 15:57 remoting.jar
      drwxr-xr-x. 2 jenkins jenkins       6 Aug 31 15:53 workspace
      

       

      Below is the log of the permission issues that occurs. 

      Changelog calculated successfully.
      Checkout done
      [Pipeline] withEnv 
      [Pipeline] { 
      [Pipeline] isUnix 
      [Pipeline] withEnv 
      [Pipeline] { 
      [Pipeline] sh
      + docker inspect -f . registry.internal.mycomp.com/rhel:72
      .
      [Pipeline] } 
      [Pipeline] // withEnv 
      [Pipeline] withDockerContainer
      podman0 does not seem to be running inside a container
      $ docker run -t -d -u 1000:1000 -w /home/jenkins/ssd/workspace/TEST_PIPELINE -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace/TEST_PIPELINE:rw,z -v /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** registry.internal.mycomp.com/rhel:72 cat 
      $ docker top 26dac8250b54021d18ad6006d3c42747eb2af447a993fff59621d0b6053f5dcb -eo pid,comm 
      [Pipeline] { 
      [Pipeline] sh 
      sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-log.txt: Permission denied 
      sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt.tmp: Permission denied 
      mv: failed to access '/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt': Permission denied

      It is understood that Jenkins Pipeline is meant to work with Docker, but I think some small changes could allow it to work with podman's backward compatibility (ln -s podman docker) to work. The root of the problem is permissions when a volume is mapped inside the container, I see the following:

       

      sh-4.4$ ls -l /home/jenkins/ssd
      total 0
      drwxr-xr-x. 3 root root 33 Sep  1 15:24 workspace
      

       

      The permissions have changed to root:root which obviously aren't allowed with jenkins:jenkins per the -u 1000:1000 as shown above.

      I'm aware that volumes can maintain permissions from the primary host (podman0) with something like -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace:U will allow or change the permissions within the container to 1000:1000, but it then causes the permissions outside of the container to change to the subuid:subgid value, which further confuses the Jenkins agent on successive jobs. This possibly is more of a feature request vs. bug, otherwise I'm hoping that someone has a workaround for this. 

      One possibility would be to detect that "podman" is actually being used and replace the 'u 1000:1000' with '-userns=keep-id'.  This appears to create the appropriate permissions within the container and the host as well.

          [JENKINS-69504] rootless docker/podman workspace volume permissions

          Shane Walton created issue -
          Shane Walton made changes -
          Description Original: Utilizing rootless podman (alias docker) with Jenkins Pipeline has workspace permission issues.

          A given agent "Remote root directory" is set to /home/jenkins/ssd  The idea is to provide a specific SSD for workspace speed purposes.

           
          {code:java}
          jenkins@podman0 ~]$ ls -l /home/jenkins/ssd/
          total 1488
          drwx------. 3 jenkins jenkins      26 Aug 29 16:55 caches
          drwx------. 4 jenkins jenkins      34 Aug 29 15:57 remoting
          -rw-------. 1 jenkins jenkins 1522481 Aug 29 15:57 remoting.jar
          drwxr-xr-x. 2 jenkins jenkins       6 Aug 31 15:53 workspace
          {code}
           

          Below is the log of the permission issues that occurs. 
          {code:java}
          Changelog calculated successfully.
          Checkout done
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] isUnix
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] sh
          + docker inspect -f . registry.internal.mycomp.com/rhel:72
          .
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] withDockerContainer
          podman0 does not seem to be running inside a container
          $ docker run -t -d -u 1000:1000 -w /home/jenkins/ssd/workspace/TEST_PIPELINE -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace/TEST_PIPELINE:rw,z -v /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** registry.internal.mycomp.com/rhel:72 cat
          $ docker top 26dac8250b54021d18ad6006d3c42747eb2af447a993fff59621d0b6053f5dcb -eo pid,comm
          [Pipeline] {
          [Pipeline] sh
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-log.txt: Permission denied
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt.tmp: Permission denied
          mv: failed to access '/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt': Permission denied{code}
          It is understood that Jenkins Pipeline is meant to work with Docker, but I think some small changes could allow it to work with podman's backward compatibility (ln -s podman docker) to work. The root of the problem is permissions when a volume is mapped inside the container, I see the following:

           
          {code:java}
          sh-4.4$ ls -l /home/jenkins/ssd
          total 0
          drwxr-xr-x. 3 root root 33 Sep  1 15:24 workspace
          {code}
           

          The permissions have changed to root:root which obviously aren't allowed with jenkins:jenkins per the -u 1000:1000 as shown above.

          I'm aware that volumes can maintain permissions from the primary host (podman0) with something like -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace:U will allow or change the permissions within the container to 1000:1000, but it then causes the permissions outside of the container to change to the subuid:subgid value, which further confuses the Jenkins agent on successive jobs. This possibly is more of a feature request vs. bug, otherwise I'm hoping that someone has a workaround for this. 
          New: Utilizing rootless podman (alias docker) with Jenkins Pipeline has workspace permission issues.

          A given agent "Remote root directory" is set to /home/jenkins/ssd  The idea is to provide a specific SSD for workspace speed purposes.

           
          {code:java}
          jenkins@podman0 ~]$ ls -l /home/jenkins/ssd/
          total 1488
          drwx------. 3 jenkins jenkins      26 Aug 29 16:55 caches
          drwx------. 4 jenkins jenkins      34 Aug 29 15:57 remoting
          -rw-------. 1 jenkins jenkins 1522481 Aug 29 15:57 remoting.jar
          drwxr-xr-x. 2 jenkins jenkins       6 Aug 31 15:53 workspace
          {code}
           

          Below is the log of the permission issues that occurs. 
          {code:java}
          Changelog calculated successfully.
          Checkout done
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] isUnix
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] sh
          + docker inspect -f . registry.internal.mycomp.com/rhel:72
          .
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] withDockerContainer
          podman0 does not seem to be running inside a container
          $ docker run -t -d -u 1000:1000 -w /home/jenkins/ssd/workspace/TEST_PIPELINE -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace/TEST_PIPELINE:rw,z -v /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** registry.internal.mycomp.com/rhel:72 cat
          $ docker top 26dac8250b54021d18ad6006d3c42747eb2af447a993fff59621d0b6053f5dcb -eo pid,comm
          [Pipeline] {
          [Pipeline] sh
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-log.txt: Permission denied
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt.tmp: Permission denied
          mv: failed to access '/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt': Permission denied{code}
          It is understood that Jenkins Pipeline is meant to work with Docker, but I think some small changes could allow it to work with podman's backward compatibility (ln -s podman docker) to work. The root of the problem is permissions when a volume is mapped inside the container, I see the following:

           
          {code:java}
          sh-4.4$ ls -l /home/jenkins/ssd
          total 0
          drwxr-xr-x. 3 root root 33 Sep  1 15:24 workspace
          {code}
           

          The permissions have changed to root:root which obviously aren't allowed with jenkins:jenkins per the -u 1000:1000 as shown above.

          I'm aware that volumes can maintain permissions from the primary host (podman0) with something like -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace:U will allow or change the permissions within the container to 1000:1000, but it then causes the permissions outside of the container to change to the subuid:subgid value, which further confuses the Jenkins agent on successive jobs. This possibly is more of a feature request vs. bug, otherwise I'm hoping that someone has a workaround for this. 

          One possibility would be to detect that "podman" is actually being used and replace the '-u 1000:1000' with '

          --userns=keep-id'.  This appears to create the appropriate permissions within the container and the host as well.
          Shane Walton made changes -
          Description Original: Utilizing rootless podman (alias docker) with Jenkins Pipeline has workspace permission issues.

          A given agent "Remote root directory" is set to /home/jenkins/ssd  The idea is to provide a specific SSD for workspace speed purposes.

           
          {code:java}
          jenkins@podman0 ~]$ ls -l /home/jenkins/ssd/
          total 1488
          drwx------. 3 jenkins jenkins      26 Aug 29 16:55 caches
          drwx------. 4 jenkins jenkins      34 Aug 29 15:57 remoting
          -rw-------. 1 jenkins jenkins 1522481 Aug 29 15:57 remoting.jar
          drwxr-xr-x. 2 jenkins jenkins       6 Aug 31 15:53 workspace
          {code}
           

          Below is the log of the permission issues that occurs. 
          {code:java}
          Changelog calculated successfully.
          Checkout done
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] isUnix
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] sh
          + docker inspect -f . registry.internal.mycomp.com/rhel:72
          .
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] withDockerContainer
          podman0 does not seem to be running inside a container
          $ docker run -t -d -u 1000:1000 -w /home/jenkins/ssd/workspace/TEST_PIPELINE -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace/TEST_PIPELINE:rw,z -v /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** registry.internal.mycomp.com/rhel:72 cat
          $ docker top 26dac8250b54021d18ad6006d3c42747eb2af447a993fff59621d0b6053f5dcb -eo pid,comm
          [Pipeline] {
          [Pipeline] sh
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-log.txt: Permission denied
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt.tmp: Permission denied
          mv: failed to access '/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt': Permission denied{code}
          It is understood that Jenkins Pipeline is meant to work with Docker, but I think some small changes could allow it to work with podman's backward compatibility (ln -s podman docker) to work. The root of the problem is permissions when a volume is mapped inside the container, I see the following:

           
          {code:java}
          sh-4.4$ ls -l /home/jenkins/ssd
          total 0
          drwxr-xr-x. 3 root root 33 Sep  1 15:24 workspace
          {code}
           

          The permissions have changed to root:root which obviously aren't allowed with jenkins:jenkins per the -u 1000:1000 as shown above.

          I'm aware that volumes can maintain permissions from the primary host (podman0) with something like -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace:U will allow or change the permissions within the container to 1000:1000, but it then causes the permissions outside of the container to change to the subuid:subgid value, which further confuses the Jenkins agent on successive jobs. This possibly is more of a feature request vs. bug, otherwise I'm hoping that someone has a workaround for this. 

          One possibility would be to detect that "podman" is actually being used and replace the '-u 1000:1000' with '

          --userns=keep-id'.  This appears to create the appropriate permissions within the container and the host as well.
          New: Utilizing rootless podman (alias docker) with Jenkins Pipeline has workspace permission issues.

          A given agent "Remote root directory" is set to /home/jenkins/ssd  The idea is to provide a specific SSD for workspace speed purposes.

           
          {code:java}
          jenkins@podman0 ~]$ ls -l /home/jenkins/ssd/
          total 1488
          drwx------. 3 jenkins jenkins      26 Aug 29 16:55 caches
          drwx------. 4 jenkins jenkins      34 Aug 29 15:57 remoting
          -rw-------. 1 jenkins jenkins 1522481 Aug 29 15:57 remoting.jar
          drwxr-xr-x. 2 jenkins jenkins       6 Aug 31 15:53 workspace
          {code}
           

          Below is the log of the permission issues that occurs. 
          {code:java}
          Changelog calculated successfully.
          Checkout done
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] isUnix
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] sh
          + docker inspect -f . registry.internal.mycomp.com/rhel:72
          .
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] withDockerContainer
          podman0 does not seem to be running inside a container
          $ docker run -t -d -u 1000:1000 -w /home/jenkins/ssd/workspace/TEST_PIPELINE -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace/TEST_PIPELINE:rw,z -v /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** registry.internal.mycomp.com/rhel:72 cat
          $ docker top 26dac8250b54021d18ad6006d3c42747eb2af447a993fff59621d0b6053f5dcb -eo pid,comm
          [Pipeline] {
          [Pipeline] sh
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-log.txt: Permission denied
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt.tmp: Permission denied
          mv: failed to access '/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt': Permission denied{code}
          It is understood that Jenkins Pipeline is meant to work with Docker, but I think some small changes could allow it to work with podman's backward compatibility (ln -s podman docker) to work. The root of the problem is permissions when a volume is mapped inside the container, I see the following:

           
          {code:java}
          sh-4.4$ ls -l /home/jenkins/ssd
          total 0
          drwxr-xr-x. 3 root root 33 Sep  1 15:24 workspace
          {code}
           

          The permissions have changed to root:root which obviously aren't allowed with jenkins:jenkins per the -u 1000:1000 as shown above.

          I'm aware that volumes can maintain permissions from the primary host (podman0) with something like -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace:U will allow or change the permissions within the container to 1000:1000, but it then causes the permissions outside of the container to change to the subuid:subgid value, which further confuses the Jenkins agent on successive jobs. This possibly is more of a feature request vs. bug, otherwise I'm hoping that someone has a workaround for this. 

          One possibility would be to detect that "podman" is actually being used and replace the '-u 1000:1000' with '--userns=keep-id'.  This appears to create the appropriate permissions within the container and the host as well.
          Shane Walton made changes -
          Labels Original: docker jenkins pipeline plugin podman New: docker jenkins jenkins-plugin pipeline plugin podman

          Shane Walton added a comment -

          I was able to workaround this issue with wrapping podman with a bash script named docker.

           

          #!/bin/bash
           
          while [[ $# -gt 0 ]]; do
            case $1 in
              -u)
                POSITIONAL_ARGS+=("$1") # save -u
                POSITIONAL_ARGS+=("$2") # save uid:gid
                shift # past argument
                shift # past value
                POSITIONAL_ARGS+=("--userns=keep-id")
                ;;
              *)
                POSITIONAL_ARGS+=("$1") # save positional arg
                shift # past argument
                ;;
            esac
          done
           
          set -- "${POSITIONAL_ARGS[@]}"
           
          /usr/bin/podman "$@"
          

           

          Shane Walton added a comment - I was able to workaround this issue with wrapping podman with a bash script named docker.   #!/bin/bash   while [[ $# -gt 0 ]]; do   case $1 in     -u)       POSITIONAL_ARGS+=( "$1" ) # save -u       POSITIONAL_ARGS+=( "$2" ) # save uid:gid       shift # past argument       shift # past value       POSITIONAL_ARGS+=( "--userns=keep-id" )       ;;     *)       POSITIONAL_ARGS+=( "$1" ) # save positional arg       shift # past argument       ;;   esac done   set -- "${POSITIONAL_ARGS[@]}"   /usr/bin/podman "$@"  

            Unassigned Unassigned
            shane_walton Shane Walton
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: