Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-69504

rootless docker/podman workspace volume permissions

XMLWordPrintable

      Utilizing rootless podman (alias docker) with Jenkins Pipeline has workspace permission issues.

      A given agent "Remote root directory" is set to /home/jenkins/ssd  The idea is to provide a specific SSD for workspace speed purposes.

       

      jenkins@podman0 ~]$ ls -l /home/jenkins/ssd/
total 1488
drwx------. 3 jenkins jenkins      26 Aug 29 16:55 caches
drwx------. 4 jenkins jenkins      34 Aug 29 15:57 remoting
-rw-------. 1 jenkins jenkins 1522481 Aug 29 15:57 remoting.jar
drwxr-xr-x. 2 jenkins jenkins       6 Aug 31 15:53 workspace

       

      Below is the log of the permission issues that occurs. 

      Changelog calculated successfully.
Checkout done
[Pipeline] withEnv 
[Pipeline] { 
[Pipeline] isUnix 
[Pipeline] withEnv 
[Pipeline] { 
[Pipeline] sh
+ docker inspect -f . registry.internal.mycomp.com/rhel:72
.
[Pipeline] } 
[Pipeline] // withEnv 
[Pipeline] withDockerContainer
podman0 does not seem to be running inside a container
$ docker run -t -d -u 1000:1000 -w /home/jenkins/ssd/workspace/TEST_PIPELINE -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace/TEST_PIPELINE:rw,z -v /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** registry.internal.mycomp.com/rhel:72 cat 
$ docker top 26dac8250b54021d18ad6006d3c42747eb2af447a993fff59621d0b6053f5dcb -eo pid,comm 
[Pipeline] { 
[Pipeline] sh 
sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-log.txt: Permission denied 
sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt.tmp: Permission denied 
mv: failed to access '/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt': Permission denied

      It is understood that Jenkins Pipeline is meant to work with Docker, but I think some small changes could allow it to work with podman's backward compatibility (ln -s podman docker) to work. The root of the problem is permissions when a volume is mapped inside the container, I see the following:

       

      sh-4.4$ ls -l /home/jenkins/ssd
total 0
drwxr-xr-x. 3 root root 33 Sep  1 15:24 workspace

       

      The permissions have changed to root:root which obviously aren't allowed with jenkins:jenkins per the -u 1000:1000 as shown above.

      I'm aware that volumes can maintain permissions from the primary host (podman0) with something like -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace:U will allow or change the permissions within the container to 1000:1000, but it then causes the permissions outside of the container to change to the subuid:subgid value, which further confuses the Jenkins agent on successive jobs. This possibly is more of a feature request vs. bug, otherwise I'm hoping that someone has a workaround for this. 

      One possibility would be to detect that "podman" is actually being used and replace the 'u 1000:1000' with '-userns=keep-id'.  This appears to create the appropriate permissions within the container and the host as well.

          [JENKINS-69504] rootless docker/podman workspace volume permissions

          Shane Walton created issue -
          Shane Walton made changes -
          Description Original: Utilizing rootless podman (alias docker) with Jenkins Pipeline has workspace permission issues.

          A given agent "Remote root directory" is set to /home/jenkins/ssd  The idea is to provide a specific SSD for workspace speed purposes.

           
          {code:java}
          jenkins@podman0 ~]$ ls -l /home/jenkins/ssd/
          total 1488
          drwx------. 3 jenkins jenkins      26 Aug 29 16:55 caches
          drwx------. 4 jenkins jenkins      34 Aug 29 15:57 remoting
          -rw-------. 1 jenkins jenkins 1522481 Aug 29 15:57 remoting.jar
          drwxr-xr-x. 2 jenkins jenkins       6 Aug 31 15:53 workspace
          {code}
           

          Below is the log of the permission issues that occurs. 
          {code:java}
          Changelog calculated successfully.
          Checkout done
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] isUnix
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] sh
          + docker inspect -f . registry.internal.mycomp.com/rhel:72
          .
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] withDockerContainer
          podman0 does not seem to be running inside a container
          $ docker run -t -d -u 1000:1000 -w /home/jenkins/ssd/workspace/TEST_PIPELINE -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace/TEST_PIPELINE:rw,z -v /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** registry.internal.mycomp.com/rhel:72 cat
          $ docker top 26dac8250b54021d18ad6006d3c42747eb2af447a993fff59621d0b6053f5dcb -eo pid,comm
          [Pipeline] {
          [Pipeline] sh
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-log.txt: Permission denied
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt.tmp: Permission denied
          mv: failed to access '/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt': Permission denied{code}
          It is understood that Jenkins Pipeline is meant to work with Docker, but I think some small changes could allow it to work with podman's backward compatibility (ln -s podman docker) to work. The root of the problem is permissions when a volume is mapped inside the container, I see the following:

           
          {code:java}
          sh-4.4$ ls -l /home/jenkins/ssd
          total 0
          drwxr-xr-x. 3 root root 33 Sep  1 15:24 workspace
          {code}
           

          The permissions have changed to root:root which obviously aren't allowed with jenkins:jenkins per the -u 1000:1000 as shown above.

          I'm aware that volumes can maintain permissions from the primary host (podman0) with something like -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace:U will allow or change the permissions within the container to 1000:1000, but it then causes the permissions outside of the container to change to the subuid:subgid value, which further confuses the Jenkins agent on successive jobs. This possibly is more of a feature request vs. bug, otherwise I'm hoping that someone has a workaround for this.            		New: Utilizing rootless podman (alias docker) with Jenkins Pipeline has workspace permission issues.

          A given agent "Remote root directory" is set to /home/jenkins/ssd  The idea is to provide a specific SSD for workspace speed purposes.

           
          {code:java}
          jenkins@podman0 ~]$ ls -l /home/jenkins/ssd/
          total 1488
          drwx------. 3 jenkins jenkins      26 Aug 29 16:55 caches
          drwx------. 4 jenkins jenkins      34 Aug 29 15:57 remoting
          -rw-------. 1 jenkins jenkins 1522481 Aug 29 15:57 remoting.jar
          drwxr-xr-x. 2 jenkins jenkins       6 Aug 31 15:53 workspace
          {code}
           

          Below is the log of the permission issues that occurs. 
          {code:java}
          Changelog calculated successfully.
          Checkout done
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] isUnix
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] sh
          + docker inspect -f . registry.internal.mycomp.com/rhel:72
          .
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] withDockerContainer
          podman0 does not seem to be running inside a container
          $ docker run -t -d -u 1000:1000 -w /home/jenkins/ssd/workspace/TEST_PIPELINE -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace/TEST_PIPELINE:rw,z -v /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** registry.internal.mycomp.com/rhel:72 cat
          $ docker top 26dac8250b54021d18ad6006d3c42747eb2af447a993fff59621d0b6053f5dcb -eo pid,comm
          [Pipeline] {
          [Pipeline] sh
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-log.txt: Permission denied
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt.tmp: Permission denied
          mv: failed to access '/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt': Permission denied{code}
          It is understood that Jenkins Pipeline is meant to work with Docker, but I think some small changes could allow it to work with podman's backward compatibility (ln -s podman docker) to work. The root of the problem is permissions when a volume is mapped inside the container, I see the following:

           
          {code:java}
          sh-4.4$ ls -l /home/jenkins/ssd
          total 0
          drwxr-xr-x. 3 root root 33 Sep  1 15:24 workspace
          {code}
           

          The permissions have changed to root:root which obviously aren't allowed with jenkins:jenkins per the -u 1000:1000 as shown above.

          I'm aware that volumes can maintain permissions from the primary host (podman0) with something like -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace:U will allow or change the permissions within the container to 1000:1000, but it then causes the permissions outside of the container to change to the subuid:subgid value, which further confuses the Jenkins agent on successive jobs. This possibly is more of a feature request vs. bug, otherwise I'm hoping that someone has a workaround for this. 

          One possibility would be to detect that "podman" is actually being used and replace the '-u 1000:1000' with '

          --userns=keep-id'.  This appears to create the appropriate permissions within the container and the host as well.
          Shane Walton made changes -
          Description Original: Utilizing rootless podman (alias docker) with Jenkins Pipeline has workspace permission issues.

          A given agent "Remote root directory" is set to /home/jenkins/ssd  The idea is to provide a specific SSD for workspace speed purposes.

           
          {code:java}
          jenkins@podman0 ~]$ ls -l /home/jenkins/ssd/
          total 1488
          drwx------. 3 jenkins jenkins      26 Aug 29 16:55 caches
          drwx------. 4 jenkins jenkins      34 Aug 29 15:57 remoting
          -rw-------. 1 jenkins jenkins 1522481 Aug 29 15:57 remoting.jar
          drwxr-xr-x. 2 jenkins jenkins       6 Aug 31 15:53 workspace
          {code}
           

          Below is the log of the permission issues that occurs. 
          {code:java}
          Changelog calculated successfully.
          Checkout done
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] isUnix
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] sh
          + docker inspect -f . registry.internal.mycomp.com/rhel:72
          .
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] withDockerContainer
          podman0 does not seem to be running inside a container
          $ docker run -t -d -u 1000:1000 -w /home/jenkins/ssd/workspace/TEST_PIPELINE -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace/TEST_PIPELINE:rw,z -v /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** registry.internal.mycomp.com/rhel:72 cat
          $ docker top 26dac8250b54021d18ad6006d3c42747eb2af447a993fff59621d0b6053f5dcb -eo pid,comm
          [Pipeline] {
          [Pipeline] sh
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-log.txt: Permission denied
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt.tmp: Permission denied
          mv: failed to access '/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt': Permission denied{code}
          It is understood that Jenkins Pipeline is meant to work with Docker, but I think some small changes could allow it to work with podman's backward compatibility (ln -s podman docker) to work. The root of the problem is permissions when a volume is mapped inside the container, I see the following:

           
          {code:java}
          sh-4.4$ ls -l /home/jenkins/ssd
          total 0
          drwxr-xr-x. 3 root root 33 Sep  1 15:24 workspace
          {code}
           

          The permissions have changed to root:root which obviously aren't allowed with jenkins:jenkins per the -u 1000:1000 as shown above.

          I'm aware that volumes can maintain permissions from the primary host (podman0) with something like -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace:U will allow or change the permissions within the container to 1000:1000, but it then causes the permissions outside of the container to change to the subuid:subgid value, which further confuses the Jenkins agent on successive jobs. This possibly is more of a feature request vs. bug, otherwise I'm hoping that someone has a workaround for this. 

          One possibility would be to detect that "podman" is actually being used and replace the '-u 1000:1000' with '

          --userns=keep-id'.  This appears to create the appropriate permissions within the container and the host as well.           		New: Utilizing rootless podman (alias docker) with Jenkins Pipeline has workspace permission issues.

          A given agent "Remote root directory" is set to /home/jenkins/ssd  The idea is to provide a specific SSD for workspace speed purposes.

           
          {code:java}
          jenkins@podman0 ~]$ ls -l /home/jenkins/ssd/
          total 1488
          drwx------. 3 jenkins jenkins      26 Aug 29 16:55 caches
          drwx------. 4 jenkins jenkins      34 Aug 29 15:57 remoting
          -rw-------. 1 jenkins jenkins 1522481 Aug 29 15:57 remoting.jar
          drwxr-xr-x. 2 jenkins jenkins       6 Aug 31 15:53 workspace
          {code}
           

          Below is the log of the permission issues that occurs. 
          {code:java}
          Changelog calculated successfully.
          Checkout done
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] isUnix
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] sh
          + docker inspect -f . registry.internal.mycomp.com/rhel:72
          .
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] withDockerContainer
          podman0 does not seem to be running inside a container
          $ docker run -t -d -u 1000:1000 -w /home/jenkins/ssd/workspace/TEST_PIPELINE -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace/TEST_PIPELINE:rw,z -v /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** registry.internal.mycomp.com/rhel:72 cat
          $ docker top 26dac8250b54021d18ad6006d3c42747eb2af447a993fff59621d0b6053f5dcb -eo pid,comm
          [Pipeline] {
          [Pipeline] sh
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-log.txt: Permission denied
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt.tmp: Permission denied
          mv: failed to access '/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt': Permission denied{code}
          It is understood that Jenkins Pipeline is meant to work with Docker, but I think some small changes could allow it to work with podman's backward compatibility (ln -s podman docker) to work. The root of the problem is permissions when a volume is mapped inside the container, I see the following:

           
          {code:java}
          sh-4.4$ ls -l /home/jenkins/ssd
          total 0
          drwxr-xr-x. 3 root root 33 Sep  1 15:24 workspace
          {code}
           

          The permissions have changed to root:root which obviously aren't allowed with jenkins:jenkins per the -u 1000:1000 as shown above.

          I'm aware that volumes can maintain permissions from the primary host (podman0) with something like -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace:U will allow or change the permissions within the container to 1000:1000, but it then causes the permissions outside of the container to change to the subuid:subgid value, which further confuses the Jenkins agent on successive jobs. This possibly is more of a feature request vs. bug, otherwise I'm hoping that someone has a workaround for this. 

          One possibility would be to detect that "podman" is actually being used and replace the '-u 1000:1000' with '--userns=keep-id'.  This appears to create the appropriate permissions within the container and the host as well.
          Shane Walton made changes -
          Labels Original: docker jenkins pipeline plugin podman New: docker jenkins jenkins-plugin pipeline plugin podman

          Shane Walton added a comment -

          I was able to workaround this issue with wrapping podman with a bash script named docker.

           

          #!/bin/bash
 
while [[ $# -gt 0 ]]; do
  case $1 in
    -u)
      POSITIONAL_ARGS+=("$1") # save -u
      POSITIONAL_ARGS+=("$2") # save uid:gid
      shift # past argument
      shift # past value
      POSITIONAL_ARGS+=("--userns=keep-id")
      ;;
    *)
      POSITIONAL_ARGS+=("$1") # save positional arg
      shift # past argument
      ;;
  esac
done
 
set -- "${POSITIONAL_ARGS[@]}"
 
/usr/bin/podman "$@"

           

          Shane Walton added a comment - I was able to workaround this issue with wrapping podman with a bash script named docker.   #!/bin/bash   while [[ $# -gt 0 ]]; do   case $1 in     -u)       POSITIONAL_ARGS+=( "$1" ) # save -u       POSITIONAL_ARGS+=( "$2" ) # save uid:gid       shift # past argument       shift # past value       POSITIONAL_ARGS+=( "--userns=keep-id" )       ;;     *)       POSITIONAL_ARGS+=( "$1" ) # save positional arg       shift # past argument       ;;   esac done   set -- "${POSITIONAL_ARGS[@]}"   /usr/bin/podman "$@"  

            Unassigned Unassigned
            shane_walton Shane Walton
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: