Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-69504

rootless docker/podman workspace volume permissions

      Utilizing rootless podman (alias docker) with Jenkins Pipeline has workspace permission issues.

      A given agent "Remote root directory" is set to /home/jenkins/ssd  The idea is to provide a specific SSD for workspace speed purposes.

       

      jenkins@podman0 ~]$ ls -l /home/jenkins/ssd/
      total 1488
      drwx------. 3 jenkins jenkins      26 Aug 29 16:55 caches
      drwx------. 4 jenkins jenkins      34 Aug 29 15:57 remoting
      -rw-------. 1 jenkins jenkins 1522481 Aug 29 15:57 remoting.jar
      drwxr-xr-x. 2 jenkins jenkins       6 Aug 31 15:53 workspace
      

       

      Below is the log of the permission issues that occurs. 

      Changelog calculated successfully.
      Checkout done
      [Pipeline] withEnv 
      [Pipeline] { 
      [Pipeline] isUnix 
      [Pipeline] withEnv 
      [Pipeline] { 
      [Pipeline] sh
      + docker inspect -f . registry.internal.mycomp.com/rhel:72
      .
      [Pipeline] } 
      [Pipeline] // withEnv 
      [Pipeline] withDockerContainer
      podman0 does not seem to be running inside a container
      $ docker run -t -d -u 1000:1000 -w /home/jenkins/ssd/workspace/TEST_PIPELINE -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace/TEST_PIPELINE:rw,z -v /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** registry.internal.mycomp.com/rhel:72 cat 
      $ docker top 26dac8250b54021d18ad6006d3c42747eb2af447a993fff59621d0b6053f5dcb -eo pid,comm 
      [Pipeline] { 
      [Pipeline] sh 
      sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-log.txt: Permission denied 
      sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt.tmp: Permission denied 
      mv: failed to access '/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt': Permission denied

      It is understood that Jenkins Pipeline is meant to work with Docker, but I think some small changes could allow it to work with podman's backward compatibility (ln -s podman docker) to work. The root of the problem is permissions when a volume is mapped inside the container, I see the following:

       

      sh-4.4$ ls -l /home/jenkins/ssd
      total 0
      drwxr-xr-x. 3 root root 33 Sep  1 15:24 workspace
      

       

      The permissions have changed to root:root which obviously aren't allowed with jenkins:jenkins per the -u 1000:1000 as shown above.

      I'm aware that volumes can maintain permissions from the primary host (podman0) with something like -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace:U will allow or change the permissions within the container to 1000:1000, but it then causes the permissions outside of the container to change to the subuid:subgid value, which further confuses the Jenkins agent on successive jobs. This possibly is more of a feature request vs. bug, otherwise I'm hoping that someone has a workaround for this. 

      One possibility would be to detect that "podman" is actually being used and replace the 'u 1000:1000' with '-userns=keep-id'.  This appears to create the appropriate permissions within the container and the host as well.

          [JENKINS-69504] rootless docker/podman workspace volume permissions

          Shane Walton created issue -
          Shane Walton made changes -
          Description Original: Utilizing rootless podman (alias docker) with Jenkins Pipeline has workspace permission issues.

          A given agent "Remote root directory" is set to /home/jenkins/ssd  The idea is to provide a specific SSD for workspace speed purposes.

           
          {code:java}
          jenkins@podman0 ~]$ ls -l /home/jenkins/ssd/
          total 1488
          drwx------. 3 jenkins jenkins      26 Aug 29 16:55 caches
          drwx------. 4 jenkins jenkins      34 Aug 29 15:57 remoting
          -rw-------. 1 jenkins jenkins 1522481 Aug 29 15:57 remoting.jar
          drwxr-xr-x. 2 jenkins jenkins       6 Aug 31 15:53 workspace
          {code}
           

          Below is the log of the permission issues that occurs. 
          {code:java}
          Changelog calculated successfully.
          Checkout done
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] isUnix
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] sh
          + docker inspect -f . registry.internal.mycomp.com/rhel:72
          .
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] withDockerContainer
          podman0 does not seem to be running inside a container
          $ docker run -t -d -u 1000:1000 -w /home/jenkins/ssd/workspace/TEST_PIPELINE -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace/TEST_PIPELINE:rw,z -v /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** registry.internal.mycomp.com/rhel:72 cat
          $ docker top 26dac8250b54021d18ad6006d3c42747eb2af447a993fff59621d0b6053f5dcb -eo pid,comm
          [Pipeline] {
          [Pipeline] sh
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-log.txt: Permission denied
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt.tmp: Permission denied
          mv: failed to access '/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt': Permission denied{code}
          It is understood that Jenkins Pipeline is meant to work with Docker, but I think some small changes could allow it to work with podman's backward compatibility (ln -s podman docker) to work. The root of the problem is permissions when a volume is mapped inside the container, I see the following:

           
          {code:java}
          sh-4.4$ ls -l /home/jenkins/ssd
          total 0
          drwxr-xr-x. 3 root root 33 Sep  1 15:24 workspace
          {code}
           

          The permissions have changed to root:root which obviously aren't allowed with jenkins:jenkins per the -u 1000:1000 as shown above.

          I'm aware that volumes can maintain permissions from the primary host (podman0) with something like -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace:U will allow or change the permissions within the container to 1000:1000, but it then causes the permissions outside of the container to change to the subuid:subgid value, which further confuses the Jenkins agent on successive jobs. This possibly is more of a feature request vs. bug, otherwise I'm hoping that someone has a workaround for this. 
          New: Utilizing rootless podman (alias docker) with Jenkins Pipeline has workspace permission issues.

          A given agent "Remote root directory" is set to /home/jenkins/ssd  The idea is to provide a specific SSD for workspace speed purposes.

           
          {code:java}
          jenkins@podman0 ~]$ ls -l /home/jenkins/ssd/
          total 1488
          drwx------. 3 jenkins jenkins      26 Aug 29 16:55 caches
          drwx------. 4 jenkins jenkins      34 Aug 29 15:57 remoting
          -rw-------. 1 jenkins jenkins 1522481 Aug 29 15:57 remoting.jar
          drwxr-xr-x. 2 jenkins jenkins       6 Aug 31 15:53 workspace
          {code}
           

          Below is the log of the permission issues that occurs. 
          {code:java}
          Changelog calculated successfully.
          Checkout done
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] isUnix
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] sh
          + docker inspect -f . registry.internal.mycomp.com/rhel:72
          .
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] withDockerContainer
          podman0 does not seem to be running inside a container
          $ docker run -t -d -u 1000:1000 -w /home/jenkins/ssd/workspace/TEST_PIPELINE -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace/TEST_PIPELINE:rw,z -v /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** registry.internal.mycomp.com/rhel:72 cat
          $ docker top 26dac8250b54021d18ad6006d3c42747eb2af447a993fff59621d0b6053f5dcb -eo pid,comm
          [Pipeline] {
          [Pipeline] sh
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-log.txt: Permission denied
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt.tmp: Permission denied
          mv: failed to access '/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt': Permission denied{code}
          It is understood that Jenkins Pipeline is meant to work with Docker, but I think some small changes could allow it to work with podman's backward compatibility (ln -s podman docker) to work. The root of the problem is permissions when a volume is mapped inside the container, I see the following:

           
          {code:java}
          sh-4.4$ ls -l /home/jenkins/ssd
          total 0
          drwxr-xr-x. 3 root root 33 Sep  1 15:24 workspace
          {code}
           

          The permissions have changed to root:root which obviously aren't allowed with jenkins:jenkins per the -u 1000:1000 as shown above.

          I'm aware that volumes can maintain permissions from the primary host (podman0) with something like -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace:U will allow or change the permissions within the container to 1000:1000, but it then causes the permissions outside of the container to change to the subuid:subgid value, which further confuses the Jenkins agent on successive jobs. This possibly is more of a feature request vs. bug, otherwise I'm hoping that someone has a workaround for this. 

          One possibility would be to detect that "podman" is actually being used and replace the '-u 1000:1000' with '

          --userns=keep-id'.  This appears to create the appropriate permissions within the container and the host as well.
          Shane Walton made changes -
          Description Original: Utilizing rootless podman (alias docker) with Jenkins Pipeline has workspace permission issues.

          A given agent "Remote root directory" is set to /home/jenkins/ssd  The idea is to provide a specific SSD for workspace speed purposes.

           
          {code:java}
          jenkins@podman0 ~]$ ls -l /home/jenkins/ssd/
          total 1488
          drwx------. 3 jenkins jenkins      26 Aug 29 16:55 caches
          drwx------. 4 jenkins jenkins      34 Aug 29 15:57 remoting
          -rw-------. 1 jenkins jenkins 1522481 Aug 29 15:57 remoting.jar
          drwxr-xr-x. 2 jenkins jenkins       6 Aug 31 15:53 workspace
          {code}
           

          Below is the log of the permission issues that occurs. 
          {code:java}
          Changelog calculated successfully.
          Checkout done
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] isUnix
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] sh
          + docker inspect -f . registry.internal.mycomp.com/rhel:72
          .
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] withDockerContainer
          podman0 does not seem to be running inside a container
          $ docker run -t -d -u 1000:1000 -w /home/jenkins/ssd/workspace/TEST_PIPELINE -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace/TEST_PIPELINE:rw,z -v /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** registry.internal.mycomp.com/rhel:72 cat
          $ docker top 26dac8250b54021d18ad6006d3c42747eb2af447a993fff59621d0b6053f5dcb -eo pid,comm
          [Pipeline] {
          [Pipeline] sh
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-log.txt: Permission denied
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt.tmp: Permission denied
          mv: failed to access '/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt': Permission denied{code}
          It is understood that Jenkins Pipeline is meant to work with Docker, but I think some small changes could allow it to work with podman's backward compatibility (ln -s podman docker) to work. The root of the problem is permissions when a volume is mapped inside the container, I see the following:

           
          {code:java}
          sh-4.4$ ls -l /home/jenkins/ssd
          total 0
          drwxr-xr-x. 3 root root 33 Sep  1 15:24 workspace
          {code}
           

          The permissions have changed to root:root which obviously aren't allowed with jenkins:jenkins per the -u 1000:1000 as shown above.

          I'm aware that volumes can maintain permissions from the primary host (podman0) with something like -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace:U will allow or change the permissions within the container to 1000:1000, but it then causes the permissions outside of the container to change to the subuid:subgid value, which further confuses the Jenkins agent on successive jobs. This possibly is more of a feature request vs. bug, otherwise I'm hoping that someone has a workaround for this. 

          One possibility would be to detect that "podman" is actually being used and replace the '-u 1000:1000' with '

          --userns=keep-id'.  This appears to create the appropriate permissions within the container and the host as well.
          New: Utilizing rootless podman (alias docker) with Jenkins Pipeline has workspace permission issues.

          A given agent "Remote root directory" is set to /home/jenkins/ssd  The idea is to provide a specific SSD for workspace speed purposes.

           
          {code:java}
          jenkins@podman0 ~]$ ls -l /home/jenkins/ssd/
          total 1488
          drwx------. 3 jenkins jenkins      26 Aug 29 16:55 caches
          drwx------. 4 jenkins jenkins      34 Aug 29 15:57 remoting
          -rw-------. 1 jenkins jenkins 1522481 Aug 29 15:57 remoting.jar
          drwxr-xr-x. 2 jenkins jenkins       6 Aug 31 15:53 workspace
          {code}
           

          Below is the log of the permission issues that occurs. 
          {code:java}
          Changelog calculated successfully.
          Checkout done
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] isUnix
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] sh
          + docker inspect -f . registry.internal.mycomp.com/rhel:72
          .
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] withDockerContainer
          podman0 does not seem to be running inside a container
          $ docker run -t -d -u 1000:1000 -w /home/jenkins/ssd/workspace/TEST_PIPELINE -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace/TEST_PIPELINE:rw,z -v /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** registry.internal.mycomp.com/rhel:72 cat
          $ docker top 26dac8250b54021d18ad6006d3c42747eb2af447a993fff59621d0b6053f5dcb -eo pid,comm
          [Pipeline] {
          [Pipeline] sh
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-log.txt: Permission denied
          sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt.tmp: Permission denied
          mv: failed to access '/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt': Permission denied{code}
          It is understood that Jenkins Pipeline is meant to work with Docker, but I think some small changes could allow it to work with podman's backward compatibility (ln -s podman docker) to work. The root of the problem is permissions when a volume is mapped inside the container, I see the following:

           
          {code:java}
          sh-4.4$ ls -l /home/jenkins/ssd
          total 0
          drwxr-xr-x. 3 root root 33 Sep  1 15:24 workspace
          {code}
           

          The permissions have changed to root:root which obviously aren't allowed with jenkins:jenkins per the -u 1000:1000 as shown above.

          I'm aware that volumes can maintain permissions from the primary host (podman0) with something like -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace:U will allow or change the permissions within the container to 1000:1000, but it then causes the permissions outside of the container to change to the subuid:subgid value, which further confuses the Jenkins agent on successive jobs. This possibly is more of a feature request vs. bug, otherwise I'm hoping that someone has a workaround for this. 

          One possibility would be to detect that "podman" is actually being used and replace the '-u 1000:1000' with '--userns=keep-id'.  This appears to create the appropriate permissions within the container and the host as well.
          Shane Walton made changes -
          Labels Original: docker jenkins pipeline plugin podman New: docker jenkins jenkins-plugin pipeline plugin podman

            Unassigned Unassigned
            shane_walton Shane Walton
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: