[JENKINS-69504] rootless docker/podman workspace volume permissions

Type: Bug
Resolution: Unresolved
Priority: Minor
Component/s: docker-workflow-plugin
Labels:
- docker
- jenkins
- jenkins-plugin
- pipeline
- plugin
- podman
Environment:
Jenkins: 2.346.3 docker.io/jenkins/jenkins:lts-jdk11
RedHat: 8.6
Podman: 4.1.1
user/group: jenkins (1000) / jenkins (1000)

Similar Issues:
Powered by SuggestiMate

Show

Utilizing rootless podman (alias docker) with Jenkins Pipeline has workspace permission issues.

A given agent "Remote root directory" is set to /home/jenkins/ssd The idea is to provide a specific SSD for workspace speed purposes.

jenkins@podman0 ~]$ ls -l /home/jenkins/ssd/
total 1488
drwx------. 3 jenkins jenkins      26 Aug 29 16:55 caches
drwx------. 4 jenkins jenkins      34 Aug 29 15:57 remoting
-rw-------. 1 jenkins jenkins 1522481 Aug 29 15:57 remoting.jar
drwxr-xr-x. 2 jenkins jenkins       6 Aug 31 15:53 workspace

Below is the log of the permission issues that occurs.

Changelog calculated successfully.
Checkout done
[Pipeline] withEnv 
[Pipeline] { 
[Pipeline] isUnix 
[Pipeline] withEnv 
[Pipeline] { 
[Pipeline] sh
+ docker inspect -f . registry.internal.mycomp.com/rhel:72
.
[Pipeline] } 
[Pipeline] // withEnv 
[Pipeline] withDockerContainer
podman0 does not seem to be running inside a container
$ docker run -t -d -u 1000:1000 -w /home/jenkins/ssd/workspace/TEST_PIPELINE -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace/TEST_PIPELINE:rw,z -v /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** registry.internal.mycomp.com/rhel:72 cat 
$ docker top 26dac8250b54021d18ad6006d3c42747eb2af447a993fff59621d0b6053f5dcb -eo pid,comm 
[Pipeline] { 
[Pipeline] sh 
sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-log.txt: Permission denied 
sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt.tmp: Permission denied 
mv: failed to access '/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt': Permission denied

It is understood that Jenkins Pipeline is meant to work with Docker, but I think some small changes could allow it to work with podman's backward compatibility (ln -s podman docker) to work. The root of the problem is permissions when a volume is mapped inside the container, I see the following:

sh-4.4$ ls -l /home/jenkins/ssd
total 0
drwxr-xr-x. 3 root root 33 Sep  1 15:24 workspace

The permissions have changed to root:root which obviously aren't allowed with jenkins:jenkins per the -u 1000:1000 as shown above.

I'm aware that volumes can maintain permissions from the primary host (podman0) with something like -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace:U will allow or change the permissions within the container to 1000:1000, but it then causes the permissions outside of the container to change to the subuid:subgid value, which further confuses the Jenkins agent on successive jobs. This possibly is more of a feature request vs. bug, otherwise I'm hoping that someone has a workaround for this.

One possibility would be to detect that "podman" is actually being used and replace the '~~u 1000:1000' with '~~-userns=keep-id'. This appears to create the appropriate permissions within the container and the host as well.

Shane Walton created issue - 2022-09-01 19:50

Shane Walton made changes - 2022-09-01 19:58

Description

Original: Utilizing rootless podman (alias docker) with Jenkins Pipeline has workspace permission issues.

A given agent "Remote root directory" is set to /home/jenkins/ssd The idea is to provide a specific SSD for workspace speed purposes.

{code:java}
jenkins@podman0 ~]$ ls -l /home/jenkins/ssd/
total 1488
drwx------. 3 jenkins jenkins 26 Aug 29 16:55 caches
drwx------. 4 jenkins jenkins 34 Aug 29 15:57 remoting
-rw-------. 1 jenkins jenkins 1522481 Aug 29 15:57 remoting.jar
drwxr-xr-x. 2 jenkins jenkins 6 Aug 31 15:53 workspace
{code}

Below is the log of the permission issues that occurs.
{code:java}
Changelog calculated successfully.
Checkout done
[Pipeline] withEnv
[Pipeline] {
[Pipeline] isUnix
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
+ docker inspect -f . registry.internal.mycomp.com/rhel:72
.
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] withDockerContainer
podman0 does not seem to be running inside a container
$ docker run -t -d -u 1000:1000 -w /home/jenkins/ssd/workspace/TEST_PIPELINE -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace/TEST_PIPELINE:rw,z -v /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** registry.internal.mycomp.com/rhel:72 cat
$ docker top 26dac8250b54021d18ad6006d3c42747eb2af447a993fff59621d0b6053f5dcb -eo pid,comm
[Pipeline] {
[Pipeline] sh
sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-log.txt: Permission denied
sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt.tmp: Permission denied
mv: failed to access '/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt': Permission denied{code}
It is understood that Jenkins Pipeline is meant to work with Docker, but I think some small changes could allow it to work with podman's backward compatibility (ln -s podman docker) to work. The root of the problem is permissions when a volume is mapped inside the container, I see the following:

{code:java}
sh-4.4$ ls -l /home/jenkins/ssd
total 0
drwxr-xr-x. 3 root root 33 Sep 1 15:24 workspace
{code}

The permissions have changed to root:root which obviously aren't allowed with jenkins:jenkins per the -u 1000:1000 as shown above.

I'm aware that volumes can maintain permissions from the primary host (podman0) with something like -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace:U will allow or change the permissions within the container to 1000:1000, but it then causes the permissions outside of the container to change to the subuid:subgid value, which further confuses the Jenkins agent on successive jobs. This possibly is more of a feature request vs. bug, otherwise I'm hoping that someone has a workaround for this.

New: Utilizing rootless podman (alias docker) with Jenkins Pipeline has workspace permission issues.

A given agent "Remote root directory" is set to /home/jenkins/ssd The idea is to provide a specific SSD for workspace speed purposes.

{code:java}
jenkins@podman0 ~]$ ls -l /home/jenkins/ssd/
total 1488
drwx------. 3 jenkins jenkins 26 Aug 29 16:55 caches
drwx------. 4 jenkins jenkins 34 Aug 29 15:57 remoting
-rw-------. 1 jenkins jenkins 1522481 Aug 29 15:57 remoting.jar
drwxr-xr-x. 2 jenkins jenkins 6 Aug 31 15:53 workspace
{code}

Below is the log of the permission issues that occurs.
{code:java}
Changelog calculated successfully.
Checkout done
[Pipeline] withEnv
[Pipeline] {
[Pipeline] isUnix
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
+ docker inspect -f . registry.internal.mycomp.com/rhel:72
.
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] withDockerContainer
podman0 does not seem to be running inside a container
$ docker run -t -d -u 1000:1000 -w /home/jenkins/ssd/workspace/TEST_PIPELINE -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace/TEST_PIPELINE:rw,z -v /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** registry.internal.mycomp.com/rhel:72 cat
$ docker top 26dac8250b54021d18ad6006d3c42747eb2af447a993fff59621d0b6053f5dcb -eo pid,comm
[Pipeline] {
[Pipeline] sh
sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-log.txt: Permission denied
sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt.tmp: Permission denied
mv: failed to access '/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt': Permission denied{code}
It is understood that Jenkins Pipeline is meant to work with Docker, but I think some small changes could allow it to work with podman's backward compatibility (ln -s podman docker) to work. The root of the problem is permissions when a volume is mapped inside the container, I see the following:

{code:java}
sh-4.4$ ls -l /home/jenkins/ssd
total 0
drwxr-xr-x. 3 root root 33 Sep 1 15:24 workspace
{code}

The permissions have changed to root:root which obviously aren't allowed with jenkins:jenkins per the -u 1000:1000 as shown above.

I'm aware that volumes can maintain permissions from the primary host (podman0) with something like -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace:U will allow or change the permissions within the container to 1000:1000, but it then causes the permissions outside of the container to change to the subuid:subgid value, which further confuses the Jenkins agent on successive jobs. This possibly is more of a feature request vs. bug, otherwise I'm hoping that someone has a workaround for this.

One possibility would be to detect that "podman" is actually being used and replace the '-u 1000:1000' with '

--userns=keep-id'. This appears to create the appropriate permissions within the container and the host as well.

Shane Walton made changes - 2022-09-01 20:25

Description

Original: Utilizing rootless podman (alias docker) with Jenkins Pipeline has workspace permission issues.

A given agent "Remote root directory" is set to /home/jenkins/ssd The idea is to provide a specific SSD for workspace speed purposes.

{code:java}
jenkins@podman0 ~]$ ls -l /home/jenkins/ssd/
total 1488
drwx------. 3 jenkins jenkins 26 Aug 29 16:55 caches
drwx------. 4 jenkins jenkins 34 Aug 29 15:57 remoting
-rw-------. 1 jenkins jenkins 1522481 Aug 29 15:57 remoting.jar
drwxr-xr-x. 2 jenkins jenkins 6 Aug 31 15:53 workspace
{code}

Below is the log of the permission issues that occurs.
{code:java}
Changelog calculated successfully.
Checkout done
[Pipeline] withEnv
[Pipeline] {
[Pipeline] isUnix
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
+ docker inspect -f . registry.internal.mycomp.com/rhel:72
.
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] withDockerContainer
podman0 does not seem to be running inside a container
$ docker run -t -d -u 1000:1000 -w /home/jenkins/ssd/workspace/TEST_PIPELINE -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace/TEST_PIPELINE:rw,z -v /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** registry.internal.mycomp.com/rhel:72 cat
$ docker top 26dac8250b54021d18ad6006d3c42747eb2af447a993fff59621d0b6053f5dcb -eo pid,comm
[Pipeline] {
[Pipeline] sh
sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-log.txt: Permission denied
sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt.tmp: Permission denied
mv: failed to access '/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt': Permission denied{code}
It is understood that Jenkins Pipeline is meant to work with Docker, but I think some small changes could allow it to work with podman's backward compatibility (ln -s podman docker) to work. The root of the problem is permissions when a volume is mapped inside the container, I see the following:

{code:java}
sh-4.4$ ls -l /home/jenkins/ssd
total 0
drwxr-xr-x. 3 root root 33 Sep 1 15:24 workspace
{code}

The permissions have changed to root:root which obviously aren't allowed with jenkins:jenkins per the -u 1000:1000 as shown above.

I'm aware that volumes can maintain permissions from the primary host (podman0) with something like -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace:U will allow or change the permissions within the container to 1000:1000, but it then causes the permissions outside of the container to change to the subuid:subgid value, which further confuses the Jenkins agent on successive jobs. This possibly is more of a feature request vs. bug, otherwise I'm hoping that someone has a workaround for this.

One possibility would be to detect that "podman" is actually being used and replace the '-u 1000:1000' with '

--userns=keep-id'. This appears to create the appropriate permissions within the container and the host as well.

New: Utilizing rootless podman (alias docker) with Jenkins Pipeline has workspace permission issues.

A given agent "Remote root directory" is set to /home/jenkins/ssd The idea is to provide a specific SSD for workspace speed purposes.

{code:java}
jenkins@podman0 ~]$ ls -l /home/jenkins/ssd/
total 1488
drwx------. 3 jenkins jenkins 26 Aug 29 16:55 caches
drwx------. 4 jenkins jenkins 34 Aug 29 15:57 remoting
-rw-------. 1 jenkins jenkins 1522481 Aug 29 15:57 remoting.jar
drwxr-xr-x. 2 jenkins jenkins 6 Aug 31 15:53 workspace
{code}

Below is the log of the permission issues that occurs.
{code:java}
Changelog calculated successfully.
Checkout done
[Pipeline] withEnv
[Pipeline] {
[Pipeline] isUnix
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
+ docker inspect -f . registry.internal.mycomp.com/rhel:72
.
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] withDockerContainer
podman0 does not seem to be running inside a container
$ docker run -t -d -u 1000:1000 -w /home/jenkins/ssd/workspace/TEST_PIPELINE -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace/TEST_PIPELINE:rw,z -v /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** registry.internal.mycomp.com/rhel:72 cat
$ docker top 26dac8250b54021d18ad6006d3c42747eb2af447a993fff59621d0b6053f5dcb -eo pid,comm
[Pipeline] {
[Pipeline] sh
sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-log.txt: Permission denied
sh: /home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt.tmp: Permission denied
mv: failed to access '/home/jenkins/ssd/workspace/TEST_PIPELINE@tmp/durable-8dea4d1d/jenkins-result.txt': Permission denied{code}
It is understood that Jenkins Pipeline is meant to work with Docker, but I think some small changes could allow it to work with podman's backward compatibility (ln -s podman docker) to work. The root of the problem is permissions when a volume is mapped inside the container, I see the following:

{code:java}
sh-4.4$ ls -l /home/jenkins/ssd
total 0
drwxr-xr-x. 3 root root 33 Sep 1 15:24 workspace
{code}

The permissions have changed to root:root which obviously aren't allowed with jenkins:jenkins per the -u 1000:1000 as shown above.

I'm aware that volumes can maintain permissions from the primary host (podman0) with something like -v /home/jenkins/ssd/workspace/TEST_PIPELINE:/home/jenkins/ssd/workspace:U will allow or change the permissions within the container to 1000:1000, but it then causes the permissions outside of the container to change to the subuid:subgid value, which further confuses the Jenkins agent on successive jobs. This possibly is more of a feature request vs. bug, otherwise I'm hoping that someone has a workaround for this.

One possibility would be to detect that "podman" is actually being used and replace the '-u 1000:1000' with '--userns=keep-id'. This appears to create the appropriate permissions within the container and the host as well.

Shane Walton made changes - 2022-09-02 14:12

Labels

Original: docker jenkins pipeline plugin podman

New: docker jenkins jenkins-plugin pipeline plugin podman

Jenkins

Details

Description

Attachments

Activity

People

Dates