-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins: 2.190.2
kubernetes-plugin: 1.21.1
jnlp slave: jenkins/jnlp-slave:3.35-5-alpine
This setup used to work in older versions, I suspect regression happened somewhere around changing working directory to /home/jenkins/agent.
When running container with non-root user (of course having fsGroup/supplementalGroups to 1000 to match jnlp slave GUID) getting the following error:
process apparently never started in /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-79996fba
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
Doing that reveals:
LAUNCHER: open /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-cc0559df/jenkins-log.txt: permission denied
I know there's a lot of similar issues being reported recently, but this one is different and not related to the workingDir in jnlp slave. I am using jenkins/jnlp-slave:3.35-5-alpine image and have workingDir correctly defaulting to /home/jenkins/agent.
I've seen mainly two suggested workarounds to the issue - change user/group in the container and use runAsUser in pod definition. None of this is relevant. To build a flexible CICD you want users to have an ability to specify a dockerhub image (well, at least a tag to whitelisted images) so change of image is not viable. Adding runAsUser: 1000 might work in some limited cases but in reality not viable either. A lot of stuff (let's start with git client) will complain the user for that UID doesn't exist and crash.
Just to summarize, expected behaviour would be: as long as
securityContext: fsGroup: 1000 supplementalGroups: - 1000
is set on the pod, using containers with non-root and non-/home/jenkins home should work just fine like they used to.
I found another workaround, which is applying this on jnlp container:
command: - /bin/bash args: - "-c" - "umask 002; jenkins-agent"
But I would imagine it wouldn't be secure? Or is it not?
[JENKINS-60054] Wrong umask? Containers with non-root user don't have permissions to agent durable script folders
Dee Kryvenko
created issue -
Dee Kryvenko
made changes -
| Description |
Original:
This setup used to work in older versions, I suspect regression happened somewhere around changing working directory to `/home/jenkins/agent`. When running container with non-root user (of course having `fsGroup`/`supplementalGroups` to `1000` to match jnlp slave GUID) getting the following error: ``` process apparently never started in /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-79996fba (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer) ``` Doing that reveals: ``` LAUNCHER: open /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-cc0559df/jenkins-log.txt: permission denied ``` I know there's a lot of similar issues being reported recently, but this one is different and not related to the `workingDir` in `jnlp` slave. I am using `jenkins/jnlp-slave:3.35-5-alpine` image and have `workingDir` correctly defaulting to `/home/jenkins/agent`. I've seen mainly two suggested workarounds to the issue - change user/group in the container and use `runAsUser` in pod definition. None of this is relevant. To build a flexible CICD you want users to have an ability to specify a dockerhub image (well, at least a tag to whitelisted images) so change of image is not viable. Adding `runAsUser: 1000` might work in some limited cases but in reality not viable either. A lot of stuff (let's start with git client) will complain the user for that UID doesn't exist and crash. Just to summarize, expected behaviour would be: as long as ``` securityContext: fsGroup: 1000 supplementalGroups: - 1000 ``` is set on the pod, using containers with non-root and non-/home/jenkins home should work just fine like they used to. I found another workaround, which is applying this on jnlp container: ``` command: - /bin/bash args: - "-c" - "umask 0002; jenkins-agent" ``` But I would imagine it wouldn't be secure? Or is it not? |
New:
This setup used to work in older versions, I suspect regression happened somewhere around changing working directory to {{/home/jenkins/agent}}. When running container with non-root user (of course having {{fsGroup}}/{{supplementalGroups}} to {{1000}} to match jnlp slave GUID) getting the following error: {{code}} process apparently never started in /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-79996fba (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer) {{code}} Doing that reveals: {{code}} LAUNCHER: open /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-cc0559df/jenkins-log.txt: permission denied {{code}} I know there's a lot of similar issues being reported recently, but this one is different and not related to the {{workingDir}} in {{jnlp}} slave. I am using {{jenkins/jnlp-slave:3.35-5-alpine}} image and have {{workingDir}} correctly defaulting to {{/home/jenkins/agent}}. I've seen mainly two suggested workarounds to the issue - change user/group in the container and use {{runAsUser}} in pod definition. None of this is relevant. To build a flexible CICD you want users to have an ability to specify a dockerhub image (well, at least a tag to whitelisted images) so change of image is not viable. Adding {{runAsUser: 1000}} might work in some limited cases but in reality not viable either. A lot of stuff (let's start with git client) will complain the user for that UID doesn't exist and crash. Just to summarize, expected behaviour would be: as long as {{code}} securityContext: fsGroup: 1000 supplementalGroups: - 1000 {{code}} is set on the pod, using containers with non-root and non-{{/home/jenkins}} home should work just fine like they used to. I found another workaround, which is applying this on jnlp container: {{code}} command: - /bin/bash args: - "-c" - "umask 0002; jenkins-agent" {{code}} But I would imagine it wouldn't be secure? Or is it not? |
Dee Kryvenko
made changes -
| Description |
Original:
This setup used to work in older versions, I suspect regression happened somewhere around changing working directory to {{/home/jenkins/agent}}. When running container with non-root user (of course having {{fsGroup}}/{{supplementalGroups}} to {{1000}} to match jnlp slave GUID) getting the following error: {{code}} process apparently never started in /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-79996fba (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer) {{code}} Doing that reveals: {{code}} LAUNCHER: open /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-cc0559df/jenkins-log.txt: permission denied {{code}} I know there's a lot of similar issues being reported recently, but this one is different and not related to the {{workingDir}} in {{jnlp}} slave. I am using {{jenkins/jnlp-slave:3.35-5-alpine}} image and have {{workingDir}} correctly defaulting to {{/home/jenkins/agent}}. I've seen mainly two suggested workarounds to the issue - change user/group in the container and use {{runAsUser}} in pod definition. None of this is relevant. To build a flexible CICD you want users to have an ability to specify a dockerhub image (well, at least a tag to whitelisted images) so change of image is not viable. Adding {{runAsUser: 1000}} might work in some limited cases but in reality not viable either. A lot of stuff (let's start with git client) will complain the user for that UID doesn't exist and crash. Just to summarize, expected behaviour would be: as long as {{code}} securityContext: fsGroup: 1000 supplementalGroups: - 1000 {{code}} is set on the pod, using containers with non-root and non-{{/home/jenkins}} home should work just fine like they used to. I found another workaround, which is applying this on jnlp container: {{code}} command: - /bin/bash args: - "-c" - "umask 0002; jenkins-agent" {{code}} But I would imagine it wouldn't be secure? Or is it not? |
New:
This setup used to work in older versions, I suspect regression happened somewhere around changing working directory to {{/home/jenkins/agent}}. When running container with non-root user (of course having {{fsGroup}}/{{supplementalGroups}} to {{1000}} to match jnlp slave GUID) getting the following error: {code} process apparently never started in /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-79996fba (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer) {code} Doing that reveals: {code} LAUNCHER: open /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-cc0559df/jenkins-log.txt: permission denied {code} I know there's a lot of similar issues being reported recently, but this one is different and not related to the {{workingDir}} in {{jnlp}} slave. I am using {{jenkins/jnlp-slave:3.35-5-alpine}} image and have {{workingDir}} correctly defaulting to {{/home/jenkins/agent}}. I've seen mainly two suggested workarounds to the issue - change user/group in the container and use {{runAsUser}} in pod definition. None of this is relevant. To build a flexible CICD you want users to have an ability to specify a dockerhub image (well, at least a tag to whitelisted images) so change of image is not viable. Adding {{runAsUser: 1000}} might work in some limited cases but in reality not viable either. A lot of stuff (let's start with git client) will complain the user for that UID doesn't exist and crash. Just to summarize, expected behaviour would be: as long as {code} securityContext: fsGroup: 1000 supplementalGroups: - 1000 {code} is set on the pod, using containers with non-root and non-{{/home/jenkins}} home should work just fine like they used to. I found another workaround, which is applying this on jnlp container: {code} command: - /bin/bash args: - "-c" - "umask 0002; jenkins-agent" {code} But I would imagine it wouldn't be secure? Or is it not? |
Dee Kryvenko
made changes -
| Description |
Original:
This setup used to work in older versions, I suspect regression happened somewhere around changing working directory to {{/home/jenkins/agent}}. When running container with non-root user (of course having {{fsGroup}}/{{supplementalGroups}} to {{1000}} to match jnlp slave GUID) getting the following error: {code} process apparently never started in /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-79996fba (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer) {code} Doing that reveals: {code} LAUNCHER: open /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-cc0559df/jenkins-log.txt: permission denied {code} I know there's a lot of similar issues being reported recently, but this one is different and not related to the {{workingDir}} in {{jnlp}} slave. I am using {{jenkins/jnlp-slave:3.35-5-alpine}} image and have {{workingDir}} correctly defaulting to {{/home/jenkins/agent}}. I've seen mainly two suggested workarounds to the issue - change user/group in the container and use {{runAsUser}} in pod definition. None of this is relevant. To build a flexible CICD you want users to have an ability to specify a dockerhub image (well, at least a tag to whitelisted images) so change of image is not viable. Adding {{runAsUser: 1000}} might work in some limited cases but in reality not viable either. A lot of stuff (let's start with git client) will complain the user for that UID doesn't exist and crash. Just to summarize, expected behaviour would be: as long as {code} securityContext: fsGroup: 1000 supplementalGroups: - 1000 {code} is set on the pod, using containers with non-root and non-{{/home/jenkins}} home should work just fine like they used to. I found another workaround, which is applying this on jnlp container: {code} command: - /bin/bash args: - "-c" - "umask 0002; jenkins-agent" {code} But I would imagine it wouldn't be secure? Or is it not? |
New:
This setup used to work in older versions, I suspect regression happened somewhere around changing working directory to {{/home/jenkins/agent}}. When running container with non-root user (of course having {{fsGroup}}/{{supplementalGroups}} to {{1000}} to match jnlp slave GUID) getting the following error: {code} process apparently never started in /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-79996fba (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer) {code} Doing that reveals: {code} LAUNCHER: open /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-cc0559df/jenkins-log.txt: permission denied {code} I know there's a lot of similar issues being reported recently, but this one is different and not related to the {{workingDir}} in {{jnlp}} slave. I am using {{jenkins/jnlp-slave:3.35-5-alpine}} image and have {{workingDir}} correctly defaulting to {{/home/jenkins/agent}}. I've seen mainly two suggested workarounds to the issue - change user/group in the container and use {{runAsUser}} in pod definition. None of this is relevant. To build a flexible CICD you want users to have an ability to specify a dockerhub image (well, at least a tag to whitelisted images) so change of image is not viable. Adding {{runAsUser: 1000}} might work in some limited cases but in reality not viable either. A lot of stuff (let's start with git client) will complain the user for that UID doesn't exist and crash. Just to summarize, expected behaviour would be: as long as {code} securityContext: fsGroup: 1000 supplementalGroups: - 1000 {code} is set on the pod, using containers with non-root and non-{{/home/jenkins}} home should work just fine like they used to. I found another workaround, which is applying this on jnlp container: {code} command: - /bin/bash args: - "-c" - "umask 002; jenkins-agent" {code} But I would imagine it wouldn't be secure? Or is it not? |
Dee Kryvenko
made changes -
| Description |
Original:
This setup used to work in older versions, I suspect regression happened somewhere around changing working directory to {{/home/jenkins/agent}}. When running container with non-root user (of course having {{fsGroup}}/{{supplementalGroups}} to {{1000}} to match jnlp slave GUID) getting the following error: {code} process apparently never started in /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-79996fba (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer) {code} Doing that reveals: {code} LAUNCHER: open /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-cc0559df/jenkins-log.txt: permission denied {code} I know there's a lot of similar issues being reported recently, but this one is different and not related to the {{workingDir}} in {{jnlp}} slave. I am using {{jenkins/jnlp-slave:3.35-5-alpine}} image and have {{workingDir}} correctly defaulting to {{/home/jenkins/agent}}. I've seen mainly two suggested workarounds to the issue - change user/group in the container and use {{runAsUser}} in pod definition. None of this is relevant. To build a flexible CICD you want users to have an ability to specify a dockerhub image (well, at least a tag to whitelisted images) so change of image is not viable. Adding {{runAsUser: 1000}} might work in some limited cases but in reality not viable either. A lot of stuff (let's start with git client) will complain the user for that UID doesn't exist and crash. Just to summarize, expected behaviour would be: as long as {code} securityContext: fsGroup: 1000 supplementalGroups: - 1000 {code} is set on the pod, using containers with non-root and non-{{/home/jenkins}} home should work just fine like they used to. I found another workaround, which is applying this on jnlp container: {code} command: - /bin/bash args: - "-c" - "umask 002; jenkins-agent" {code} But I would imagine it wouldn't be secure? Or is it not? |
New:
This setup used to work in older versions, I suspect regression happened somewhere around changing working directory to {{/home/jenkins/agent}}. When running container with non-root user (of course having {{fsGroup}}/{{supplementalGroups}} to {{1000}} to match jnlp slave GUID) getting the following error: {code} process apparently never started in /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-79996fba (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer) {code} Doing that reveals: {code} LAUNCHER: open /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-cc0559df/jenkins-log.txt: permission denied {code} I know there's a lot of similar issues being reported recently, but this one is different and not related to the {{workingDir}} in {{jnlp}} slave. I am using {{jenkins/jnlp-slave:3.35-5-alpine}} image and have {{workingDir}} correctly defaulting to {{/home/jenkins/agent}}. I've seen mainly two suggested workarounds to the issue - change user/group in the container and use {{runAsUser}} in pod definition. None of this is relevant. To build a flexible CICD you want users to have an ability to specify a dockerhub image (well, at least a tag to whitelisted images) so change of image is not viable. Adding {{runAsUser: 1000}} might work in some limited cases but in reality not viable either. A lot of stuff (let's start with git client) will complain the user for that UID doesn't exist and crash. Just to summarize, expected behaviour would be: as long as {code} securityContext: fsGroup: 1000 supplementalGroups: - 1000 {code} is set on the pod, using containers with non-root and non-{{/home/jenkins}} home should work just fine like they used to. I found another workaround, which is applying this on jnlp container: {code} command: - /bin/bash args: - "-c" - "umask 002; jenkins-agent" {code} But I would imagine it wouldn't be secure? Or is it not? |
Dee Kryvenko
made changes -
| Priority | Original: Minor [ 4 ] | New: Major [ 3 ] |
Dee Kryvenko
added a comment - Dee Kryvenko
added a comment - Proposed PR: https://github.com/jenkinsci/kubernetes-plugin/pull/1500
I wanted to bump this issue. While original issue described above can easily worked around by setting umask in a jnlp container command/args before starting `jenkins-agent` - there are deeper issues down the road.
It is kind of a common pattern, a build pod with multiple containers are sharing data using a volume, WORKSPACE being one of them. Problem is - different containers will use different uid/gid, and they cannot be changed when using off the shelf java/go/python/ruby/etc images. Suggesting to re-build in-house versions of java image version X and java image version Y and java image version X and then do the same with go, ruby, python, terraform etc, and do it continually - would be counter productive. `runAsUser` and `runAsGroup` also do not work as many common commands and software fails if the current uid/gid does not belong to a real user with a shell and a home directory set to it. So, a common pattern to work around that is to set `fsGroup` and `supplementalGroups` to a matching value in a pod security context.
Sadly, many docker images are using `umask 022` by default, which means files that are created by one container with user A - can be read but not modified by another container with a user B. To make the matters worse - steps that are done by Jenkins from jnlp container, which is non-root, can't even do anything to files that are created by another container which happens to be using root user.
One could argue that a user can set umask in their build steps. It is not always possible if it is not a `sh` step and implementation is dictated by another plugin.
A solution would be simple - when kubernetes plugin exec into a container, set umask to 002 before running actual commands.