-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins: 2.190.2
kubernetes-plugin: 1.21.1
jnlp slave: jenkins/jnlp-slave:3.35-5-alpine
This setup used to work in older versions, I suspect regression happened somewhere around changing working directory to /home/jenkins/agent.
When running container with non-root user (of course having fsGroup/supplementalGroups to 1000 to match jnlp slave GUID) getting the following error:
process apparently never started in /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-79996fba
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
Doing that reveals:
LAUNCHER: open /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-cc0559df/jenkins-log.txt: permission denied
I know there's a lot of similar issues being reported recently, but this one is different and not related to the workingDir in jnlp slave. I am using jenkins/jnlp-slave:3.35-5-alpine image and have workingDir correctly defaulting to /home/jenkins/agent.
I've seen mainly two suggested workarounds to the issue - change user/group in the container and use runAsUser in pod definition. None of this is relevant. To build a flexible CICD you want users to have an ability to specify a dockerhub image (well, at least a tag to whitelisted images) so change of image is not viable. Adding runAsUser: 1000 might work in some limited cases but in reality not viable either. A lot of stuff (let's start with git client) will complain the user for that UID doesn't exist and crash.
Just to summarize, expected behaviour would be: as long as
securityContext: fsGroup: 1000 supplementalGroups: - 1000
is set on the pod, using containers with non-root and non-/home/jenkins home should work just fine like they used to.
I found another workaround, which is applying this on jnlp container:
command: - /bin/bash args: - "-c" - "umask 002; jenkins-agent"
But I would imagine it wouldn't be secure? Or is it not?
I wanted to bump this issue. While original issue described above can easily worked around by setting umask in a jnlp container command/args before starting `jenkins-agent` - there are deeper issues down the road.
It is kind of a common pattern, a build pod with multiple containers are sharing data using a volume, WORKSPACE being one of them. Problem is - different containers will use different uid/gid, and they cannot be changed when using off the shelf java/go/python/ruby/etc images. Suggesting to re-build in-house versions of java image version X and java image version Y and java image version X and then do the same with go, ruby, python, terraform etc, and do it continually - would be counter productive. `runAsUser` and `runAsGroup` also do not work as many common commands and software fails if the current uid/gid does not belong to a real user with a shell and a home directory set to it. So, a common pattern to work around that is to set `fsGroup` and `supplementalGroups` to a matching value in a pod security context.
Sadly, many docker images are using `umask 022` by default, which means files that are created by one container with user A - can be read but not modified by another container with a user B. To make the matters worse - steps that are done by Jenkins from jnlp container, which is non-root, can't even do anything to files that are created by another container which happens to be using root user.
One could argue that a user can set umask in their build steps. It is not always possible if it is not a `sh` step and implementation is dictated by another plugin.
A solution would be simple - when kubernetes plugin exec into a container, set umask to 002 before running actual commands.