Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60054

Wrong umask? Containers with non-root user don't have permissions to agent durable script folders

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • kubernetes-plugin
    • None
    • Jenkins: 2.190.2
      kubernetes-plugin: 1.21.1
      jnlp slave: jenkins/jnlp-slave:3.35-5-alpine

      This setup used to work in older versions, I suspect regression happened somewhere around changing working directory to /home/jenkins/agent.

      When running container with non-root user (of course having fsGroup/supplementalGroups to 1000 to match jnlp slave GUID) getting the following error:

       process apparently never started in /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-79996fba
       (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
      

      Doing that reveals:

       LAUNCHER: open /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-cc0559df/jenkins-log.txt: permission denied
      

      I know there's a lot of similar issues being reported recently, but this one is different and not related to the workingDir in jnlp slave. I am using jenkins/jnlp-slave:3.35-5-alpine image and have workingDir correctly defaulting to /home/jenkins/agent.

      I've seen mainly two suggested workarounds to the issue - change user/group in the container and use runAsUser in pod definition. None of this is relevant. To build a flexible CICD you want users to have an ability to specify a dockerhub image (well, at least a tag to whitelisted images) so change of image is not viable. Adding runAsUser: 1000 might work in some limited cases but in reality not viable either. A lot of stuff (let's start with git client) will complain the user for that UID doesn't exist and crash.

      Just to summarize, expected behaviour would be: as long as

       securityContext:
         fsGroup: 1000
         supplementalGroups:
         - 1000
      

      is set on the pod, using containers with non-root and non-/home/jenkins home should work just fine like they used to.

      I found another workaround, which is applying this on jnlp container:

       command:
       - /bin/bash
       args:
       - "-c"
       - "umask 002; jenkins-agent"
      

      But I would imagine it wouldn't be secure? Or is it not?

          [JENKINS-60054] Wrong umask? Containers with non-root user don't have permissions to agent durable script folders

          Dee Kryvenko added a comment -

          I wanted to bump this issue. While original issue described above can easily worked around by setting umask in a jnlp container command/args before starting `jenkins-agent` - there are deeper issues down the road.

          It is kind of a common pattern, a build pod with multiple containers are sharing data using a volume, WORKSPACE being one of them. Problem is - different containers will use different uid/gid, and they cannot be changed when using off the shelf java/go/python/ruby/etc images. Suggesting to re-build in-house versions of java image version X and java image version Y and java image version X and then do the same with go, ruby, python, terraform etc, and do it continually - would be counter productive. `runAsUser` and `runAsGroup` also do not work as many common commands and software fails if the current uid/gid does not belong to a real user with a shell and a home directory set to it. So, a common pattern to work around that is to set `fsGroup` and `supplementalGroups` to a matching value in a pod security context.

          Sadly, many docker images are using `umask 022` by default, which means files that are created by one container with user A - can be read but not modified by another container with a user B. To make the matters worse - steps that are done by Jenkins from jnlp container, which is non-root, can't even do anything to files that are created by another container which happens to be using root user.

          One could argue that a user can set umask in their build steps. It is not always possible if it is not a `sh` step and implementation is dictated by another plugin.

          A solution would be simple - when kubernetes plugin exec into a container, set umask to 002 before running actual commands.

          Dee Kryvenko added a comment - I wanted to bump this issue. While original issue described above can easily worked around by setting umask in a jnlp container command/args before starting `jenkins-agent` - there are deeper issues down the road. It is kind of a common pattern, a build pod with multiple containers are sharing data using a volume, WORKSPACE being one of them. Problem is - different containers will use different uid/gid, and they cannot be changed when using off the shelf java/go/python/ruby/etc images. Suggesting to re-build in-house versions of java image version X and java image version Y and java image version X and then do the same with go, ruby, python, terraform etc, and do it continually - would be counter productive. `runAsUser` and `runAsGroup` also do not work as many common commands and software fails if the current uid/gid does not belong to a real user with a shell and a home directory set to it. So, a common pattern to work around that is to set `fsGroup` and `supplementalGroups` to a matching value in a pod security context. Sadly, many docker images are using `umask 022` by default, which means files that are created by one container with user A - can be read but not modified by another container with a user B. To make the matters worse - steps that are done by Jenkins from jnlp container, which is non-root, can't even do anything to files that are created by another container which happens to be using root user. One could argue that a user can set umask in their build steps. It is not always possible if it is not a `sh` step and implementation is dictated by another plugin. A solution would be simple - when kubernetes plugin exec into a container, set umask to 002 before running actual commands.

          Dee Kryvenko added a comment -

          Dee Kryvenko added a comment - Proposed PR: https://github.com/jenkinsci/kubernetes-plugin/pull/1500

            Unassigned Unassigned
            llibicpep Dee Kryvenko
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: