This setup used to work in older versions, I suspect regression happened somewhere around changing working directory to /home/jenkins/agent.

When running container with non-root user (of course having fsGroup/supplementalGroups to 1000 to match jnlp slave GUID) getting the following error:

 process apparently never started in /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-79996fba
 (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)

Doing that reveals:

 LAUNCHER: open /home/jenkins/agent/workspace/ure-xxx-terraform-module_PR-1@tmp/durable-cc0559df/jenkins-log.txt: permission denied

I know there's a lot of similar issues being reported recently, but this one is different and not related to the workingDir in jnlp slave. I am using jenkins/jnlp-slave:3.35-5-alpine image and have workingDir correctly defaulting to /home/jenkins/agent.

I've seen mainly two suggested workarounds to the issue - change user/group in the container and use runAsUser in pod definition. None of this is relevant. To build a flexible CICD you want users to have an ability to specify a dockerhub image (well, at least a tag to whitelisted images) so change of image is not viable. Adding runAsUser: 1000 might work in some limited cases but in reality not viable either. A lot of stuff (let's start with git client) will complain the user for that UID doesn't exist and crash.

Just to summarize, expected behaviour would be: as long as

 securityContext:
   fsGroup: 1000
   supplementalGroups:
   - 1000

is set on the pod, using containers with non-root and non-/home/jenkins home should work just fine like they used to.

I found another workaround, which is applying this on jnlp container:

 command:
 - /bin/bash
 args:
 - "-c"
 - "umask 002; jenkins-agent"

But I would imagine it wouldn't be secure? Or is it not?