Details
-
Bug
-
Status: Resolved (View Workflow)
-
Minor
-
Resolution: Fixed
-
jenkins version 2.263.1
docker-workflow version 1.25
docker 20.10.2
arch linux with kernel 5.4.86-1-lts (/proc/cmdline contains systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all)
cgroups v2 enabled
cgroups v1 disabled
Description
With cgroups v2 running on the host there is a change to the behavior of the docker daemon regarding cgroups:
The default cgroup namespace mode (docker run --cgroupns) is "private" on v2, "host" on v1. (cf. https://github.com/docker/docker.github.io/blob/master/config/containers/runmetrics.md#running-docker-on-cgroup-v2).
Without any changes to the docker configuration the coding in src/main/java/org/jenkinsci/plugins/docker/workflow/client/ControlGroup.java of the docker-workflow-plugin fails, since /proc/self/cgroup does no longer contain the host-cgroup for the container.
A workaround is to set the option default-cgroupns-mode to "host" for the docker daemon.
Another way of detecting the containerization is needed.
Attachments
Issue Links
- is related to
-
-
- Resolved
-
- relates to
-
-
- Open
-
- links to
Activity
After upgrading to Debian 11 I was facing the same problem and it took so much time to even figure out what is going on.
I wanted to suggest to add an environment variable e.g. JENKINS_RUNNING_IN_CONTAINER to directly tell Jenkins if it is running in a container or not. The code parsing the cgroup output was changed several times to support a wide variety of distributions. The docker-workflow-plugin could easily check if it is running in a container by evaluating the environment and get its container ID from /etc/hostname.
The environment variable could be set in the Dockerfile of Jenkins, so no configuration change (in compose, kubernetes etc.) would be required.
Edit:
I opened a PR proposing my idea here:
https://github.com/jenkinsci/docker-workflow-plugin/pull/241
Ivan Milicevic
added a comment - I got bit by same issue after upgrading to Debian 11. It took me running separate installations and comparing everything between deb 10 and 11 to notice difference in console output:
[Pipeline] withDockerContainer jenkins_agent_00 seems to be running inside container abc123 VS [Pipeline] withDockerContainer jenkins_agent_00 does not seem to be running inside a container
That led me to https://github.com/jenkinsci/docker-workflow-plugin/blob/master/src/main/java/org/jenkinsci/plugins/docker/workflow/client/DockerClient.java and method
getContainerIdIfContainerized.
Is there any feedback on this issue? Pull request in comment above looks like it would fix this issue
Ivan Milicevic
added a comment - I got bit by same issue after upgrading to Debian 11. It took me running separate installations and comparing everything between deb 10 and 11 to notice difference in console output:
[Pipeline] withDockerContainer
jenkins_agent_00 seems to be running inside container abc123
VS
[Pipeline] withDockerContainer
jenkins_agent_00 does not seem to be running inside a container
That led me to https://github.com/jenkinsci/docker-workflow-plugin/blob/master/src/main/java/org/jenkinsci/plugins/docker/workflow/client/DockerClient.java and method
getContainerIdIfContainerized.
Is there any feedback on this issue? Pull request in comment above looks like it would fix this issue This issue became a real problem in Ubuntu 22.04 and the latest patch of 21.10 where cgroup v2 is used by default! I spent way to much time searching for a solution before I came accross the --cgroupns host parameter.
For now this is an acceptable solution for me but it should be documented on the plugin page that the parameter is a requirement!
Sebastian Hjelm
added a comment - - edited This issue became a real problem in Ubuntu 22.04 and the latest patch of 21.10 where cgroup v2 is used by default! I spent way to much time searching for a solution before I came accross the --cgroupns host parameter.
For now this is an acceptable solution for me but it should be documented on the plugin page that the parameter is a requirement! 
See also my comment on a related issue here: JENKINS-65602 Durable task pipeline failed at sh initialisation - Jenkins Jira