Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64608

Detection "running inside container" fails with cgroup namespace "private" for docker daemon

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • docker-workflow-plugin
    • jenkins version 2.263.1
      docker-workflow version 1.25
      docker 20.10.2
      arch linux with kernel 5.4.86-1-lts (/proc/cmdline contains systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all)
      cgroups v2 enabled
      cgroups v1 disabled

      With cgroups v2 running on the host there is a change to the behavior of the docker daemon regarding cgroups:

      The default cgroup namespace mode (docker run --cgroupns) is "private" on v2, "host" on v1. (cf. https://github.com/docker/docker.github.io/blob/master/config/containers/runmetrics.md#running-docker-on-cgroup-v2).

      Without any changes to the docker configuration the coding in src/main/java/org/jenkinsci/plugins/docker/workflow/client/ControlGroup.java of the docker-workflow-plugin fails, since /proc/self/cgroup does no longer contain the host-cgroup for the container.

      A workaround is to set the option default-cgroupns-mode to "host" for the docker daemon.

      Another way of detecting the containerization is needed.

          [JENKINS-64608] Detection "running inside container" fails with cgroup namespace "private" for docker daemon

          m t added a comment -

          See also my comment on a related issue here: JENKINS-65602 Durable task pipeline failed at sh initialisation - Jenkins Jira

          m t added a comment - See also my comment on a related issue here:  JENKINS-65602 Durable task pipeline failed at sh initialisation - Jenkins Jira

          Simon added a comment - - edited

          After upgrading to Debian 11 I was facing the same problem and it took so much time to even figure out what is going on.

          I wanted to suggest to add an environment variable e.g. JENKINS_RUNNING_IN_CONTAINER to directly tell Jenkins if it is running in a container or not. The code parsing the cgroup output was changed several times to support a wide variety of distributions. The docker-workflow-plugin could easily check if it is running in a container by evaluating the environment and get its container ID from /etc/hostname.

          The environment variable could be set in the Dockerfile of Jenkins, so no configuration change (in compose, kubernetes etc.) would be required.

           

          Edit:

          I opened a PR proposing my idea here:

          https://github.com/jenkinsci/docker-workflow-plugin/pull/241

          Simon added a comment - - edited After upgrading to Debian 11 I was facing the same problem and it took so much time to even figure out what is going on. I wanted to suggest to add an environment variable e.g. JENKINS_RUNNING_IN_CONTAINER to directly tell Jenkins if it is running in a container or not. The code parsing the cgroup output was changed several times to support a wide variety of distributions. The docker-workflow-plugin could easily check if it is running in a container by evaluating the environment and get its container ID from /etc/hostname. The environment variable could be set in the Dockerfile of Jenkins, so no configuration change (in compose, kubernetes etc.) would be required.   Edit: I opened a PR proposing my idea here: https://github.com/jenkinsci/docker-workflow-plugin/pull/241

          I got bit by same issue after upgrading to Debian 11. It took me running separate installations and comparing everything between deb 10 and 11 to notice difference in console output:

           

          [Pipeline] withDockerContainer
          jenkins_agent_00 seems to be running inside container abc123
          
          VS
          
          [Pipeline] withDockerContainer
          jenkins_agent_00 does not seem to be running inside a container
          

          That led me to https://github.com/jenkinsci/docker-workflow-plugin/blob/master/src/main/java/org/jenkinsci/plugins/docker/workflow/client/DockerClient.java and method 
          getContainerIdIfContainerized.
           

          Is there any feedback on this issue? Pull request in comment above looks like it would fix this issue

          Ivan Milicevic added a comment - I got bit by same issue after upgrading to Debian 11. It took me running separate installations and comparing everything between deb 10 and 11 to notice difference in console output:   [Pipeline] withDockerContainer jenkins_agent_00 seems to be running inside container abc123 VS [Pipeline] withDockerContainer jenkins_agent_00 does not seem to be running inside a container That led me to https://github.com/jenkinsci/docker-workflow-plugin/blob/master/src/main/java/org/jenkinsci/plugins/docker/workflow/client/DockerClient.java  and method  getContainerIdIfContainerized.   Is there any feedback on this issue? Pull request in comment above looks like it would fix this issue

          Sebastian Hjelm added a comment - - edited

          This issue became a real problem in Ubuntu 22.04 and the latest patch of 21.10 where cgroup v2 is used by default! I spent way to much time searching for a solution before I came accross the --cgroupns host parameter.

          For now this is an acceptable solution for me but it should be documented on the plugin page that the parameter is a requirement! 

          Sebastian Hjelm added a comment - - edited This issue became a real problem in Ubuntu 22.04 and the latest patch of 21.10 where cgroup v2 is used by default! I spent way to much time searching for a solution before I came accross the --cgroupns host parameter. For now this is an acceptable solution for me but it should be documented on the plugin page that the parameter is a requirement! 

          Jesse Glick added a comment -

          Jesse Glick added a comment - Newer: https://github.com/jenkinsci/docker-workflow-plugin/pull/276

            jglick Jesse Glick
            joergschray Jörg
            Votes:
            4 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: