Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64608

Detection "running inside container" fails with cgroup namespace "private" for docker daemon

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Minor
    • Resolution: Fixed
    • docker-workflow-plugin
    • jenkins version 2.263.1
      docker-workflow version 1.25
      docker 20.10.2
      arch linux with kernel 5.4.86-1-lts (/proc/cmdline contains systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all)
      cgroups v2 enabled
      cgroups v1 disabled

    Description

      With cgroups v2 running on the host there is a change to the behavior of the docker daemon regarding cgroups:

      The default cgroup namespace mode (docker run --cgroupns) is "private" on v2, "host" on v1. (cf. https://github.com/docker/docker.github.io/blob/master/config/containers/runmetrics.md#running-docker-on-cgroup-v2).

      Without any changes to the docker configuration the coding in src/main/java/org/jenkinsci/plugins/docker/workflow/client/ControlGroup.java of the docker-workflow-plugin fails, since /proc/self/cgroup does no longer contain the host-cgroup for the container.

      A workaround is to set the option default-cgroupns-mode to "host" for the docker daemon.

      Another way of detecting the containerization is needed.

      Attachments

        Issue Links

          Activity

            mus65 m t added a comment -

            See also my comment on a related issue here: JENKINS-65602 Durable task pipeline failed at sh initialisation - Jenkins Jira

            mus65 m t added a comment - See also my comment on a related issue here:  JENKINS-65602 Durable task pipeline failed at sh initialisation - Jenkins Jira
            gmasil Simon added a comment - - edited

            After upgrading to Debian 11 I was facing the same problem and it took so much time to even figure out what is going on.

            I wanted to suggest to add an environment variable e.g. JENKINS_RUNNING_IN_CONTAINER to directly tell Jenkins if it is running in a container or not. The code parsing the cgroup output was changed several times to support a wide variety of distributions. The docker-workflow-plugin could easily check if it is running in a container by evaluating the environment and get its container ID from /etc/hostname.

            The environment variable could be set in the Dockerfile of Jenkins, so no configuration change (in compose, kubernetes etc.) would be required.

             

            Edit:

            I opened a PR proposing my idea here:

            https://github.com/jenkinsci/docker-workflow-plugin/pull/241

            gmasil Simon added a comment - - edited After upgrading to Debian 11 I was facing the same problem and it took so much time to even figure out what is going on. I wanted to suggest to add an environment variable e.g. JENKINS_RUNNING_IN_CONTAINER to directly tell Jenkins if it is running in a container or not. The code parsing the cgroup output was changed several times to support a wide variety of distributions. The docker-workflow-plugin could easily check if it is running in a container by evaluating the environment and get its container ID from /etc/hostname. The environment variable could be set in the Dockerfile of Jenkins, so no configuration change (in compose, kubernetes etc.) would be required.   Edit: I opened a PR proposing my idea here: https://github.com/jenkinsci/docker-workflow-plugin/pull/241

            I got bit by same issue after upgrading to Debian 11. It took me running separate installations and comparing everything between deb 10 and 11 to notice difference in console output:

             

            [Pipeline] withDockerContainer
            jenkins_agent_00 seems to be running inside container abc123
            
            VS
            
            [Pipeline] withDockerContainer
            jenkins_agent_00 does not seem to be running inside a container
            

            That led me to https://github.com/jenkinsci/docker-workflow-plugin/blob/master/src/main/java/org/jenkinsci/plugins/docker/workflow/client/DockerClient.java and method 
            getContainerIdIfContainerized.
             

            Is there any feedback on this issue? Pull request in comment above looks like it would fix this issue

            ivmilicevic Ivan Milicevic added a comment - I got bit by same issue after upgrading to Debian 11. It took me running separate installations and comparing everything between deb 10 and 11 to notice difference in console output:   [Pipeline] withDockerContainer jenkins_agent_00 seems to be running inside container abc123 VS [Pipeline] withDockerContainer jenkins_agent_00 does not seem to be running inside a container That led me to https://github.com/jenkinsci/docker-workflow-plugin/blob/master/src/main/java/org/jenkinsci/plugins/docker/workflow/client/DockerClient.java  and method  getContainerIdIfContainerized.   Is there any feedback on this issue? Pull request in comment above looks like it would fix this issue
            sebastianhjelm Sebastian Hjelm added a comment - - edited

            This issue became a real problem in Ubuntu 22.04 and the latest patch of 21.10 where cgroup v2 is used by default! I spent way to much time searching for a solution before I came accross the --cgroupns host parameter.

            For now this is an acceptable solution for me but it should be documented on the plugin page that the parameter is a requirement! 

            sebastianhjelm Sebastian Hjelm added a comment - - edited This issue became a real problem in Ubuntu 22.04 and the latest patch of 21.10 where cgroup v2 is used by default! I spent way to much time searching for a solution before I came accross the --cgroupns host parameter. For now this is an acceptable solution for me but it should be documented on the plugin page that the parameter is a requirement! 
            jglick Jesse Glick added a comment - Newer: https://github.com/jenkinsci/docker-workflow-plugin/pull/276

            People

              jglick Jesse Glick
              joergschray Jörg
              Votes:
              4 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: