Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-66568

Pipeline withDockerContainer (docker-in-docker) unable to detect running inside container with cgroups v2

      Running the pipeline taken from https://www.nixknight.com/2021/04/jenkins-docker-in-docker/ works fine in case of cgroups v1 but fails for cgroups v2.

      pipeline {
        agent {
          docker {
            image "bash:5.1.4"
          }
        }
        stages {
          stage('Donkey Work') {
            steps {
              sh '''
              if grep -sq 'docker' /proc/1/cgroup ; then
                echo -e "I'm running in a docker container."
              fi
              '''
            }
          }
        }
      }
      
      Result (cgroups v1)

      -> Jenkins seems to be running inside container 3a9...acc

      [Pipeline] withDockerContainer
      Jenkins seems to be running inside container 3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
      $ docker run -t -d -u 1000:1000 -w /var/jenkins_home/workspace/test --volumes-from 3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** bash:5.1.4 cat
      $ docker top 4cf8f914963f286da3a9a830ae37b68463e12d822fe9811548a66a4694dd0139 -eo pid,comm
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Donkey Work)
      [Pipeline] sh
      + grep -sq docker /proc/1/cgroup
      + echo -e 'I'"'"'m running in a docker container.'
      I'm running in a docker container.
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      
      Result (cgroups v2)

      -> Jenkins does not seem to be running inside a container

      [Pipeline] withDockerContainer
      Jenkins does not seem to be running inside a container
      $ docker run -t -d -u 1000:1000 -w /var/jenkins_home/workspace/test -v /var/jenkins_home/workspace/test:/var/jenkins_home/workspace/test:rw,z -v /var/jenkins_home/workspace/test@tmp:/var/jenkins_home/workspace/test@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** bash:5.1.4 cat
      $ docker top 3a6ec19ad44c8d4f9390c64f83b26f3a31fe6f5224a9c05ea63ed2e32f431563 -eo pid,comm
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Donkey Work)
      [Pipeline] sh
      process apparently never started in /var/jenkins_home/workspace/test@tmp/durable-097b92c0
      (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      

      Further investigation shows that detecting if the process runs within a docker container is currently not easily feasible for cgroups v2, see: https://stackoverflow.com/questions/67155739/how-to-check-if-process-runs-within-a-docker-container-cgroup-v2-linux-host. However, I was wondering if this is a known issue and if anyone is trying to come up with a solution?

      cgroups v1
      $ cat /proc/1/cgroup
      12:pids:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
      11:hugetlb:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
      10:freezer:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
      9:cpu,cpuacct:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
      8:memory:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
      7:devices:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
      6:perf_event:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
      5:cpuset:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
      4:net_cls,net_prio:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
      3:rdma:/
      2:blkio:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
      1:name=systemd:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
      0::/system.slice/containerd.service
      
      cgroups v2
      $ cat /proc/1/cgroup
      0::/
      

          [JENKINS-66568] Pipeline withDockerContainer (docker-in-docker) unable to detect running inside container with cgroups v2

          We're stuck with the same problem and had to force cgroup v1 after the upgrade to Debian 11.

          It would be helpful if the docker-workflow plugin were to provide some kind of escape hatch to provide the missing information externally: For example an environment variable or a node-local file containing the container id.

          (In our case, the agent's hostname matches the docker container id...)

          Fabian Knittel added a comment - We're stuck with the same problem and had to force cgroup v1 after the upgrade to Debian 11. It would be helpful if the docker-workflow plugin were to provide some kind of escape hatch to provide the missing information externally: For example an environment variable or a node-local file containing the container id. (In our case, the agent's hostname matches the docker container id...)

            Unassigned Unassigned
            chrisss404 Christian
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: