[JENKINS-66568] Pipeline withDockerContainer (docker-in-docker) unable to detect running inside container with cgroups v2

Type: Bug
Resolution: Duplicate
Priority: Major
Component/s: docker-workflow-plugin, pipeline
Labels:
None

Similar Issues:
Powered by SuggestiMate

Show

Running the pipeline taken from https://www.nixknight.com/2021/04/jenkins-docker-in-docker/ works fine in case of cgroups v1 but fails for cgroups v2.

pipeline {
  agent {
    docker {
      image "bash:5.1.4"
    }
  }
  stages {
    stage('Donkey Work') {
      steps {
        sh '''
        if grep -sq 'docker' /proc/1/cgroup ; then
          echo -e "I'm running in a docker container."
        fi
        '''
      }
    }
  }
}

Result (cgroups v1)

-> Jenkins seems to be running inside container 3a9...acc

[Pipeline] withDockerContainer
Jenkins seems to be running inside container 3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
$ docker run -t -d -u 1000:1000 -w /var/jenkins_home/workspace/test --volumes-from 3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** bash:5.1.4 cat
$ docker top 4cf8f914963f286da3a9a830ae37b68463e12d822fe9811548a66a4694dd0139 -eo pid,comm
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Donkey Work)
[Pipeline] sh
+ grep -sq docker /proc/1/cgroup
+ echo -e 'I'"'"'m running in a docker container.'
I'm running in a docker container.
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

Result (cgroups v2)

-> Jenkins does not seem to be running inside a container

[Pipeline] withDockerContainer
Jenkins does not seem to be running inside a container
$ docker run -t -d -u 1000:1000 -w /var/jenkins_home/workspace/test -v /var/jenkins_home/workspace/test:/var/jenkins_home/workspace/test:rw,z -v /var/jenkins_home/workspace/test@tmp:/var/jenkins_home/workspace/test@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** bash:5.1.4 cat
$ docker top 3a6ec19ad44c8d4f9390c64f83b26f3a31fe6f5224a9c05ea63ed2e32f431563 -eo pid,comm
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Donkey Work)
[Pipeline] sh
process apparently never started in /var/jenkins_home/workspace/test@tmp/durable-097b92c0
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

Further investigation shows that detecting if the process runs within a docker container is currently not easily feasible for cgroups v2, see: https://stackoverflow.com/questions/67155739/how-to-check-if-process-runs-within-a-docker-container-cgroup-v2-linux-host. However, I was wondering if this is a known issue and if anyone is trying to come up with a solution?

cgroups v1

$ cat /proc/1/cgroup
12:pids:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
11:hugetlb:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
10:freezer:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
9:cpu,cpuacct:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
8:memory:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
7:devices:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
6:perf_event:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
5:cpuset:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
4:net_cls,net_prio:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
3:rdma:/
2:blkio:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
1:name=systemd:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
0::/system.slice/containerd.service

cgroups v2

$ cat /proc/1/cgroup
0::/

is related to

JENKINS-64608 Detection "running inside container" fails with cgroup namespace "private" for docker daemon

Resolved

Christian created issue - 2021-09-06 16:15

Christian made changes - 2021-09-06 16:17

Description

Original: Running the pipeline taken from [https://www.nixknight.com/2021/04/jenkins-docker-in-docker/] works fine in case of cgroups v1 but fails for cgroups v2.
{noformat}
pipeline {
  agent {
    docker {
      image "bash:5.1.4"
    }
  }
  stages {
    stage('Donkey Work') {
      steps {
        sh '''
        if grep -sq 'docker' /proc/1/cgroup ; then
          echo -e "I'm running in a docker container."
        fi
        '''
      }
    }
  }
}
{noformat}
h5. Result (cgroups v1)

-> Jenkins seems to be running inside container 3a9...acc
{noformat}
[Pipeline] withDockerContainer
Jenkins seems to be running inside container 3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
$ docker run -t -d -u 1000:1000 -w /var/jenkins_home/workspace/test --volumes-from 3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** bash:5.1.4 cat
$ docker top 4cf8f914963f286da3a9a830ae37b68463e12d822fe9811548a66a4694dd0139 -eo pid,comm
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Donkey Work)
[Pipeline] sh
+ grep -sq docker /proc/1/cgroup
+ echo -e 'I'"'"'m running in a docker container.'
I'm running in a docker container.
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
{noformat}
h5. Result (cgroups v2)

-> Jenkins does not seem to be running inside a container
{noformat}
[Pipeline] withDockerContainer
Jenkins does not seem to be running inside a container
$ docker run -t -d -u 1000:1000 -w /var/jenkins_home/workspace/test -v /var/jenkins_home/workspace/test:/var/jenkins_home/workspace/test:rw,z -v /var/jenkins_home/workspace/test@tmp:/var/jenkins_home/workspace/test@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** bash:5.1.4 cat
$ docker top 3a6ec19ad44c8d4f9390c64f83b26f3a31fe6f5224a9c05ea63ed2e32f431563 -eo pid,comm
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Donkey Work)
[Pipeline] sh
process apparently never started in /var/jenkins_home/workspace/test@tmp/durable-097b92c0
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
{noformat}
Further investigation shows that detecting if the process runs within a docker container is currently not easily feasible for cgroups v2, see: [https://stackoverflow.com/questions/67155739/how-to-check-if-process-runs-within-a-docker-container-cgroup-v2-linux-host].
h5. cgroups v1
{noformat}
$ cat /proc/1/cgroup
12:pids:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
11:hugetlb:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
10:freezer:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
9:cpu,cpuacct:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
8:memory:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
7:devices:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
6:perf_event:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
5:cpuset:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
4:net_cls,net_prio:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
3:rdma:/
2:blkio:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
1:name=systemd:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
0::/system.slice/containerd.service
{noformat}
h5. cgroups v2
{noformat}
$ cat /proc/1/cgroup
0::/
{noformat}
However, I was wondering if this is a known issue and if anyone is trying to come up with a solution?

New: Running the pipeline taken from [https://www.nixknight.com/2021/04/jenkins-docker-in-docker/] works fine in case of cgroups v1 but fails for cgroups v2.
{noformat}
pipeline {
  agent {
    docker {
      image "bash:5.1.4"
    }
  }
  stages {
    stage('Donkey Work') {
      steps {
        sh '''
        if grep -sq 'docker' /proc/1/cgroup ; then
          echo -e "I'm running in a docker container."
        fi
        '''
      }
    }
  }
}
{noformat}
h5. Result (cgroups v1)

-> Jenkins seems to be running inside container 3a9...acc
{noformat}
[Pipeline] withDockerContainer
Jenkins seems to be running inside container 3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
$ docker run -t -d -u 1000:1000 -w /var/jenkins_home/workspace/test --volumes-from 3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** bash:5.1.4 cat
$ docker top 4cf8f914963f286da3a9a830ae37b68463e12d822fe9811548a66a4694dd0139 -eo pid,comm
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Donkey Work)
[Pipeline] sh
+ grep -sq docker /proc/1/cgroup
+ echo -e 'I'"'"'m running in a docker container.'
I'm running in a docker container.
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
{noformat}
h5. Result (cgroups v2)

-> Jenkins does not seem to be running inside a container
{noformat}
[Pipeline] withDockerContainer
Jenkins does not seem to be running inside a container
$ docker run -t -d -u 1000:1000 -w /var/jenkins_home/workspace/test -v /var/jenkins_home/workspace/test:/var/jenkins_home/workspace/test:rw,z -v /var/jenkins_home/workspace/test@tmp:/var/jenkins_home/workspace/test@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** bash:5.1.4 cat
$ docker top 3a6ec19ad44c8d4f9390c64f83b26f3a31fe6f5224a9c05ea63ed2e32f431563 -eo pid,comm
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Donkey Work)
[Pipeline] sh
process apparently never started in /var/jenkins_home/workspace/test@tmp/durable-097b92c0
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
{noformat}
Further investigation shows that detecting if the process runs within a docker container is currently not easily feasible for cgroups v2, see: [https://stackoverflow.com/questions/67155739/how-to-check-if-process-runs-within-a-docker-container-cgroup-v2-linux-host]. However, I was wondering if this is a known issue and if anyone is trying to come up with a solution?
h5. cgroups v1
{noformat}
$ cat /proc/1/cgroup
12:pids:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
11:hugetlb:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
10:freezer:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
9:cpu,cpuacct:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
8:memory:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
7:devices:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
6:perf_event:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
5:cpuset:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
4:net_cls,net_prio:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
3:rdma:/
2:blkio:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
1:name=systemd:/docker/3a930717d4967175b00e0e42f8756fe6cf508448b75fdd4763a02eb6d913eacc
0::/system.slice/containerd.service
{noformat}
h5. cgroups v2
{noformat}
$ cat /proc/1/cgroup
0::/
{noformat}

Christian made changes - 2021-09-06 16:18

Summary

Original: Pipeline withDockerContainer (docker-in-docker) unable to detect docker with cgroups v2

New: Pipeline withDockerContainer (docker-in-docker) unable to detect running inside container with cgroups v2

Christian made changes - 2021-10-17 09:47

Link

New: This issue is related to ~~JENKINS-64608~~ [ ~~JENKINS-64608~~ ]

Jesse Glick made changes - 2022-10-04 17:52

Resolution		New: Duplicate [ 3 ]
Status	Original: Open [ 1 ]	New: Resolved [ 5 ]

Jenkins

Details

Description

Result (cgroups v1)

Result (cgroups v2)

cgroups v1

cgroups v2

Attachments

Issue Links

Activity

People

Dates