Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48502

image.inside stopped working on ECS agents

XMLWordPrintable

      Hi,

      We are running Jenkins with ECS cloud, meaning the agents are docker based agents.

      Some of our pipelines are using image.inside to run arbitrary build tools which are not part of the slave images.

      Before upgrading to amazon-ecs-agent-1.16.0 we got the following message in the log files:

       

      ecs-cloud-33394f9575f0e2 seems to be running inside container 01b572fc3c671c11087fcb8166d686792d2b725e570af941b37ff3f28986762f 

      After the upgrade, we get:

      ecs-cloud-dc46d9e888c4 does not seem to be running inside a container

      Looking at The DockerClient code I see that the way the plugin understands it is running within docker container, is by matching docker cgroups pattern with the default hierarchy name docker.

      The problem is that AWS ECS changed their default hierarchy to be ecs

      root@20f3571647fc:/usr/local/tomcat# cat /proc/self/cgroup
 9:perf_event:/ecs/0410eff2-7e59-4111-823e-1e0d98ef7f30/20f3571647fc2a6c718909b85f2bfd79ffae4f6afe8c8121c8b8415f0619f013
 8:memory:/ecs/0410eff2-7e59-4111-823e-1e0d98ef7f30/20f3571647fc2a6c718909b85f2bfd79ffae4f6afe8c8121c8b8415f0619f013
 7:hugetlb:/ecs/0410eff2-7e59-4111-823e-1e0d98ef7f30/20f3571647fc2a6c718909b85f2bfd79ffae4f6afe8c8121c8b8415f0619f013
...

      Because of that, image.inside step stopped working and we get the following error each build.

      sh: can't create /var/jenkins_home/workspace/...@tmp/durable-9cddf800/jenkins-log.txt: nonexistent directory

       

      I believe a quick fix would be to change the regex to match ecs as well, or add a cgroups hierarchy configuration parameter to the plugin.

      Best regards,

      Ohad

          [JENKINS-48502] image.inside stopped working on ECS agents

          Jonatan Landsberg added a comment -

          For what it's worth I want to confirm this bug. My team has the exact same issue and resolved it by downgrading the ECS agent back to 1.15.2.

          Would really want a better solution as we want to use the latest version of the agent, and eventually will have to upgrade it.

          Jonatan Landsberg added a comment - For what it's worth I want to confirm this bug. My team has the exact same issue and resolved it by downgrading the ECS agent back to 1.15.2. Would really want a better solution as we want to use the latest version of the agent, and eventually will have to upgrade it.

          Jakob Kylberg added a comment -

          I can confirm this as well. Our team ran into this issue and had to downgrade the ECS agent to 1.15.2 in order to get our Jenkins setup to work again. I hope this issue can get prioritized as it is a showstopper that stops us from updating our EC2 instances and ECS agents.

          Jakob Kylberg added a comment - I can confirm this as well. Our team ran into this issue and had to downgrade the ECS agent to 1.15.2 in order to get our Jenkins setup to work again. I hope this issue can get prioritized as it is a showstopper that stops us from updating our EC2 instances and ECS agents.

          Ohad David added a comment - - edited

          While https://issues.jenkins-ci.org/browse/JENKINS-47947 is related to kubernetes and this issue is related to ecs, they share the same pattern.

          Ohad David added a comment - - edited While https://issues.jenkins-ci.org/browse/JENKINS-47947  is related to kubernetes and this issue is related to ecs, they share the same pattern.

          Ohad David added a comment -

          I've created a PR: https://github.com/jenkinsci/docker-workflow-plugin/pull/124

           

          Ohad David added a comment - I've created a PR: https://github.com/jenkinsci/docker-workflow-plugin/pull/124  

            ndeloof Nicolas De Loof
            odavid Ohad David
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: