Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56888

Unexpected cgroup syntax when using docker-in-docker slave launched via SSH

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Duplicate
    • Component/s: docker-workflow-plugin
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      In our configuration, we use the Docker-Workflow plugin as a build cloud to dynamically spawn Jenkins slaves (https://plugins.jenkins.io/docker-workflow).

      In some cases, we need a sandboxed docker environment for building images, so the slaves start a docker-in-docker daemon when they're launched. The slaves are connected via SSH using the Docker plugin's normal "Connect with SSH" launch method. The result is that the default `agent` environment is a remote slave connected via SSH that's running in a docker container. The slave has a docker client installed, and the default docker host is the docker-in-docker daemon running within the slave container.

      When using the dockerfile type of agent parameter (https://jenkins.io/doc/book/pipeline/syntax/#agent-parameters), the withDockerContainer step fails with error message:

      java.io.IOException: Unexpected cgroup syntax /docker/a913fc1766a972d99ae545d6ba2508fe797c2df9b6371ff0be629436a30502a4/docker/a913fc1766a972d99ae545d6ba2508fe797c2df9b6371ff0be629436a30502a4/user/jenkins/0
          at org.jenkinsci.plugins.docker.workflow.client.ControlGroup.getContainerId(ControlGroup.java:60)
          at org.jenkinsci.plugins.docker.workflow.client.ControlGroup.getContainerId(ControlGroup.java:45)
          at org.jenkinsci.plugins.docker.workflow.client.ControlGroup.getContainerId(ControlGroup.java:37)
          at org.jenkinsci.plugins.docker.workflow.client.DockerClient.getContainerIdIfContainerized(DockerClient.java:336)
          at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Execution.start(WithContainerStep.java:159)
          at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:268)
          at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:176)
          at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
          at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
          at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
          at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
          at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
          at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/var/jenkins_home/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:134)
          at org.jenkinsci.plugins.docker.workflow.Docker.node(jar:file:/var/jenkins_home/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:66)
          at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/var/jenkins_home/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:122)
          at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.DockerPipelineFromDockerfileScript.runImage(jar:file:/var/jenkins_home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineFromDockerfileScript.groovy:56)
          at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AbstractDockerPipelineScript.configureRegistry(jar:file:/var/jenkins_home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/AbstractDockerPipelineScript.groovy:73)
          at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AbstractDockerPipelineScript.run(jar:file:/var/jenkins_home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/AbstractDockerPipelineScript.groovy:52)
          at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.checkoutAndRun(jar:file:/var/jenkins_home/plugins/pipeline-model-extensions/WEB-INF/lib/pipeline-model-extensions.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/CheckoutScript.groovy:63)
          at ___cps.transform___(Native Method)
          at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
          at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
          at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
          at sun.reflect.GeneratedMethodAccessor110.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
          at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
          at com.cloudbees.groovy.cps.Next.step(Next.java:83)
          at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
          at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
          at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
          at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
          at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
          at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
          at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
          at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:136)
          at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
          at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:182)
          at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
          at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
          at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
          at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
          at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
          at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
          at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
      

       

      This happens with a very simple pipeline configuration with just two files in the git repo:

       

      Jenkinsfile:

      pipeline {
          agent { dockerfile true }
          stages {
              stage('Test 1') {
              steps {
                  sh '''#!/bin/sh -x
                  echo "If we got to this point, the DockerClient was correctly initialized"
                  cat /proc/self/cgroup
                  '''
                  }
              }
          }
      }
      

      Dockerfile:

      FROM busybox
      RUN echo 'Add a dummy layer'
      

      The unexpected "/user/jenkins/0" at the end of the cgroup path comes from the fact that we're logging in to a docker-based slave via SSH. It first started appearing when we upgraded our slave container's base OS from ubuntu:trusty to ubuntu:xenial. I've set up a basic demonstration of the difference in cgroup naming convention here: https://github.com/PeppyHare/demo-docker-workflow-issue

        Attachments

          Issue Links

            Activity

            Hide
            evanbluhm Evan Bluhm added a comment -
            Show
            evanbluhm Evan Bluhm added a comment - I've opened  https://github.com/jenkinsci/docker-workflow-plugin/pull/169  as a potential fix
            Hide
            fraz3alpha Andy Taylor added a comment -

            This bug is causing us a lot of pain - is there any chance someone could review and merge the fix provided by Evan please?!

            Show
            fraz3alpha Andy Taylor added a comment - This bug is causing us a lot of pain - is there any chance someone could review and merge the fix provided by Evan please?!
            Hide
            carroll Carroll Chiou added a comment -
            Show
            carroll Carroll Chiou added a comment - Closed in favor of https://issues.jenkins-ci.org/browse/JENKINS-57129

              People

              Assignee:
              fraz3alpha Andy Taylor
              Reporter:
              evanbluhm Evan Bluhm
              Votes:
              3 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: