Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64954

Docker-in-docker connection misuse

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Component/s: docker-workflow-plugin
    • Labels:
      None
    • Environment:
      Jenkins: 2.190.3
      docker-workflow-plugin: docker-workflow-1.26-1-gb174d46

      (Running via `mvn hpi:run` from docker-workflow-plugin revision gb174d46)
    • Similar Issues:

      Description

      We are trying to start a docker daemon inside a docker container from a Jenkinsfile and selectivelly use that daemon for certain operations.
      However it seems that Jenkins gets confused about which docker daemon to contact, failing the build.

      Reproduction steps

      Tested with docker-workflow-plugin docker-workflow-1.26-1-gb174d46 (current git version).

      Jenkinsfile

      node {
          docker.image("docker:dind").withRun("--privileged", "dockerd --tls=false --host 0.0.0.0") { dockerContainer ->
              docker.image("docker:dind").inside("--link ${dockerContainer.id}:docker --entrypoint=''") {
                  docker.withServer("tcp://docker") {
                      sh "echo with nested docker"
                  }
              }
          }
      }
      

      Job output

      Started by user unknown or anonymous
      Running in Durability level: MAX_SURVIVABILITY
      [Pipeline] Start of Pipeline
      [Pipeline] node
      Running on Jenkins in ...
      [Pipeline] {
      [Pipeline] isUnix
      [Pipeline] sh
      + docker run -d --privileged docker:dind dockerd --tls=false --host 0.0.0.0
      [Pipeline] isUnix
      [Pipeline] sh
      + docker inspect -f . docker:dind
      .
      [Pipeline] withDockerContainer
      Jenkins does not seem to be running inside a container
      $ docker run -t -d -u 62407:100 --link 983fa1038732d92d6490e70a13d02b81287bf737eecce485efd48d356c2ac593:docker --entrypoint= -w ... -v ...:rw,z -v ...:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** docker:dind cat
      $ docker top 42a5f9bbb303b7c37736f712fe6d24ca6f7d1b247aa0c89f5fb2cfc8087a3e56 -eo pid,comm
      [Pipeline] {
      [Pipeline] withDockerServer
      [Pipeline] {
      [Pipeline] sh
      process apparently never started in .../workspace/dind@tmp/durable-1c014c22
      (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
      [Pipeline] }
      [Pipeline] // withDockerServer
      [Pipeline] }
      $ docker stop --time=1 42a5f9bbb303b7c37736f712fe6d24ca6f7d1b247aa0c89f5fb2cfc8087a3e56
      $ docker rm -f 42a5f9bbb303b7c37736f712fe6d24ca6f7d1b247aa0c89f5fb2cfc8087a3e56
      [Pipeline] // withDockerContainer
      [Pipeline] isUnix
      [Pipeline] sh
      + docker stop 983fa1038732d92d6490e70a13d02b81287bf737eecce485efd48d356c2ac593
      983fa1038732d92d6490e70a13d02b81287bf737eecce485efd48d356c2ac593
      + docker rm -f 983fa1038732d92d6490e70a13d02b81287bf737eecce485efd48d356c2ac593
      983fa1038732d92d6490e70a13d02b81287bf737eecce485efd48d356c2ac593
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      ERROR: script returned exit code -2
      Finished: FAILURE
      

      Started containers

      CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
      42a5f9bbb303        docker:dind         "cat"                    5 seconds ago       Up 4 seconds        2375-2376/tcp       epic_cori
      983fa1038732        docker:dind         "dockerd-entrypoint.…"   6 seconds ago       Up 5 seconds        2375-2376/tcp       fervent_heyrovsky
      

      Command executed by DurableTask:

      # heavily trimmed down
      DOCKER_HOST=tcp://docker docker exec --env DOCKER_HOST=tcp://docker 42a5f9bbb303b7c $COMMAND
      

      Analysis

      The execution of commands is intercepted by WithContainerStep through org.jenkinsci.plugins.docker.workflow.WithContainerStep.Decorator.
      While this LauncherDecorator explicitly keeps track of the container id to launch the command in, it uses the DOCKER_HOST variable from the environment variables that are active at the time of the execution of the step instead of the one that was used during the creation of the container.

      In this example the "docker exec" invocation itself should not have the environment variable `DOCKER_HOST` set to `tcp://docker`, while it should be passed to the command started inside the container.

      I propose (and offer to implement) to keep explicitly track of the docker connection used to create a container and use it every time this container is to be interacted with.

        Attachments

          Activity

          Show
          t8ch Thomas Weißschuh added a comment - Pullrequest: https://github.com/jenkinsci/docker-workflow-plugin/pull/239

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            t8ch Thomas Weißschuh
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated: