Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34194

docker volume mounts not working as expected when run from within swarm container

    • Icon: Bug Bug
    • Resolution: Not A Defect
    • Icon: Trivial Trivial
    • Ubuntu 14.04.4, Jenkins 1.642.2 (workflow/pipeline 1.15, docker-workflow 1.2), Jenkins 1.642.4 (workflow/pipeline 2.x, docker-workflow 1.4), swarm 2.0

      It appears that Jenkins is running within a jail under swarm and/or workflow (running in a docker container with /var/lib/jenkins and /var/run/docker.sock bind-mounted from host) which is preventing bind mounts from working as expected from both docker.image().inside and sh workflow steps.

      PLEASE NOTE: this is also happening when simply running docker from `sh` (see script/results below)

      Script, results, and plugins below:

      workflow/pipeline script
      node ('build && docker') {
          wrap([$class: 'TimestamperBuildWrapper']) {
              sh 'hostname'
          
              sh 'docker version'
          
              echo 'DOCKER (as jenkins)'
              sh 'docker run -t -u 1000:1000 -v $(pwd):$(pwd):rw -w $(pwd) buildpack-deps:jessie-scm bash -c \'pwd; touch herp; ls -alF\''
          
              echo 'DOCKER (as root)'
              sh 'docker run -t -u 0:0 -v $(pwd):$(pwd):rw -w $(pwd) buildpack-deps:jessie-scm bash -c \'pwd; touch herp; ls -alF\''
          
              echo 'BASH'
              sh 'pwd; touch derp; ls -alF'
              
              docker.image('buildpack-deps:jessie-scm').inside {
                  sh 'bash -c \'pwd; touch inside; ls -alF\''
              }
          }
      }
      
      workflow/pipeline results
      Started by user Jacob Blain Christen
      [Pipeline] Allocate node : Start
      Running on build-c8c3e683 in /var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-workflow-bug
      [Pipeline] node {
      [Pipeline] General Build Wrapper : Start
      [Pipeline] wrap {
      [Pipeline] sh
      23:44:47 [jenkins-docker-workflow-bug] Running shell script
      23:44:47 + hostname
      23:44:47 8dcf8118ebe6
      [Pipeline] sh
      23:44:47 [jenkins-docker-workflow-bug] Running shell script
      23:44:47 + docker version
      23:44:48 Client:
      23:44:48  Version:      1.10.3
      23:44:48  API version:  1.22
      23:44:48  Go version:   go1.5.3
      23:44:48  Git commit:   20f81dd
      23:44:48  Built:        Thu Mar 10 21:49:11 2016
      23:44:48  OS/Arch:      linux/amd64
      23:44:48 
      23:44:48 Server:
      23:44:48  Version:      1.10.2
      23:44:48  API version:  1.22
      23:44:48  Go version:   go1.5.3
      23:44:48  Git commit:   c3959b1
      23:44:48  Built:        Mon Feb 22 21:37:01 2016
      23:44:48  OS/Arch:      linux/amd64
      [Pipeline] echo
      23:44:48 DOCKER (as jenkins)
      [Pipeline] sh
      23:44:48 [jenkins-docker-workflow-bug] Running shell script
      23:44:49 + pwd
      23:44:49 + pwd
      23:44:49 + pwd
      23:44:49 + docker run -t -u 1000:1000 -v /var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-workflow-bug:/var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-workflow-bug:rw -w /var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-workflow-bug buildpack-deps:jessie-scm bash -c pwd; touch herp; ls -alF
      23:44:49 /var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-workflow-bug
      23:44:49 touch: cannot touch 'herp': Permission denied
      23:44:49 total 8
      23:44:49 drwxr-xr-x 2 root root 4096 Apr 13 06:39 ./
      23:44:49 drwxr-xr-x 3 root root 4096 Apr 13 06:45 ../
      [Pipeline] echo
      23:44:49 DOCKER (as root)
      [Pipeline] sh
      23:44:49 [jenkins-docker-workflow-bug] Running shell script
      23:44:50 + pwd
      23:44:50 + pwd
      23:44:50 + pwd
      23:44:50 + docker run -t -u 0:0 -v /var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-workflow-bug:/var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-workflow-bug:rw -w /var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-workflow-bug buildpack-deps:jessie-scm bash -c pwd; touch herp; ls -alF
      23:44:50 /var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-workflow-bug
      23:44:50 total 8
      23:44:50 drwxr-xr-x 2 root root 4096 Apr 13 06:45 ./
      23:44:50 drwxr-xr-x 3 root root 4096 Apr 13 06:45 ../
      23:44:50 -rw-r--r-- 1 root root    0 Apr 13 06:45 herp
      [Pipeline] echo
      23:44:51 BASH
      [Pipeline] sh
      23:44:51 [jenkins-docker-workflow-bug] Running shell script
      23:44:51 + pwd
      23:44:51 /var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-workflow-bug
      23:44:51 + touch derp
      23:44:51 + ls -alF
      23:44:51 total 8
      23:44:51 drwxr-xr-x  2 jenkins jenkins 4096 Apr 13 06:39 ./
      23:44:51 drwxr-xr-x 13 jenkins jenkins 4096 Apr 13 06:39 ../
      23:44:51 -rw-r--r--  1 jenkins jenkins    0 Apr 13 06:45 derp
      [Pipeline] sh
      23:44:52 [jenkins-docker-workflow-bug] Running shell script
      23:44:52 + docker inspect -f . buildpack-deps:jessie-scm
      23:44:52 .
      [Pipeline] Run build steps inside a Docker container : Start
      23:44:52 $ docker run -t -d -u 1000:1000 -w /var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-workflow-bug -v /var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-workflow-bug:/var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-workflow-bug:rw -v /var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-workflow-bug@tmp:/var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-workflow-bug@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** buildpack-deps:jessie-scm cat
      [Pipeline] withDockerContainer {
      [Pipeline] sh
      23:44:53 [jenkins-docker-workflow-bug] Running shell script
      23:44:53 sh: 1: cannot create /var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-workflow-bug@tmp/durable-aff93f64/pid: Directory nonexistent
      23:44:53 sh: 1: cannot create /var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-workflow-bug@tmp/durable-aff93f64/jenkins-log.txt: Directory nonexistent
      23:44:53 sh: 1: cannot create /var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-workflow-bug@tmp/durable-aff93f64/jenkins-result.txt: Directory nonexistent
      [Pipeline] } //withDockerContainer
      23:45:11 $ docker stop e2b796e31144c833523fef97cff5a21352d05abb2c13453b494dd60dfcd5e95c
      23:45:21 $ docker rm -f e2b796e31144c833523fef97cff5a21352d05abb2c13453b494dd60dfcd5e95c
      [Pipeline] Run build steps inside a Docker container : End
      [Pipeline] } //wrap
      [Pipeline] General Build Wrapper : End
      [Pipeline] } //node
      [Pipeline] Allocate node : End
      [Pipeline] End of Pipeline
      ERROR: script returned exit code -2
      Finished: FAILURE
      
      plugins
      ace-editor	1.1	true	false
      ant	1.2	true	false
      antisamy-markup-formatter	1.1	true	false
      authentication-tokens	1.2	true	true
      branch-api	1.6	true	false
      build-monitor-plugin	1.8+build.201601112328	true	false
      build-name-setter	1.5.1	true	true
      build-with-parameters	1.3	true	false
      cloudbees-bitbucket-branch-source	1.4	true	false
      cloudbees-folder	5.8	true	true
      credentials	1.27	true	true
      credentials-binding	1.7	true	false
      cvs	2.11	true	false
      docker-commons	1.3.1	true	false
      docker-workflow	1.4	true	false
      durable-task	1.9	true	false
      embeddable-build-status	1.9	true	false
      external-monitor-job	1.4	true	false
      git	2.4.4	true	false
      git-client	1.19.6	true	false
      git-server	1.6	true	false
      github	1.18.2	true	false
      github-api	1.72.1	true	false
      github-branch-source	1.5	true	false
      gravatar	2.1	true	false
      greenballs	1.15	true	false
      groovy	1.29	true	false
      groovy-postbuild	2.3.1	true	false
      handlebars	1.1.1	true	false
      hipchat	1.0.0	true	false
      icon-shim	2.0.3	true	true
      javadoc	1.1	true	false
      job-dsl	1.45	true	false
      jquery-detached	1.2.1	true	false
      junit	1.11	true	true
      ldap	1.11	true	true
      mailer	1.16	true	true
      mapdb-api	1.0.6.0	true	false
      matrix-auth	1.3.2	true	true
      matrix-project	1.6	true	true
      maven-plugin	2.12.1	true	true
      mercurial	1.54	true	false
      momentjs	1.1.1	true	false
      pam-auth	1.1	true	false
      parameterized-trigger	2.30	true	false
      pegdown-formatter	1.3	true	true
      pipeline-build-step	2.0	true	false
      pipeline-input-step	2.0	true	false
      pipeline-rest-api	1.3	true	false
      pipeline-stage-step	2.0	true	false
      pipeline-stage-view	1.3	true	false
      pipeline-utility-steps	1.1.4	true	false
      plain-credentials	1.1	true	false
      project-description-setter	1.1	true	false
      rebuild	1.25	true	false
      scm-api	1.2	true	false
      script-security	1.18.1	true	true
      ssh-agent	1.10	true	false
      ssh-credentials	1.11	true	true
      ssh-slaves	1.10	true	true
      structs	1.1	true	false
      subversion	2.5.7	true	true
      swarm	2.0	true	true
      systemloadaverage-monitor	1.2	true	false
      throttle-concurrents	1.9.0	true	false
      timestamper	1.7.4	true	true
      token-macro	1.12.1	true	true
      translation	1.10	true	false
      violations	0.7.11	true	false
      windows-slaves	1.1	true	true
      workflow-aggregator	2.0	true	false
      workflow-api	2.0	true	false
      workflow-basic-steps	2.0	true	false
      workflow-cps	2.1	true	false
      workflow-cps-global-lib	2.0	true	false
      workflow-durable-task-step	2.0	true	false
      workflow-job	2.0	true	false
      workflow-multibranch	2.1	true	false
      workflow-scm-step	2.0	true	false
      workflow-step-api	2.0	true	false
      workflow-support	2.0	true	false
      

          [JENKINS-34194] docker volume mounts not working as expected when run from within swarm container

          Jacob Blain Christen added a comment - - edited

          I believe that this is happening in the hudson.Launcher implementation, gonna try and setup a local workspace and poke around.

          Jacob Blain Christen added a comment - - edited I believe that this is happening in the hudson.Launcher implementation, gonna try and setup a local workspace and poke around.

          Jacob Blain Christen added a comment - - edited

          This is also happening in a Freestyle shell step:

          freestyle shell script
          docker run -t -v $(pwd):$(pwd):rw -w $(pwd) -u $(id -u):$(id -g) buildpack-deps:jessie-scm bash -c 'pwd; touch derpstylze; ls -alF'
          
          freestyle results
          Started by user Jacob Blain Christen
          Building remotely on build-51a6c63e (swarm build docker) in workspace /var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-freestyle-shell
          [jenkins-docker-freestyle-shell] $ /bin/sh -xe /tmp/hudson5400900401427204197.sh
          + pwd
          + pwd
          + pwd
          + id -u
          + id -g
          + docker run -t -v /var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-freestyle-shell:/var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-freestyle-shell:rw -w /var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-freestyle-shell -u 1000:1000 buildpack-deps:jessie-scm bash -c pwd; touch derpstylze; ls -alF
          /var/lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-freestyle-shell
          touch: cannot touch 'derpstylze': Permission denied
          total 8
          drwxr-xr-x 2 root root 4096 Apr 13 07:23 ./
          drwxr-xr-x 3 root root 4096 Apr 13 07:23 ../
          Finished: SUCCESS
          

          Jacob Blain Christen added a comment - - edited This is also happening in a Freestyle shell step: freestyle shell script docker run -t -v $(pwd):$(pwd):rw -w $(pwd) -u $(id -u):$(id -g) buildpack-deps:jessie-scm bash -c 'pwd; touch derpstylze; ls -alF' freestyle results Started by user Jacob Blain Christen Building remotely on build-51a6c63e (swarm build docker) in workspace / var /lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-freestyle-shell [jenkins-docker-freestyle-shell] $ /bin/sh -xe /tmp/hudson5400900401427204197.sh + pwd + pwd + pwd + id -u + id -g + docker run -t -v / var /lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-freestyle-shell:/ var /lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-freestyle-shell:rw -w / var /lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-freestyle-shell -u 1000:1000 buildpack-deps:jessie-scm bash -c pwd; touch derpstylze; ls -alF / var /lib/jenkins/workspace/USERS/JCHRISTEN/jenkins-docker-freestyle-shell touch: cannot touch 'derpstylze' : Permission denied total 8 drwxr-xr-x 2 root root 4096 Apr 13 07:23 ./ drwxr-xr-x 3 root root 4096 Apr 13 07:23 ../ Finished: SUCCESS

          Jacob Blain Christen added a comment - - edited
          slave Dockerfile
          FROM java:8-jdk
          
          MAINTAINER Jacob Blain Christen <mailto:dweomer5@gmail.com, https://github.com/dweomer, https://twitter.com/dweomer>
          
          ARG DOCKER_VERSION=1.10.3
          ARG GOSU_VERSION=1.7
          ARG TINI_VERSION=0.8.4
          
          LABEL   "ARG:DOCKER_VERSION"="${DOCKER_VERSION}" \
                  "ARG:GOSU_VERSION"="${GOSU_VERSION}" \
                  "ARG:TINI_VERSION"="${TINI_VERSION}"
          
          ENV JENKINS_GROUP=jenkins \
              JENKINS_HOME=/var/lib/jenkins \
              JENKINS_USER=jenkins
          
          RUN set -x \
           && export DEBIAN_FRONTEND=noninteractive \
           && apt-get update \
          ### Install netstat to allow connection health check with `netstat -tan | grep ESTABLISHED`
           && apt-get install -y \
                  curl \
                  git \
                  net-tools \
                  wget \
          ### Install Docker
           && curl -fsSL --create-dirs --output /usr/local/bin/docker \
                  "https://get.docker.com/builds/$(uname -s)/$(uname -m)/docker-${DOCKER_VERSION}" \
           && chmod -v +x /usr/local/bin/docker \
          ### Install Gosu
           && curl -fsSL --create-dirs --output /usr/local/bin/gosu \
                  "https://github.com/tianon/gosu/releases/download/${GOSU_VERSION}/gosu-$(dpkg --print-architecture)" \
           && chmod -v +x /usr/local/bin/gosu \
          ### Install Tini
           && curl --create-dirs -sSLo /usr/local/bin/tini \
                  "https://github.com/krallin/tini/releases/download/v${TINI_VERSION}/tini-static" \
           && chmod -v +x /usr/local/bin/tini \
          ### Create Jenkins User
           && groupadd ${JENKINS_GROUP} \
           && useradd -c "Jenkins Swarm Agent" -d ${JENKINS_HOME} -g ${JENKINS_GROUP} -m ${JENKINS_USER} \
          ### Cleanup
           && apt-get clean \
           && rm -rf \
                  /tmp/* \
                  /var/lib/apt/lists/* \
                  /var/tmp/*
          
          COPY rootfs /
          ENTRYPOINT ["/jenkins-swarm-slave.sh"]
          
          RUN set -x \
           && chmod -v +x /jenkins-swarm-slave.sh \
           && find /jenkins-swarm-slave* -type f
          
          WORKDIR ${JENKINS_HOME}
          VOLUME ${JENKINS_HOME}
          
          slave docker-compose.yml (snippet)
          version: '2'
          
          networks:
            jenkins:
              driver: overlay
          
          services:
            build:
              command: "-name build -executors 1 -labels 'docker build' -master http://jenkins-master:8080 -mode normal -username jenkins -password derp"
              environment:
                - "constraint:jenkins==build"
                - "JAVA_OPTS=-Djava.awt.headless=true -Duser.timezone=America/Phoenix"
              image: jenkins-swarm-slave
              networks:
                - jenkins
              restart: "on-failure:10"
              volumes:
                - /var/lib:/var/lib:rw
                - /var/run:/var/run:rw
          

          Jacob Blain Christen added a comment - - edited slave Dockerfile FROM java:8-jdk MAINTAINER Jacob Blain Christen <mailto:dweomer5@gmail.com, https: //github.com/dweomer, https://twitter.com/dweomer> ARG DOCKER_VERSION=1.10.3 ARG GOSU_VERSION=1.7 ARG TINI_VERSION=0.8.4 LABEL "ARG:DOCKER_VERSION" = "${DOCKER_VERSION}" \ "ARG:GOSU_VERSION" = "${GOSU_VERSION}" \ "ARG:TINI_VERSION" = "${TINI_VERSION}" ENV JENKINS_GROUP=jenkins \ JENKINS_HOME=/ var /lib/jenkins \ JENKINS_USER=jenkins RUN set -x \ && export DEBIAN_FRONTEND=noninteractive \ && apt-get update \ ### Install netstat to allow connection health check with `netstat -tan | grep ESTABLISHED` && apt-get install -y \ curl \ git \ net-tools \ wget \ ### Install Docker && curl -fsSL --create-dirs --output /usr/local/bin/docker \ "https: //get.docker.com/builds/$(uname -s)/$(uname -m)/docker-${DOCKER_VERSION}" \ && chmod -v +x /usr/local/bin/docker \ ### Install Gosu && curl -fsSL --create-dirs --output /usr/local/bin/gosu \ "https: //github.com/tianon/gosu/releases/download/${GOSU_VERSION}/gosu-$(dpkg --print-architecture)" \ && chmod -v +x /usr/local/bin/gosu \ ### Install Tini && curl --create-dirs -sSLo /usr/local/bin/tini \ "https: //github.com/krallin/tini/releases/download/v${TINI_VERSION}/tini- static " \ && chmod -v +x /usr/local/bin/tini \ ### Create Jenkins User && groupadd ${JENKINS_GROUP} \ && useradd -c "Jenkins Swarm Agent" -d ${JENKINS_HOME} -g ${JENKINS_GROUP} -m ${JENKINS_USER} \ ### Cleanup && apt-get clean \ && rm -rf \ /tmp/* \ / var /lib/apt/lists/* \ / var /tmp/* COPY rootfs / ENTRYPOINT [ "/jenkins-swarm-slave.sh" ] RUN set -x \ && chmod -v +x /jenkins-swarm-slave.sh \ && find /jenkins-swarm-slave* -type f WORKDIR ${JENKINS_HOME} VOLUME ${JENKINS_HOME} slave docker-compose.yml (snippet) version: '2' networks: jenkins: driver: overlay services: build: command: "-name build -executors 1 -labels 'docker build' -master http: //jenkins-master:8080 -mode normal -username jenkins -password derp" environment: - "constraint:jenkins==build" - "JAVA_OPTS=-Djava.awt.headless= true -Duser.timezone=America/Phoenix" image: jenkins-swarm-slave networks: - jenkins restart: "on-failure:10" volumes: - / var /lib:/ var /lib:rw - / var /run:/ var /run:rw

          Well, this is embarrassing. Neglected to match the path of the VOLUME from the container. Testing...

          swarm container docker inspect
          [
              {
                  "Id": "8dcf8118ebe64973c50c49599f72750960cf023eaa5076629c5f11c45526c310",
                  "Created": "2016-04-13T05:29:20.571482012Z",
                  "Path": "/jenkins-swarm-slave.sh",
                  "Args": [
                      "-name",
                      "build",
                      "-executors",
                      "1",
                      "-labels",
                      "docker build",
                      "-master",
                      "http://jenkins-master:8080",
                      "-mode",
                      "normal",
                      "-username",
                      "jenkins",
                      "-password",
                      "derp"
                  ],
                  "State": {
                      "Status": "running",
                      "Running": true,
                      "Paused": false,
                      "Restarting": false,
                      "OOMKilled": false,
                      "Dead": false,
                      "Pid": 24473,
                      "ExitCode": 0,
                      "Error": "",
                      "StartedAt": "2016-04-13T08:07:21.13390774Z",
                      "FinishedAt": "2016-04-13T08:07:20.61867963Z"
                  },
                  "Image": "sha256:6b2499a8274a28875144c8547a6f783fa74621c0a6c4dd99ca8ea6d759f0c859",
                  "ResolvConfPath": "/var/lib/docker/containers/8dcf8118ebe64973c50c49599f72750960cf023eaa5076629c5f11c45526c310/resolv.conf",
                  "HostnamePath": "/var/lib/docker/containers/8dcf8118ebe64973c50c49599f72750960cf023eaa5076629c5f11c45526c310/hostname",
                  "HostsPath": "/var/lib/docker/containers/8dcf8118ebe64973c50c49599f72750960cf023eaa5076629c5f11c45526c310/hosts",
                  "LogPath": "/var/lib/docker/containers/8dcf8118ebe64973c50c49599f72750960cf023eaa5076629c5f11c45526c310/8dcf8118ebe64973c50c49599f72750960cf023eaa5076629c5f11c45526c310-json.log",
                  "Name": "/jenkins_build_8",
                  "RestartCount": 1,
                  "Driver": "aufs",
                  "MountLabel": "",
                  "ProcessLabel": "",
                  "AppArmorProfile": "",
                  "ExecIDs": null,
                  "HostConfig": {
                      "Binds": [
                          "e6790efed69bac5ce78b2aee98beea46abf6be9ac0b9fc586a6ca184639ecf07:/var/lib/jenkins:rw",
                          "/var/run:/var/run:rw",
                          "/var/lib:/var/lib:rw"
                      ],
                      "ContainerIDFile": "",
                      "LogConfig": {
                          "Type": "json-file",
                          "Config": {
                              "max-file": "10",
                              "max-size": "10m"
                          }
                      },
                      "NetworkMode": "jenkins_jenkins",
                      "PortBindings": {},
                      "RestartPolicy": {
                          "Name": "on-failure",
                          "MaximumRetryCount": 10
                      },
                      "VolumeDriver": "",
                      "VolumesFrom": [],
                      "CapAdd": null,
                      "CapDrop": null,
                      "Dns": null,
                      "DnsOptions": null,
                      "DnsSearch": null,
                      "ExtraHosts": null,
                      "GroupAdd": null,
                      "IpcMode": "",
                      "Links": null,
                      "OomScoreAdj": 0,
                      "PidMode": "",
                      "Privileged": false,
                      "PublishAllPorts": false,
                      "ReadonlyRootfs": false,
                      "SecurityOpt": null,
                      "UTSMode": "",
                      "ShmSize": 67108864,
                      "ConsoleSize": [
                          0,
                          0
                      ],
                      "Isolation": "",
                      "CpuShares": 512,
                      "CgroupParent": "",
                      "BlkioWeight": 0,
                      "BlkioWeightDevice": null,
                      "BlkioDeviceReadBps": null,
                      "BlkioDeviceWriteBps": null,
                      "BlkioDeviceReadIOps": null,
                      "BlkioDeviceWriteIOps": null,
                      "CpuPeriod": 0,
                      "CpuQuota": 0,
                      "CpusetCpus": "",
                      "CpusetMems": "",
                      "Devices": null,
                      "KernelMemory": 0,
                      "Memory": 0,
                      "MemoryReservation": 0,
                      "MemorySwap": 0,
                      "MemorySwappiness": -1,
                      "OomKillDisable": false,
                      "PidsLimit": 0,
                      "Ulimits": null
                  },
                  "GraphDriver": {
                      "Name": "aufs",
                      "Data": null
                  },
                  "Mounts": [
                      {
                          "Source": "/var/run",
                          "Destination": "/var/run",
                          "Mode": "rw",
                          "RW": true,
                          "Propagation": "rprivate"
                      },
                      {
                          "Source": "/var/lib",
                          "Destination": "/var/lib",
                          "Mode": "rw",
                          "RW": true,
                          "Propagation": "rprivate"
                      },
                      {
                          "Name": "e6790efed69bac5ce78b2aee98beea46abf6be9ac0b9fc586a6ca184639ecf07",
                          "Source": "/var/lib/docker/volumes/e6790efed69bac5ce78b2aee98beea46abf6be9ac0b9fc586a6ca184639ecf07/_data",
                          "Destination": "/var/lib/jenkins",
                          "Driver": "local",
                          "Mode": "rw",
                          "RW": true,
                          "Propagation": "rprivate"
                      }
                  ],
                  "Config": {
                      "Hostname": "8dcf8118ebe6",
                      "Domainname": "",
                      "User": "",
                      "AttachStdin": false,
                      "AttachStdout": false,
                      "AttachStderr": false,
                      "Tty": false,
                      "OpenStdin": false,
                      "StdinOnce": false,
                      "Env": [
                          "JAVA_OPTS=-Djava.awt.headless=true -Duser.timezone=America/Phoenix",
                          "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                          "LANG=C.UTF-8",
                          "JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64",
                          "JAVA_VERSION=8u72",
                          "JAVA_DEBIAN_VERSION=8u72-b15-1~bpo8+1",
                          "CA_CERTIFICATES_JAVA_VERSION=20140324",
                          "JENKINS_GROUP=jenkins",
                          "JENKINS_HOME=/var/lib/jenkins",
                          "JENKINS_USER=jenkins"
                      ],
                      "Cmd": [
                          "-name",
                          "build",
                          "-executors",
                          "1",
                          "-labels",
                          "docker build",
                          "-master",
                          "http://jenkins-master:8080",
                          "-mode",
                          "normal",
                          "-username",
                          "jenkins",
                          "-password",
                          "jenkins1"
                      ],
                      "Image": "repo.int.parchment.com:5000/jenkins-swarm-slave",
                      "Volumes": {
                          "/var/lib": {},
                          "/var/lib/jenkins": {},
                          "/var/run": {}
                      },
                      "WorkingDir": "/var/lib/jenkins",
                      "Entrypoint": [
                          "/jenkins-swarm-slave.sh"
                      ],
                      "OnBuild": null,
                      "Labels": {
                          "ARG:DOCKER_VERSION": "1.10.3",
                          "ARG:GOSU_VERSION": "1.7",
                          "ARG:TINI_VERSION": "0.8.4",
                          "com.docker.compose.config-hash": "b35c78cdd4eeee86eb9fdf6904c479bb253785aecd94f69f75fdeac9b3e2abb3",
                          "com.docker.compose.container-number": "8",
                          "com.docker.compose.oneoff": "False",
                          "com.docker.compose.project": "jenkins",
                          "com.docker.compose.service": "build",
                          "com.docker.compose.version": "1.6.2",
                          "com.docker.swarm.affinities": "[\"container==5d78093c02d782262bf68372d1278139ba6d76a9edae2134b222708eeecb1d5b\"]",
                          "com.docker.swarm.constraints": "[\"jenkins==build\"]",
                          "com.docker.swarm.id": "d4f1cde009b6ea39fb1ac07ca74ec9e364aaa38d6e97b47464033cc1400d237f"
                      }
                  },
                  "NetworkSettings": {
                      "Bridge": "",
                      "SandboxID": "07599be5de5e93e46ce74d0c571e71a8110eb05f5830f44402fa67b24665e3da",
                      "HairpinMode": false,
                      "LinkLocalIPv6Address": "",
                      "LinkLocalIPv6PrefixLen": 0,
                      "Ports": {},
                      "SandboxKey": "/var/run/docker/netns/07599be5de5e",
                      "SecondaryIPAddresses": null,
                      "SecondaryIPv6Addresses": null,
                      "EndpointID": "",
                      "Gateway": "",
                      "GlobalIPv6Address": "",
                      "GlobalIPv6PrefixLen": 0,
                      "IPAddress": "",
                      "IPPrefixLen": 0,
                      "IPv6Gateway": "",
                      "MacAddress": "",
                      "Networks": {
                          "jenkins_jenkins": {
                              "IPAMConfig": null,
                              "Links": null,
                              "Aliases": [
                                  "8dcf8118ebe6",
                                  "build"
                              ],
                              "NetworkID": "8860391922ea2ff4825495f28964a2233d575c276d5e6fd9f4f85decd23eae09",
                              "EndpointID": "cacad1681306b31661fa8a8b2d76d3e3114329dcc62886736c6b97f22b40fc55",
                              "Gateway": "",
                              "IPAddress": "10.0.1.6",
                              "IPPrefixLen": 24,
                              "IPv6Gateway": "",
                              "GlobalIPv6Address": "",
                              "GlobalIPv6PrefixLen": 0,
                              "MacAddress": "02:42:0a:00:01:06"
                          }
                      }
                  }
              }
          ]
          

          Jacob Blain Christen added a comment - Well, this is embarrassing. Neglected to match the path of the VOLUME from the container. Testing... swarm container docker inspect [ { "Id" : "8dcf8118ebe64973c50c49599f72750960cf023eaa5076629c5f11c45526c310" , "Created" : "2016-04-13T05:29:20.571482012Z" , "Path" : "/jenkins-swarm-slave.sh" , "Args" : [ "-name" , "build" , "-executors" , "1" , "-labels" , "docker build" , "-master" , "http: //jenkins-master:8080" , "-mode" , "normal" , "-username" , "jenkins" , "-password" , "derp" ], "State" : { "Status" : "running" , "Running" : true , "Paused" : false , "Restarting" : false , "OOMKilled" : false , "Dead" : false , "Pid" : 24473, "ExitCode" : 0, "Error" : "", "StartedAt" : "2016-04-13T08:07:21.13390774Z" , "FinishedAt" : "2016-04-13T08:07:20.61867963Z" }, "Image" : "sha256:6b2499a8274a28875144c8547a6f783fa74621c0a6c4dd99ca8ea6d759f0c859" , "ResolvConfPath" : "/ var /lib/docker/containers/8dcf8118ebe64973c50c49599f72750960cf023eaa5076629c5f11c45526c310/resolv.conf" , "HostnamePath" : "/ var /lib/docker/containers/8dcf8118ebe64973c50c49599f72750960cf023eaa5076629c5f11c45526c310/hostname" , "HostsPath" : "/ var /lib/docker/containers/8dcf8118ebe64973c50c49599f72750960cf023eaa5076629c5f11c45526c310/hosts" , "LogPath" : "/ var /lib/docker/containers/8dcf8118ebe64973c50c49599f72750960cf023eaa5076629c5f11c45526c310/8dcf8118ebe64973c50c49599f72750960cf023eaa5076629c5f11c45526c310-json.log" , "Name" : "/jenkins_build_8" , "RestartCount" : 1, "Driver" : "aufs" , "MountLabel" : "", "ProcessLabel" : "", "AppArmorProfile" : "", "ExecIDs" : null , "HostConfig" : { "Binds" : [ "e6790efed69bac5ce78b2aee98beea46abf6be9ac0b9fc586a6ca184639ecf07:/ var /lib/jenkins:rw" , "/ var /run:/ var /run:rw" , "/ var /lib:/ var /lib:rw" ], "ContainerIDFile" : "", "LogConfig" : { "Type" : "json-file" , "Config" : { "max-file" : "10" , "max-size" : "10m" } }, "NetworkMode" : "jenkins_jenkins" , "PortBindings" : {}, "RestartPolicy" : { "Name" : "on-failure" , "MaximumRetryCount" : 10 }, "VolumeDriver" : "", "VolumesFrom" : [], "CapAdd" : null , "CapDrop" : null , "Dns" : null , "DnsOptions" : null , "DnsSearch" : null , "ExtraHosts" : null , "GroupAdd" : null , "IpcMode" : "", "Links" : null , "OomScoreAdj" : 0, "PidMode" : "", "Privileged" : false , "PublishAllPorts" : false , "ReadonlyRootfs" : false , "SecurityOpt" : null , "UTSMode" : "", "ShmSize" : 67108864, "ConsoleSize" : [ 0, 0 ], "Isolation" : "", "CpuShares" : 512, "CgroupParent" : "", "BlkioWeight" : 0, "BlkioWeightDevice" : null , "BlkioDeviceReadBps" : null , "BlkioDeviceWriteBps" : null , "BlkioDeviceReadIOps" : null , "BlkioDeviceWriteIOps" : null , "CpuPeriod" : 0, "CpuQuota" : 0, "CpusetCpus" : "", "CpusetMems" : "", "Devices" : null , "KernelMemory" : 0, "Memory" : 0, "MemoryReservation" : 0, "MemorySwap" : 0, "MemorySwappiness" : -1, "OomKillDisable" : false , "PidsLimit" : 0, "Ulimits" : null }, "GraphDriver" : { "Name" : "aufs" , "Data" : null }, "Mounts" : [ { "Source" : "/ var /run" , "Destination" : "/ var /run" , "Mode" : "rw" , "RW" : true , "Propagation" : "rprivate" }, { "Source" : "/ var /lib" , "Destination" : "/ var /lib" , "Mode" : "rw" , "RW" : true , "Propagation" : "rprivate" }, { "Name" : "e6790efed69bac5ce78b2aee98beea46abf6be9ac0b9fc586a6ca184639ecf07" , "Source" : "/ var /lib/docker/volumes/e6790efed69bac5ce78b2aee98beea46abf6be9ac0b9fc586a6ca184639ecf07/_data" , "Destination" : "/ var /lib/jenkins" , "Driver" : "local" , "Mode" : "rw" , "RW" : true , "Propagation" : "rprivate" } ], "Config" : { "Hostname" : "8dcf8118ebe6" , "Domainname" : "", "User" : "", "AttachStdin" : false , "AttachStdout" : false , "AttachStderr" : false , "Tty" : false , "OpenStdin" : false , "StdinOnce" : false , "Env" : [ "JAVA_OPTS=-Djava.awt.headless= true -Duser.timezone=America/Phoenix" , "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" , "LANG=C.UTF-8" , "JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64" , "JAVA_VERSION=8u72" , "JAVA_DEBIAN_VERSION=8u72-b15-1~bpo8+1" , "CA_CERTIFICATES_JAVA_VERSION=20140324" , "JENKINS_GROUP=jenkins" , "JENKINS_HOME=/ var /lib/jenkins" , "JENKINS_USER=jenkins" ], "Cmd" : [ "-name" , "build" , "-executors" , "1" , "-labels" , "docker build" , "-master" , "http: //jenkins-master:8080" , "-mode" , "normal" , "-username" , "jenkins" , "-password" , "jenkins1" ], "Image" : "repo. int .parchment.com:5000/jenkins-swarm-slave" , "Volumes" : { "/ var /lib" : {}, "/ var /lib/jenkins" : {}, "/ var /run" : {} }, "WorkingDir" : "/ var /lib/jenkins" , "Entrypoint" : [ "/jenkins-swarm-slave.sh" ], "OnBuild" : null , "Labels" : { "ARG:DOCKER_VERSION" : "1.10.3" , "ARG:GOSU_VERSION" : "1.7" , "ARG:TINI_VERSION" : "0.8.4" , "com.docker.compose.config-hash" : "b35c78cdd4eeee86eb9fdf6904c479bb253785aecd94f69f75fdeac9b3e2abb3" , "com.docker.compose.container-number" : "8" , "com.docker.compose.oneoff" : "False" , "com.docker.compose.project" : "jenkins" , "com.docker.compose.service" : "build" , "com.docker.compose.version" : "1.6.2" , "com.docker.swarm.affinities" : "[\" container==5d78093c02d782262bf68372d1278139ba6d76a9edae2134b222708eeecb1d5b\ "]" , "com.docker.swarm.constraints" : "[\" jenkins==build\ "]" , "com.docker.swarm.id" : "d4f1cde009b6ea39fb1ac07ca74ec9e364aaa38d6e97b47464033cc1400d237f" } }, "NetworkSettings" : { "Bridge" : "", "SandboxID" : "07599be5de5e93e46ce74d0c571e71a8110eb05f5830f44402fa67b24665e3da" , "HairpinMode" : false , "LinkLocalIPv6Address" : "", "LinkLocalIPv6PrefixLen" : 0, "Ports" : {}, "SandboxKey" : "/ var /run/docker/netns/07599be5de5e" , "SecondaryIPAddresses" : null , "SecondaryIPv6Addresses" : null , "EndpointID" : "", "Gateway" : "", "GlobalIPv6Address" : "", "GlobalIPv6PrefixLen" : 0, "IPAddress" : "", "IPPrefixLen" : 0, "IPv6Gateway" : "", "MacAddress" : "", "Networks" : { "jenkins_jenkins" : { "IPAMConfig" : null , "Links" : null , "Aliases" : [ "8dcf8118ebe6" , "build" ], "NetworkID" : "8860391922ea2ff4825495f28964a2233d575c276d5e6fd9f4f85decd23eae09" , "EndpointID" : "cacad1681306b31661fa8a8b2d76d3e3114329dcc62886736c6b97f22b40fc55" , "Gateway" : "", "IPAddress" : "10.0.1.6" , "IPPrefixLen" : 24, "IPv6Gateway" : "", "GlobalIPv6Address" : "", "GlobalIPv6PrefixLen" : 0, "MacAddress" : "02:42:0a:00:01:06" } } } } ]

          Jacob Blain Christen added a comment - - edited

          Yep, the /var/lib/jenkins volume declaration was the problem. Sorry for this one, /sigh

          Jacob Blain Christen added a comment - - edited Yep, the /var/lib/jenkins volume declaration was the problem. Sorry for this one, /sigh

          PEBKAC

          Jacob Blain Christen added a comment - PEBKAC

          Jesse Glick added a comment -

          Do not like Closed status…

          Jesse Glick added a comment - Do not like Closed status…

          Walid Natat added a comment -

          I ran into a similar issue after upgrading to kubernetes 1.6 and running a Jenkins pod with a Docker-outside-of-Docker setup. According to the Kubernetes 1.6 release note:

          Enhance Kubelet QoS:
          Pods are placed under a new cgroup hierarchy by default. This feature requires draining and restarting the node as part of upgrades. To opt-out set --cgroups-per-qos=false.

           

          I believe this will cause the the regex pattern in the code below to fail matching the new default cgroup (something like 2:cpu:/kubepod/container_id)

          /**
          * Known cgroup formats
          *
          * 4:cpuset:/system.slice/docker-3dd988081e7149463c043b5d9c57d7309e079c5e9290f91feba1cc45a04d6a5b.scope
          * 2:cpu:/docker/3dd988081e7149463c043b5d9c57d7309e079c5e9290f91feba1cc45a04d6a5b
          */
          public static final String CGROUP_MATCHER_PATTERN = "(?m)^\\d+:\\w+:(?:/[\\w\\.]+)?/docker[-/](?<containerId>\\p{XDigit}{12,})(?:\\.scope)?$";
          
          

           

          A solution may be restarting kubelet on each node with option --cgroups-per-qos=false, but I haven't tried.

          Walid Natat added a comment - I ran into a similar issue after upgrading to kubernetes 1.6 and running a Jenkins pod with a Docker-outside-of-Docker setup. According to the Kubernetes 1.6 release note : Enhance Kubelet QoS: Pods are placed under a new cgroup hierarchy by default. This feature requires draining and restarting the node as part of upgrades. To opt-out set --cgroups-per-qos=false.   I believe this will cause the the regex pattern in the code below to fail matching the new default cgroup (something like 2:cpu:/kubepod/container_id) /** * Known cgroup formats * * 4:cpuset:/system.slice/docker-3dd988081e7149463c043b5d9c57d7309e079c5e9290f91feba1cc45a04d6a5b.scope * 2:cpu:/docker/3dd988081e7149463c043b5d9c57d7309e079c5e9290f91feba1cc45a04d6a5b */ public static final String CGROUP_MATCHER_PATTERN = "(?m)^\\d+:\\w+:(?:/[\\w\\.]+)?/docker[-/](?<containerId>\\p{XDigit}{12,})(?:\\.scope)?$" ;   A solution may be restarting kubelet on each node with option --cgroups-per-qos=false, but I haven't tried.

            jglick Jesse Glick
            dweomer Jacob Blain Christen
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: