Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40866

docker.inside() like syntax that keeps the workspace

    XMLWordPrintable

Details

    Description

      With the traditional pipeline we have been able to do a mixture of working with the Jenkins workspace and using docker.inside() to pull in docker containers which have additional build tools needed to do a build. Docker.inside() maps the jenkins workspace to docker volume, so that what happens inside the docker container can read from, and write to the jenkins workspace.

      Part of this is our build environment. The builds all run from slaves which run as docker containers inside Kubernetes. Those pods run privileged, so that they can reach the docker socket on the host to spin up "peer" docker containers when the docker.inside() commands are run. This is needed in order to run docker commands and to build new docker containers.

      So for instance a slave does a checkout scm. gathers information about the repository commit using git commands, then does docker.inside() to pull in a pre-configured docker image with all the necessary npm modules already installed. It runs the gulp tasks inside() the container which thanks to the way things are mapped, writes the gulp output back to the jenkins workpace.

      Then the inside() block closes, and the next steps of the pipeline do a docker build, and docker push to our registry.

      I initially tired doing this in declarative pipeline by using an agent statement at the top just inside the pipeline{
      agent label:'swarm', docker:'our-registry/npm-build:latest'

      This initially failed because while the slave has git on it, git does not exist inside that npm-build image, so I couldn't use the git commands to determine the url and git commit hash

      I initially added git to that image, and that part worked, but then I realized I could go no further, since there was no way to run docker commands inside this image which was already being run inside a docker container on the slave. I had no way of making it privileged so that I could access /var/run/docker.sock on the host.

      I tried setting
      pipeline{
      agent label:'swarm'

      which worked for the first part, of pulling in the code, and running the git commands, but then in another stage I tried

      stage('gulp build'){
      agent label:'swarm', docker:'our-registry/npm-build:latest'

      but was greeted with a completely blank workspace.

      I know I can use stash/unstash, to run the git commands on the slave workspace, then stash, then unstash inside the docker container, run the gulp commands, stash again, then grab a blank workspace again on slave and unstash, but the inefficiencies in doing that bug me. It would become a nightmare in some of our more complex workflows that use docker.inside() multiple times, and even have multiple different containers, each configured for just the tool needed for that stage.

      I assume I can probably do some of that if I use a script{} block, but was hoping to avoid that as much as possible, as I hope to hand these pipelines off to people not as familiar with Jenkins and pipeline in general.

      I'm just looking for a way to do docker.inside() like commands where we pull in another docker image and do work in there, but still have access to the workspace.

      Attachments

        Activity

          tomlarrow Tom Larrow created issue -
          michaelneale Michael Neale added a comment - - edited

          Good writeup tomlarrow. I actually do a very similar thing myself (but without k8s).

          I think this isn't an unreasonable scenario.

          So basically you want to effectively run a bunch of docker.inside like stages, each doing a different thing, but on the one shared workspace, on the one slave, until the pipeline is run, right? Ideally you would just specify the agent labelle once at the pipeline level, and then inside each stage, specify the docker image you want (and it would run on the same labelled slave in an ideal world). You are right to see if there is a way to do this without resorting to script.

          I think this may require some changes to declarative, so will see what abayer says:

          • When using agent .. docker, should checkout scm always happen outside of docker? (so the image doesn't need to have git)?
          • If there is an agent specified with a label globally, and then in a stage an agent is specified which only requires docker, could it be made to use the one slave that is already provisioned? (this would cover this entire scenario).

          Not sure how "hacky" these things are and if this opens up other surprising side effects we don't want. In general, the simplest thing for a pipeline is one workspace on one slave, so I think it is reasonable that someone may expect things to work like this in this situation.

          michaelneale Michael Neale added a comment - - edited Good writeup tomlarrow . I actually do a very similar thing myself (but without k8s). I think this isn't an unreasonable scenario. So basically you want to effectively run a bunch of docker.inside like stages, each doing a different thing, but on the one shared workspace, on the one slave, until the pipeline is run, right? Ideally you would just specify the agent labelle once at the pipeline level, and then inside each stage, specify the docker image you want (and it would run on the same labelled slave in an ideal world). You are right to see if there is a way to do this without resorting to script. I think this may require some changes to declarative, so will see what abayer says: When using agent .. docker, should checkout scm always happen outside of docker? (so the image doesn't need to have git)? If there is an agent specified with a label globally, and then in a stage an agent is specified which only requires docker, could it be made to use the one slave that is already provisioned? (this would cover this entire scenario). Not sure how "hacky" these things are and if this opens up other surprising side effects we don't want. In general, the simplest thing for a pipeline is one workspace on one slave, so I think it is reasonable that someone may expect things to work like this in this situation.
          tomlarrow Tom Larrow added a comment - - edited

          michaelneale that is generally it. I've actually been going back and forth a bit on this, and not fully sure what the right solution is.

          First a quick correction: "When using agent .. docker, should checkout scm always happen outside of docker? (so the image doesn't need to have git)?"

          The image doesn't need git to do the pull, it is doing the git clone/checkout just fine. I need it because I am trying to pull in the Git environment variables, to find the origin of the commit, and the hash so that I can bake that metadata into the docker image's metadata for tracking. If things like https://issues.jenkins-ci.org/browse/JENKINS-35230 and https://issues.jenkins-ci.org/browse/JENKINS-34455 were fixed, and I had access to that metadata from the pipeline, I wouldn't need to use the commands listed at https://jenkins.io/doc/pipeline/examples/#gitcommit which use git locally to get those variables.

          As far as my initial comment. Everything works in declarative pipeline if I do what was suggested, put the pipeline on the agent:
          pipeline {
          agent label:'swarm'

          and then use a script block, with docker.inside() in it.

          I have just been trying to avoid script blocks as much as possible to make the pipelines as usable by others with little pipeline experience as possible.

          As far as what makes sense, guess the easiest would be something along the lines of being able to do a stage with some nomenclature like this:
          stage{
          agent: same, docker: 'node'

          and that would keep things on the same agent (syntax sort of like 'none') but inside that docker image.

          That would of course force me to give up parallelization where different parallel steps use different docker build environments, as the docker container would be tied to a stage, and stages can't be parallelized. But in my mental debates that is something that could be given up to greatly simplify the pipeline syntax.

          tomlarrow Tom Larrow added a comment - - edited michaelneale that is generally it. I've actually been going back and forth a bit on this, and not fully sure what the right solution is. First a quick correction: "When using agent .. docker, should checkout scm always happen outside of docker? (so the image doesn't need to have git)?" The image doesn't need git to do the pull, it is doing the git clone/checkout just fine. I need it because I am trying to pull in the Git environment variables, to find the origin of the commit, and the hash so that I can bake that metadata into the docker image's metadata for tracking. If things like https://issues.jenkins-ci.org/browse/JENKINS-35230 and https://issues.jenkins-ci.org/browse/JENKINS-34455 were fixed, and I had access to that metadata from the pipeline, I wouldn't need to use the commands listed at https://jenkins.io/doc/pipeline/examples/#gitcommit which use git locally to get those variables. As far as my initial comment. Everything works in declarative pipeline if I do what was suggested, put the pipeline on the agent: pipeline { agent label:'swarm' and then use a script block, with docker.inside() in it. I have just been trying to avoid script blocks as much as possible to make the pipelines as usable by others with little pipeline experience as possible. As far as what makes sense, guess the easiest would be something along the lines of being able to do a stage with some nomenclature like this: stage{ agent: same, docker: 'node' and that would keep things on the same agent (syntax sort of like 'none') but inside that docker image. That would of course force me to give up parallelization where different parallel steps use different docker build environments, as the docker container would be tied to a stage, and stages can't be parallelized. But in my mental debates that is something that could be given up to greatly simplify the pipeline syntax.
          abayer Andrew Bayer added a comment -

          Hmmm...yeah, I think we can do that. If we were starting from scratch, I'd say have the default behavior for agent in a stage with docker or dockerfile to be using the current node, assuming you're using label or any at the top level (since I'd be wary of having docker-in-docker be default behavior). But since we've already got history with this, I'd say adding a flag to docker and dockerfile to say "use the existing node context for this stage" would be eminently reasonable and very easy to implement.

          abayer Andrew Bayer added a comment - Hmmm...yeah, I think we can do that. If we were starting from scratch, I'd say have the default behavior for agent in a stage with docker or dockerfile to be using the current node, assuming you're using label or any at the top level (since I'd be wary of having docker-in-docker be default behavior). But since we've already got history with this, I'd say adding a flag to docker and dockerfile to say "use the existing node context for this stage" would be eminently reasonable and very easy to implement.
          abayer Andrew Bayer added a comment -

          I've got a very work-in-progress PR for this up at https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/93 - I decided to keep it simple in that it doesn't error out if you use it with agent none at the global level; instead, it'll just say "Oh, no node context, I'll use a label". And I left it still able to do docker-in-docker because...well, I couldn't trivially figure out how to prevent that at validation time. =) Will think about it more further and sleep on this.

          abayer Andrew Bayer added a comment - I've got a very work-in-progress PR for this up at https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/93 - I decided to keep it simple in that it doesn't error out if you use it with agent none at the global level; instead, it'll just say "Oh, no node context, I'll use a label". And I left it still able to do docker-in-docker because...well, I couldn't trivially figure out how to prevent that at validation time. =) Will think about it more further and sleep on this.
          abayer Andrew Bayer added a comment -

          I've improved the logic for determining if we're on a node, so that's nice - https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/93/commits/a262c3e9fa90372fadb9d2b0a64b27658f3b2d81 is the commit with the bulk of it including the tests. tomlarrow - any thoughts on that syntax?

          abayer Andrew Bayer added a comment - I've improved the logic for determining if we're on a node, so that's nice - https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/93/commits/a262c3e9fa90372fadb9d2b0a64b27658f3b2d81 is the commit with the bulk of it including the tests. tomlarrow - any thoughts on that syntax?
          hrmpw Patrick Wolf added a comment - - edited

          Changing this to be the default behavior will cause existing pipelines to have possible unexpected outcomes. It also sounds like maintaining the workspace across stages with different containers will likely break if the top-level agent is a docker directive itself.

          In that case it is probably best to keep the default behavior that each agent call in a stage will allocate a new node and execute on that node. Users would then need to set reuseNode true inside docker for the agent to maintain the workspace context but in a new container.

          In addition, what if we add a top-level options to enable workspace reuse as the default.

          pipeline {
            options { 
              reuseNode true
            }
            agent {
              label "my-label"
            }
            
            stages {
              stage("Build") {
                steps {      
                   echo "I'm not in a container"
                 }
              }
              stage("Docker") {
                agent {
                  docker {
                    image "ubuntu"
                  }
                  steps {
                    echo "I'm in a container on same workspace"
                  }
               }
            }
          }
          
          hrmpw Patrick Wolf added a comment - - edited Changing this to be the default behavior will cause existing pipelines to have possible unexpected outcomes. It also sounds like maintaining the workspace across stages with different containers will likely break if the top-level agent is a docker directive itself. In that case it is probably best to keep the default behavior that each agent call in a stage will allocate a new node and execute on that node. Users would then need to set reuseNode true inside docker for the agent to maintain the workspace context but in a new container. In addition, what if we add a top-level options to enable workspace reuse as the default. pipeline { options { reuseNode true } agent { label "my-label" } stages { stage( "Build" ) { steps { echo "I'm not in a container" } } stage( "Docker" ) { agent { docker { image "ubuntu" } steps { echo "I'm in a container on same workspace" } } } }
          hrmpw Patrick Wolf made changes -
          Field Original Value New Value
          Status Open [ 1 ] In Progress [ 3 ]
          tomlarrow Tom Larrow added a comment -

          I agree with hrmpw that if I had already written pipelines that expected a new workspace every time, I wouldn't want the default changed on me. I like the option of reuseNode being something I could set in the options, so I wouldn't need to specify it every time I used it.

          Just to make sure I was reading the code correctly, It looked like the vars would still be available to use in the stage agent definition. We don't do true docker in docker, rather if we have one docker container building another, we pass in the arg "-v /var/run/docker.sock:/var/run/docker.sock" in order to let the docker daemon inside the container control the host's docker and build sibling containers.

          Other than that, it looks good, and will save quite a bit of script{} blocks in my declarative pipeline code.

          tomlarrow Tom Larrow added a comment - I agree with hrmpw that if I had already written pipelines that expected a new workspace every time, I wouldn't want the default changed on me. I like the option of reuseNode being something I could set in the options, so I wouldn't need to specify it every time I used it. Just to make sure I was reading the code correctly, It looked like the vars would still be available to use in the stage agent definition. We don't do true docker in docker, rather if we have one docker container building another, we pass in the arg "-v /var/run/docker.sock:/var/run/docker.sock" in order to let the docker daemon inside the container control the host's docker and build sibling containers. Other than that, it looks good, and will save quite a bit of script{} blocks in my declarative pipeline code.

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/agent/AbstractDockerAgent.java
          pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipeline.java
          pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineFromDockerfile.java
          pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineFromDockerfileScript.groovy
          pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineScript.groovy
          pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/AgentTest.java
          pipeline-model-definition/src/test/resources/agentDockerDontReuseNode.groovy
          pipeline-model-definition/src/test/resources/agentDockerReuseNode.groovy
          http://jenkins-ci.org/commit/pipeline-model-definition-plugin/a262c3e9fa90372fadb9d2b0a64b27658f3b2d81
          Log:
          JENKINS-40866 First work on reusing nodes for stage agents

          More tests still needed.

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/agent/AbstractDockerAgent.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipeline.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineFromDockerfile.java pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineFromDockerfileScript.groovy pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineScript.groovy pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/AgentTest.java pipeline-model-definition/src/test/resources/agentDockerDontReuseNode.groovy pipeline-model-definition/src/test/resources/agentDockerReuseNode.groovy http://jenkins-ci.org/commit/pipeline-model-definition-plugin/a262c3e9fa90372fadb9d2b0a64b27658f3b2d81 Log: JENKINS-40866 First work on reusing nodes for stage agents More tests still needed.

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/agent/AbstractDockerAgent.java
          pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipeline.java
          pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineFromDockerfile.java
          pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/AbstractDockerPipelineScript.groovy
          pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineFromDockerfileScript.groovy
          pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineScript.groovy
          pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/AgentTest.java
          pipeline-model-definition/src/test/resources/agentDockerDontReuseNode.groovy
          pipeline-model-definition/src/test/resources/agentDockerReuseNode.groovy
          http://jenkins-ci.org/commit/pipeline-model-definition-plugin/3222d5ababdf848fe7b81d78f40140330a9d6051
          Log:
          Merge pull request #93 from abayer/jenkins-40866

          JENKINS-40866 Docker agent in stage can reuse global node

          Compare: https://github.com/jenkinsci/pipeline-model-definition-plugin/compare/b4d556f90759...3222d5ababdf

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/agent/AbstractDockerAgent.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipeline.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineFromDockerfile.java pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/AbstractDockerPipelineScript.groovy pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineFromDockerfileScript.groovy pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineScript.groovy pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/AgentTest.java pipeline-model-definition/src/test/resources/agentDockerDontReuseNode.groovy pipeline-model-definition/src/test/resources/agentDockerReuseNode.groovy http://jenkins-ci.org/commit/pipeline-model-definition-plugin/3222d5ababdf848fe7b81d78f40140330a9d6051 Log: Merge pull request #93 from abayer/jenkins-40866 JENKINS-40866 Docker agent in stage can reuse global node Compare: https://github.com/jenkinsci/pipeline-model-definition-plugin/compare/b4d556f90759...3222d5ababdf
          abayer Andrew Bayer added a comment -

          Merged! This'll be in the 0.9 release I'll be doing in the next couple days. Only syntax change is the addition of the reuseNode boolean option on docker and dockerfile.

          abayer Andrew Bayer added a comment - Merged! This'll be in the 0.9 release I'll be doing in the next couple days. Only syntax change is the addition of the reuseNode boolean option on docker and dockerfile .
          abayer Andrew Bayer made changes -
          Resolution Fixed [ 1 ]
          Status In Progress [ 3 ] Resolved [ 5 ]
          smazurov Stepan Mazurov added a comment - - edited

          Hey, abayer, thank you for adding this, I noticed that its not in syntax.md or the wiki (under changelog), might want to add that so other people can find it!

          smazurov Stepan Mazurov added a comment - - edited Hey, abayer , thank you for adding this, I noticed that its not in syntax.md or the wiki (under changelog), might want to add that so other people can find it!
          abayer Andrew Bayer added a comment - smazurov - how's https://github.com/jenkinsci/pipeline-model-definition-plugin/wiki/Controlling-your-build-environment#reusing-nodeworkspace-with-per-stage-docker-agents for now? I'll update SYNTAX.md as well.
          bitwiseman Liam Newman added a comment -

          Bulk closing resolved issues.

          bitwiseman Liam Newman added a comment - Bulk closing resolved issues.
          bitwiseman Liam Newman made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

          People

            abayer Andrew Bayer
            tomlarrow Tom Larrow
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: