Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43590

docker.inside() breaks container environment.

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • docker-workflow-plugin
    • None
    • docker-workflow 1.10

      We have PATH set on our build slaves (via configure -> Node properties -> Environment Variables) that won't work in most docker containers.  Specifically, we use /usr/local/bin:/a/bin.  Note how /usr/bin and /bin aren't in the PATH (which is where cat usually is located.

      Normally, the docker.inside() command's docker run isn't setting the -e PATH=... flag.  But every so often it uses the PATH from the build slave configuration which causes the docker run to not find cat

       

      docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \"cat\": executable file not found in $PATH".

       

      I was able to track this down by modifying docker-workflow to not mask the PATH: https://github.com/jenkinsci/docker-workflow-plugin/pull/96

      I don't know what the expected behavior (overriding the PATH on a container seems like a "Bad Idea", since the container may-or-may-not be in LSB/POSIX layout) is supposed to be.

      It is clear that this random flipping between behaviors is bad.

          [JENKINS-43590] docker.inside() breaks container environment.

          Yeah, there are a bunch of environment variables that can potentially break a container.  And you can't "pre-determine" the list of environment variables that may break a container before hand. 

          Christian Höltje added a comment - Yeah, there are a bunch of environment variables that can potentially break a container.  And you can't "pre-determine" the list of environment variables that may break a container before hand. 

          Christian Höltje added a comment - - edited

          jglick: At the very least, do you have any idea why this code isn't working?

          EnvVars envReduced = new EnvVars(env);
          EnvVars envHost = computer.getEnvironment();
          envReduced.entrySet().removeAll(envHost.entrySet());

          ...because in my case, I have PATH set on the agent/node/slave; if this code worked it would unset PATH which would at least let cat work again for me. 

          Christian Höltje added a comment - - edited jglick : At the very least, do you have any idea why this code isn't working? EnvVars envReduced = new EnvVars(env); EnvVars envHost = computer.getEnvironment(); envReduced.entrySet().removeAll(envHost.entrySet()); ...because in my case, I have PATH set on the agent/node/slave; if this code worked it would unset PATH which would at least let cat work again for me. 

          Jesse Glick added a comment -

          Not sure offhand, would need to spend time studying it.

          Generally I would treat Image.inside as “take it or leave it”—if it works as a convenience, great; if not, ignore it and use generic techniques such as calling docker run to run desired steps (and docker cp to extra any files needed for archiving or reporting).

          Jesse Glick added a comment - Not sure offhand, would need to spend time studying it. Generally I would treat Image.inside as “take it or leave it”—if it works as a convenience, great; if not, ignore it and use generic techniques such as calling docker run to run desired steps (and docker cp to extra any files needed for archiving or reporting).

          James Johnson added a comment -

          This is an issue for me as well.  The composer docker image has the php binary in /usr/local/bin.  I've tried setting PATH via inside('-e PATH') without luck.  I was able to get around it at the moment by setting the PATH before calling the composer command.  

          Fails:

          docker.image("composer:latest").inside{
            sh 'composer install -a --no-dev --ignore-platform-reqs --no-scripts --no-interaction --no-ansi --no-progress'
          }
          

          Works:

          docker.image("composer:latest").inside{
            sh 'PATH=$PATH:/usr/local/bin composer install -a --no-dev --ignore-platform-reqs --no-scripts --no-interaction --no-ansi --no-progress'
          }
          

           

          James Johnson added a comment - This is an issue for me as well.  The composer docker image has the php binary in /usr/local/bin.  I've tried setting PATH via inside('-e PATH') without luck.  I was able to get around it at the moment by setting the PATH before calling the composer command.   Fails: docker.image( "composer:latest" ).inside{ sh 'composer install -a --no-dev --ignore-platform-reqs --no-scripts --no-interaction --no-ansi --no-progress' } Works: docker.image( "composer:latest" ).inside{ sh 'PATH=$PATH:/usr/local/bin composer install -a --no-dev --ignore-platform-reqs --no-scripts --no-interaction --no-ansi --no-progress' }  

          Christian Höltje added a comment - - edited

          It looks like multi-stage docker builds are a way around this, and they work on a workstation the same as in Jenkins.

          Note: the documentation doesn't mention this, but you can use --target= to pick which image gets tagged by the build.

          Example:

           

          docker build --target=builder --tag=build-env .
          

          ...would tag the builder from the multi-stage build so you could examine it.

          Christian Höltje added a comment - - edited It looks like multi-stage docker builds are a way around this, and they work on a workstation the same as in Jenkins. Note: the documentation doesn't mention this, but you can use --target= to pick which image gets tagged by the build. Example:   docker build --target=builder --tag=build-env . ...would tag the builder from the multi-stage build so you could examine it.

          Jesse Glick added a comment -

          Yes, though there are some limitations to that.

          Jesse Glick added a comment - Yes, though there are some limitations to that .

          Code changed in jenkins
          User: Christian Höltje
          Path:
          src/main/java/org/jenkinsci/plugins/docker/workflow/WithContainerStep.java
          http://jenkins-ci.org/commit/docker-workflow-plugin/e04959ea87d391ec89965c760b9d054b73bed317
          Log:
          Don't pass PATH to docker containers

          Docker containers usually have special PATHs; the PATH defined
          in on the Agent won't make sense for the container.

          JENKINS-43590 docker.inside()'s -e flags are random

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christian Höltje Path: src/main/java/org/jenkinsci/plugins/docker/workflow/WithContainerStep.java http://jenkins-ci.org/commit/docker-workflow-plugin/e04959ea87d391ec89965c760b9d054b73bed317 Log: Don't pass PATH to docker containers Docker containers usually have special PATHs; the PATH defined in on the Agent won't make sense for the container. JENKINS-43590 docker.inside()'s -e flags are random

          I think blocking PATH isn't a correct fix.

          Environment is built from various sources, including Computer.getEnvironment(), which returns node's `PATH`.

          Tool installers do amend this using `PATH+` syntax

          docker-pipeline should replace Computer.getEnvironment() result with the actual `env` from container, so this mechanism works as expected.

          Nicolas De Loof added a comment - I think blocking PATH isn't a correct fix. Environment is built from various sources, including Computer.getEnvironment(), which returns node's `PATH`. Tool installers do amend this using `PATH+` syntax docker-pipeline should replace Computer.getEnvironment() result with the actual `env` from container, so this mechanism works as expected.

          ndeloof

          I didn't even know that was possible, but it makes sense.  Do we need a new issue for that?

          Christian Höltje added a comment - ndeloof I didn't even know that was possible, but it makes sense.  Do we need a new issue for that?

          Rong Shen added a comment -

          I agree with ndeloof

          This code change actually break our pipeline that passes PATH environment from hosts.

          Docker container path should be an optional overwrite, instead of blocking!

          Rong Shen added a comment - I agree with ndeloof This code change actually break our pipeline that passes PATH environment from hosts. Docker container path should be an optional overwrite, instead of blocking!

            Unassigned Unassigned
            docwhat Christian Höltje
            Votes:
            12 Vote for this issue
            Watchers:
            21 Start watching this issue

              Created:
              Updated: