Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48082

Environment variables missing when both defined in pipeline and jenkins master process environment with docker agent

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • docker-workflow-plugin
    • None
    • jenkins 2.73.3, from LTS docker image
      host: ubuntu 16.04 with docker-ce 17.09

      When defining environment variables both in a declarative pipeline running a docker agent, and in the jenkins master process environment itself, then the environment variable is missing in the docker agent.

      Scenario:

       

      Jenkins setup:

      Using jenkins:lts docker image, with docker socket forwarded in the jenkins docker container for it to use docker agents; with extra environment variables:

      $ docker run -e FOO=bar -e FOO2=bar -e FOO3=bar -v /var/run/docker.sock:/var/run/docker.sock -v $(which docker):/usr/bin/docker --init -v jenkins_home:/var/jenkins_home -p 8080:8080 -p 50000:50000 --name jenkins jenkins/jenkins:lts
      # hack to add missing depencies on the docker cli, and fix permissions issues
      $ docker exec -u 0 jenkins bash -c "apt-get update && apt-get install libltdl7 && chown -R jenkins:jenkins /var/run/docker.sock"
      

      Setup jenkins with usual plugins (no manual selection).

      Pipeline job:

      Create a pipeline with the following declarative pipeline:

      pipeline {
          agent { docker 'ubuntu:16.04' }
      
          environment {
              FOO = 'bar'
              FOO2 = 'different'
          }
          stages {
              stage('Test') {
                  steps {
                      sh 'echo FOO: ${FOO:-error}'
                      sh 'echo FOO2: ${FOO2:-error}'
                      sh 'echo FOO3: ${FOO3:-error}'
                  }
              }
          }
      }
      

      Run the job

      [env-bug] Running shell script
      + echo FOO: error
      FOO: error
      [Pipeline] sh
      [env-bug] Running shell script
      + echo FOO2: different
      FOO2: different
      [Pipeline] sh
      [env-bug] Running shell script
      + echo FOO3: error
      FOO3: error
      

      Analysis

      Scenario explanation:

      • start jenkins with env vars FOO=bar and FOO2=bar and FOO3=bar
      • pipeline redeclares FOO=bar (same value) and FOO3=different (different value)
      • result: FOO disappears in the shell environment in docker agent, FOO2 is OK, FOO3 is not there (not forwarded from jenkins environment, it's an extra test).

          [JENKINS-48082] Environment variables missing when both defined in pipeline and jenkins master process environment with docker agent

          Andrew Bayer added a comment -

          Well, that's a weird combination right there. I'm guessing that master-level env vars aren't getting passed into the docker agent, and it's seeing FOO=bar as still the master-level env var. Needs some experimentation.

          Andrew Bayer added a comment - Well, that's a weird combination right there. I'm guessing that master-level env vars aren't getting passed into the docker agent, and it's seeing FOO=bar as still the master-level env var. Needs some experimentation.

          That's indeed not the most standard setup, but that's what we use, and it worked great up to now.

          I also tested without the docker agent, ie directly on the master, and the issue disappears.

           

          My current workaround: don't use the env var names already defined in the master environment when constructing the pipeline environment, and copy back to correct env var name in `sh`:

           

          environment {
                  _FOO = 'bar'
          }
          ...
          sh 'FOO=${_FOO} echo FOO: ${FOO:-error}'
          

           

           

          Thomas Riccardi added a comment - That's indeed not the most standard setup, but that's what we use, and it worked great up to now. I also tested without the docker agent, ie directly on the master, and the issue disappears.   My current workaround: don't use the env var names already defined in the master environment when constructing the pipeline environment, and copy back to correct env var name in `sh`:   environment { _FOO = 'bar' } ... sh 'FOO=${_FOO} echo FOO: ${FOO:-error}'    

          Andrew Bayer added a comment -

          A question: is there an actual agent defined in Jenkins that you're running on, or are you just running with the docker agent in Declarative?

          Andrew Bayer added a comment - A question: is there an actual agent defined in Jenkins that you're running on, or are you just running with the docker agent in Declarative?

          I've not defined any explicit agent in Jenkins: default setup (using the jenkins docker image).

          Thomas Riccardi added a comment - I've not defined any explicit agent in Jenkins: default setup (using the jenkins docker image).

          Andrew Bayer added a comment -

          Gotcha - thanks!

          Andrew Bayer added a comment - Gotcha - thanks!

          Andrew Bayer added a comment -

          Correcting the component - this is an issue in docker-workflow, which is hairier to deal with. I reproduced this in a unit test with a Scripted Pipeline:

          node {
              docker.image('ubuntu:16.04').inside {
                  withEnv(["FOO=bar","FOO2=different"]) {
                      sh 'echo FOO: ${FOO:-error}'
                      sh 'echo FOO2: ${FOO2:-error}'
                      sh 'echo FOO3: ${FOO3:-error}'
                  }
              }
          }

          Specifically, docker.image.inside (which is what's happening behind the scenes with docker 'ubuntu:16.04' in your example) is removing every environment variable defined on the agent we're running Docker on (in both cases, the master) where the variable name + value entry is the same in our current environment as well. So FOO=bar and FOO3=bar get removed, but FOO2=different doesn't, because the master's got FOO2=bar while our current environment (within the withEnv in my example, which is also what environment in your example turns into behind the scenes) has FOO2=different.

          This is...annoying. And I'm not sure there's a good solution.

          Andrew Bayer added a comment - Correcting the component - this is an issue in docker-workflow , which is hairier to deal with. I reproduced this in a unit test with a Scripted Pipeline: node { docker.image( 'ubuntu:16.04' ).inside { withEnv([ "FOO=bar" , "FOO2=different" ]) { sh 'echo FOO: ${FOO:-error}' sh 'echo FOO2: ${FOO2:-error}' sh 'echo FOO3: ${FOO3:-error}' } } } Specifically, docker.image.inside (which is what's happening behind the scenes with docker 'ubuntu:16.04' in your example) is removing every environment variable defined on the agent we're running Docker on (in both cases, the master) where the variable name + value entry is the same in our current environment as well. So FOO=bar and FOO3=bar get removed, but FOO2=different doesn't, because the master's got FOO2=bar while our current environment (within the withEnv in my example, which is also what environment in your example turns into behind the scenes) has FOO2=different . This is...annoying. And I'm not sure there's a good solution.

            Unassigned Unassigned
            thomas_deepomatic Thomas Riccardi
            Votes:
            3 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: