• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • kubernetes-plugin
    • None

      WithEnv inside a container('x'){} is not working for us, when I did sh 'printenv' none of the variables were in the ENV.

          [JENKINS-40647] With Env not working after .10 k8 plugin update

          can you post a full example?

          Carlos Sanchez added a comment - can you post a full example?

          Mike Splain added a comment - - edited

          I'm seeing this as well, with both WithEnv and WithCredentials (while populating env vars)

          Simple example

          container('jnlp') {
            withEnv(['MYTOOL_HOME=/usr/local/mytool']) {
              sh 'env'
            }
          }
          

          In the output of env, 'MYTOOL_HOME' will not be set.

          Jenkins ver. 2.38, Jenkins k8s 0.10

          Mike Splain added a comment - - edited I'm seeing this as well, with both WithEnv and WithCredentials (while populating env vars) Simple example container( 'jnlp' ) { withEnv([ 'MYTOOL_HOME=/usr/local/mytool' ]) { sh 'env' } } In the output of env, 'MYTOOL_HOME' will not be set. Jenkins ver. 2.38, Jenkins k8s 0.10

          Mike Splain added a comment -

          Nevermind it just doesn't show up in env.... it's there using substitution like this

                  echo "${env.MYTOOL_HOME}"
          

          Mike Splain added a comment - Nevermind it just doesn't show up in env.... it's there using substitution like this echo "${env.MYTOOL_HOME}"

          Lars Lawoko added a comment - - edited
          runEnv = []
          runEnv.add('JAVA_OPTS=-Xmx6g')
          runEnv.add('npm_config_registry=http://nexus.default.svc.cluster.local:8081/repository/npm/')
          withEnv(runEnv) {
                try {
                     sh './gradlew --stacktrace --parallel jenkinsBuild'
                } catch (Exception e) {
                     junit allowEmptyResults: true, testResults: '**/build/test-results/**/*.xml'
                     step([$class: 'CheckStylePublisher', canComputeNew: false, defaultEncoding: '', healthy: '', pattern: '**/main.xml,**/test.xml', unHealthy: ''])
                     throw e
          }
          

          Adding a

          sh 'printenv'
          

          Shows the env is not populated

          P.s it is inside a container which contains JDK 7 & gcloud

           node('JavaPod') {
                          container('gcloud-jdk7') {
          

          Lars Lawoko added a comment - - edited runEnv = [] runEnv.add( 'JAVA_OPTS=-Xmx6g' ) runEnv.add( 'npm_config_registry=http: //nexus. default .svc.cluster.local:8081/repository/npm/' ) withEnv(runEnv) { try { sh './gradlew --stacktrace --parallel jenkinsBuild' } catch (Exception e) { junit allowEmptyResults: true , testResults: '**/build/test-results /**/ *.xml' step([$class: 'CheckStylePublisher' , canComputeNew: false , defaultEncoding: '', healthy: ' ', pattern: ' **/main.xml,**/test.xml ', unHealthy: ' ']) throw e } Adding a sh 'printenv' Shows the env is not populated P.s it is inside a container which contains JDK 7 & gcloud node( 'JavaPod' ) { container( 'gcloud-jdk7' ) {

          Lars Lawoko added a comment -

          Following up, can confirm it is a problem with the container step, The first printenv is correctly populated, but the second is not

          node('JavaPod') {
              withEnv(runEnv) {
                      sh 'printenv'
              }
              container('gcloud-jdk7') {
                      withEnv(runEnv) {
                              sh 'printenv'
                       } 
          }
          

          Lars Lawoko added a comment - Following up, can confirm it is a problem with the container step, The first printenv is correctly populated, but the second is not node( 'JavaPod' ) { withEnv(runEnv) { sh 'printenv' } container( 'gcloud-jdk7' ) { withEnv(runEnv) { sh 'printenv' } }

          Lars Lawoko added a comment -

          Can we confirm that this is a bug with the plugin, and not our setup? It requires ugly workarounds in its present state.

          Lars Lawoko added a comment - Can we confirm that this is a bug with the plugin, and not our setup? It requires ugly workarounds in its present state.

          It was never meant to work in the first place.

          The container block is only meant to be used with the `sh` function or shell based function.

          I would add then env variables in the pod definition or if that is not possible I would set them in the `sh` command.

          Ioannis Canellos added a comment - It was never meant to work in the first place. The container block is only meant to be used with the `sh` function or shell based function. I would add then env variables in the pod definition or if that is not possible I would set them in the `sh` command.

          Jon Whitcraft added a comment -

          iocanel if that is indeed the case, then the README should be updated to state that.

          Jon Whitcraft added a comment - iocanel if that is indeed the case, then the README should be updated to state that.

          Jon Whitcraft added a comment -

          So, just adding more to this issue, as the following plugins also don't work since they can't inject the environment variable correctly

           

          I'm sure there are others but these are the ones that I ran across today.  This is a huge issue as i shouldn't have to inject environment variables into shell scripts when there are other pipeline actions doing that for me.

          csanchez and iocanel do you agree?

          Jon Whitcraft added a comment - So, just adding more to this issue, as the following plugins also don't work since they can't inject the environment variable correctly   https://wiki.jenkins-ci.org/display/JENKINS/HashiCorp+Vault+Plugin https://wiki.jenkins-ci.org/display/JENKINS/Credentials+Binding+Plugin I'm sure there are others but these are the ones that I ran across today.  This is a huge issue as i shouldn't have to inject environment variables into shell scripts when there are other pipeline actions doing that for me. csanchez and iocanel do you agree?

          AFAIR, the readme does state, that in the container block you can just add sh or functions based on sh. If not, then we definitely need to fix that.

          Now, if there are ways, that allow us to overcome that limitation and also support other things, I am all for it.

          Ioannis Canellos added a comment - AFAIR, the readme does state, that in the container block you can just add sh or functions based on sh. If not, then we definitely need to fix that. Now, if there are ways, that allow us to overcome that limitation and also support other things, I am all for it.

          Jon Whitcraft added a comment -

          iocanel,

          Right, i think when it comes down to it, this is the real issue, JENKINS-42048 since it's not actually running the shell command in the way that durable-task requires, it's causing issues.  Because even this test which is in the global pipeline library help doesn't work

          podTemplate(
          	label: 'test',
          	containers: [
          		containerTemplate(
          			name: 'nodejs',
          			image: 'node:alpine',
          			ttyEnabled: true,
          			command: 'cat',
          			args: ''
          		)
          	]
          ) {
          	stage('test') {
          		node('test') {
          			env.MYTOOL_VERSION = '1.33'
          			container('nodejs') {
          				// printenv will not have MYTOOL_VERSION
          				'printenv'
          			}
          			echo env.MYTOOL_VERSION
          		}
          	}
          }
          

          I'm not familiar with Java enough to feel conformable to try and fix this, but IMO even this basic example should work.

          Jon Whitcraft added a comment - iocanel , Right, i think when it comes down to it, this is the real issue,  JENKINS-42048 since it's not actually running the shell command in the way that durable-task requires, it's causing issues.  Because even this test which is in the global pipeline library help doesn't work podTemplate( label: 'test' , containers: [ containerTemplate( name: 'nodejs' , image: 'node:alpine' , ttyEnabled: true , command: 'cat' , args: '' ) ] ) { stage( 'test' ) { node( 'test' ) { env.MYTOOL_VERSION = '1.33' container( 'nodejs' ) { // printenv will not have MYTOOL_VERSION 'printenv' } echo env.MYTOOL_VERSION } } } I'm not familiar with Java enough to feel conformable to try and fix this, but IMO even this basic example should work.

          I've got secret text credentials that I need to inject into containers. It's my understanding that the credentials helper isn't available within a container template or container environment variable context... So, how do we work around this limitation? 

          philip champon added a comment - I've got secret text credentials that I need to inject into containers. It's my understanding that the credentials helper isn't available within a container template or container environment variable context... So, how do we work around this limitation? 

          philip champon added a comment - - edited

          Injecting credentials as interpolated strings doesn't seems to work. It seems like the value of environment variables is ****. When I exec into the pod, I see the environment variable is correctly set. Log output from jenkins redacts the value.

          withCredentials([                                                 
            string(credentialsId: 'FOO', variable: 'FOO')
          ]) {
            def FOO = env.FOO            
            podTemplate(
          	label: 'test',
          	containers: [
          		containerTemplate(
          			name: 'nodejs',
          			image: 'node:alpine',
          			ttyEnabled: true,
          			command: 'cat',
          			args: ''
          		)
          	]
          ) {
          
                  node('test') {                                        
                    container('nodejs') {                                 
            
                      stage('env test') {                                
                        sh """
                          export FOO="$FOO"
                          echo \$FOO
                        """
                      }
                    }
                  }
                
              }
            }
          

          philip champon added a comment - - edited Injecting credentials as interpolated strings doesn't seems to work. It seems like the value of environment variables is **** . When I exec into the pod, I see the environment variable is correctly set. Log output from jenkins redacts the value. withCredentials([ string(credentialsId: 'FOO' , variable: 'FOO' ) ]) { def FOO = env.FOO podTemplate( label: 'test' , containers: [ containerTemplate( name: 'nodejs' , image: 'node:alpine' , ttyEnabled: true , command: 'cat' , args: '' ) ] ) { node( 'test' ) { container( 'nodejs' ) { stage( 'env test' ) { sh """ export FOO= "$FOO" echo \$FOO """ } } } } }

          Jon Whitcraft added a comment -

          pchampon, actually it is working, the credentials plugin forces the log to not actually display credentials, but instead display the ** as you are seeing.

          Jon Whitcraft added a comment - pchampon , actually it is working, the credentials plugin forces the log to not actually display credentials, but instead display the ** as you are seeing.

          Carlos Sanchez added a comment - This should be fixed with JENKINS-46278 https://github.com/jenkinsci/kubernetes-plugin/pull/204 https://github.com/jenkinsci/kubernetes-plugin/pull/205 and tests added in https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/test/resources/org/csanchez/jenkins/plugins/kubernetes/pipeline/runWithEnvVarsFromContext.groovy

          csanchez this issue still seems to be present in scripted pipelines when environment variables are defined inside the container step.

          This scenario (the same you have in your testcase) works:

          withEnv(['foo=1']) {
            container('container') {
              sh 'echo foo=$foo' // prints foo=1
            }
          }

          But this one doesn't:

          container('container') {
            withEnv(['foo=1']) {
              sh 'echo foo=$foo' // prints foo=
            }
          }

          Which also causes problems when using e.g. sshagent, as the sh step cannot see the SSH_AUTH_SOCK environment variable:

          container('container') {
            sshagent(['credential-id']) {
              sh 'ssh-add -l'
            }
          }

          Dániel Szoboszlay added a comment - csanchez this issue still seems to be present in scripted pipelines when environment variables are defined inside the container step. This scenario (the same you have in your testcase) works: withEnv([ 'foo=1' ]) { container( 'container' ) { sh 'echo foo=$foo' // prints foo=1 } } But this one doesn't: container( 'container' ) { withEnv([ 'foo=1' ]) { sh 'echo foo=$foo' // prints foo= } } Which also causes problems when using e.g. sshagent , as the sh step cannot see the SSH_AUTH_SOCK  environment variable: container( 'container' ) { sshagent([ 'credential-id' ]) { sh 'ssh-add -l' } }

          sshagent is supposed to be working, there is a test IIRC and some issues opened about it

          Not everything will be able to run inside container()

          Please open another issue if you can provide details

          Carlos Sanchez added a comment - sshagent is supposed to be working, there is a test IIRC and some issues opened about it Not everything will be able to run inside container() Please open another issue if you can provide details

          I opened JENKINS-49370 for my problem.

          > Not everything will be able to run inside container()

          Honestly, I would expect every step that works inside a node step to work inside a container step too. Is it documented somewhere which steps are supposed to work and which are not?

          Dániel Szoboszlay added a comment - I opened JENKINS-49370 for my problem. > Not everything will be able to run inside container() Honestly, I would expect every step that works inside a node step to work inside a  container step too. Is it documented somewhere which steps are supposed to work and which are not?

          not documented, but I can't really make sure that the 1000+ plugins will work as it uses a totally new way of executing things in the containers

          Carlos Sanchez added a comment - not documented, but I can't really make sure that the 1000+ plugins will work as it uses a totally new way of executing things in the containers

          I do think it would be worth making the core plugins work as well as possible with the plugin, otherwise it become's a huge hurdle migrating things to it. I've had to completely reorganize many Jenkinsfiles to use the plugin when it really could have just worked without that if this functionality was more complete. I agree this probably isn't the highest priority, but it would be nice if this could be fixed in one of the next few releases, I'm basically holding off on some upgrades because it would be a lot of work to make their use of withEnv work with kubernetes-plugin.

          Chance Zibolski added a comment - I do think it would be worth making the core plugins work as well as possible with the plugin, otherwise it become's a huge hurdle migrating things to it. I've had to completely reorganize many Jenkinsfiles to use the plugin when it really could have just worked without that if this functionality was more complete. I agree this probably isn't the highest priority, but it would be nice if this could be fixed in one of the next few releases, I'm basically holding off on some upgrades because it would be a lot of work to make their use of withEnv work with kubernetes-plugin.

          there have been several fixes in the last releases and master as people brought up PRs for things that were not working

          Carlos Sanchez added a comment - there have been several fixes in the last releases and master as people brought up PRs for things that were not working

          Cole Mickens added a comment - - edited

          1.3.2 seems to have solved the problems with `withEnv` being nested within `container() { ... }`.

          Previously, in a declarative pipeline, the job would execute as node `{ container { withEnv { ... } } }`, so I had to redundantly use a nested `container {}` inside every `stages/steps {}` to actualize the environment variables inside.

          Since updating the plugin earlier today, and seeing [the PR that was merged](https://github.com/jenkinsci/kubernetes-plugin/pull/291), I was able to remove that, and the environment variables are working as expected.

          Cole Mickens added a comment - - edited 1.3.2 seems to have solved the problems with `withEnv` being nested within `container() { ... }`. Previously, in a declarative pipeline, the job would execute as node `{ container { withEnv { ... } } }`, so I had to redundantly use a nested `container {}` inside every `stages/steps {}` to actualize the environment variables inside. Since updating the plugin earlier today, and seeing [the PR that was merged] ( https://github.com/jenkinsci/kubernetes-plugin/pull/291 ), I was able to remove that, and the environment variables are working as expected.

            csanchez Carlos Sanchez
            larslawoko Lars Lawoko
            Votes:
            7 Vote for this issue
            Watchers:
            19 Start watching this issue

              Created:
              Updated:
              Resolved: