Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45916

Path is not getting set correctly in pipeline when there is a variable present

      After upgrading to latest Pipeline version and it's dependencies (we also upgraded the blue ocean plugins but don't think that caused an issue here), we noticed that the PATH was not getting set in our Jenkins Pipeline Jobs. After debugging found that if I set PATH to a string inside the "environment" section, as such:

       

          environment

      {         PATH="/a/new/bin:$PATH"    }

       

      The PATH was updated successfully without any issues. However if I introduce a variable, the path does not get updated, and defaults to what is on the agent that runs the job. Example:

      pipeline

      {       agent

      {  label 'docker'    }

          environment {
              PATH="/a/new/bin:${env.SCM_HOME}:$PATH"
              SCM_HOME="${env.SCM_HOME}"
         }
        stages {
          stage ('build') {
            steps

      {         echo "PATH=$PATH"         echo "SCM_HOME=$SCM_HOME"       }

          }
        }
      }

       

      OUTPUT:

       
      [Pipeline] {

      [Pipeline] withEnv

      [Pipeline] {

      [Pipeline] stage

      [Pipeline] { (build)

      [Pipeline] echoPATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin

      [Pipeline] echoSCM_HOME=/usr/local/bin/scmtools/

      [Pipeline] }

      [Pipeline] // stage

      [Pipeline] }

      [Pipeline] // withEnv

      [Pipeline] }

      [Pipeline] // node

      [Pipeline] End of PipelineFinished: SUCCESS
       

       

      Notice that SCM_HOME resolves fine outside of PATH and that /a/new/bin was not added to the path. However if I drop the variable it will set PATH fine.

          [JENKINS-45916] Path is not getting set correctly in pipeline when there is a variable present

          Andrew Bayer added a comment -

          I think this'll be resolved as part of https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/174. I'll add relevant tests there tomorrow.

          Andrew Bayer added a comment - I think this'll be resolved as part of https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/174 . I'll add relevant tests there tomorrow.

          Andrew Bayer added a comment -

          Welp, turns out now this hangs indefinitely. Good times - working on it.

          Andrew Bayer added a comment - Welp, turns out now this hangs indefinitely. Good times - working on it.

          Andrew Bayer added a comment -

          Ok, figured out the problem in the JENKINS-42753 pull request - the underlying problem you're seeing was already fixed there, but in the process I introduced another problem when you were trying to set an environment variable to a value that included that variable's previous value! I've pushed the fix for that to the JENKINS-42753 PR, so this'll be fully resolved in Declarative 1.2.

          Andrew Bayer added a comment - Ok, figured out the problem in the JENKINS-42753 pull request - the underlying problem you're seeing was already fixed there, but in the process I introduced another problem when you were trying to set an environment variable to a value that included that variable's previous value! I've pushed the fix for that to the JENKINS-42753 PR, so this'll be fully resolved in Declarative 1.2.

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/RuntimeASTTransformer.groovy
          pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/EnvironmentTest.java
          pipeline-model-definition/src/test/resources/pathInEnv.groovy
          http://jenkins-ci.org/commit/pipeline-model-definition-plugin/379db1a5f7ea0437bb2ac13eeace7b1e414cb218
          Log:
          [FIXED JENKINS-45916] Properly handle overwriting+including existing vars

          Technically, JENKINS-45916 is already fixed by earlier changes here,
          but in the process we introduced a hellish break due to trying to call
          the closure for getting the value of a variable when it's referenced
          in setting the variable in the first place. So if we're setting FOO
          and the value includes FOO, just use the original expression for that
          instead of closure-calling there.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/RuntimeASTTransformer.groovy pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/EnvironmentTest.java pipeline-model-definition/src/test/resources/pathInEnv.groovy http://jenkins-ci.org/commit/pipeline-model-definition-plugin/379db1a5f7ea0437bb2ac13eeace7b1e414cb218 Log: [FIXED JENKINS-45916] Properly handle overwriting+including existing vars Technically, JENKINS-45916 is already fixed by earlier changes here, but in the process we introduced a hellish break due to trying to call the closure for getting the value of a variable when it's referenced in setting the variable in the first place. So if we're setting FOO and the value includes FOO, just use the original expression for that instead of closure-calling there.

          Good morning, it seems this is happening to me again in Jenkins ver. 2.73.2, no matter what value i try to set to the PATH variable it always get the same value:

          echo PATH is: /sbin:/usr/sbin:/bin:/usr/bin

          the other variables set in the environment are being set correctly, only path is not being set.

           

           

          Blanca Borro Escribano added a comment - Good morning, it seems this is happening to me again in Jenkins ver. 2.73.2 , no matter what value i try to set to the PATH variable it always get the same value: echo PATH is: /sbin:/usr/sbin:/bin:/usr/bin the other variables set in the environment are being set correctly, only path is not being set.    

          Andrew Bayer added a comment -

          bbborro - please provide a Jenkinsfile that can reproduce this.

          Andrew Bayer added a comment - bbborro - please provide a Jenkinsfile that can reproduce this.

          Michael Musenbrock added a comment - - edited

          Hi, I'm not sure if this is was bbborro was referencing:

          pipeline {
              agent {
                  docker {
                      image 'alpine:latest'
                  }
              }
          
              environment {
                  PATHX = "${env.WORKSPACE}:${env.PATH}"
                  PATH = "${env.WORKSPACE}:${env.PATH}"
              }
          
              stages {
                  stage('Test') {
                      steps {
                          echo "PATH : ${env.PATH}"
                          echo "PATHX: ${env.PATHX}"
                          sh "printenv | grep PATH"
                      }
                  }
              }
          }

          Which prints:

          [Pipeline] echo PATH : /jenkins_slave/workspace/test:/usr/local/bin:/usr/bin:/bin:/usr/games
          [Pipeline] echo PATHX: /jenkins_slave/workspace/test:/jenkins_slave/workspace/test:/usr/local/bin:/usr/bin:/bin:/usr/games
          [Pipeline] sh
          [test] Running shell script
          + printenv
          + grep PATH
          PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
          CLASSPATH=
          PATHX=/jenkins_slave/workspace/test:/jenkins_slave/workspace/test:/usr/local/bin:/usr/bin:/bin:/usr/games
          

          So both variables are expanded correctly, but PATH is not passed to the environment of sh. Could also be a problem in the sh step.

          I'm also confused with the order of expansion: whether (in the environment stage) i put PATHX first or second, it seems that PATH gets always set first, because PATHX has the WORKSPACE variable content always prefixed twice

          Michael Musenbrock added a comment - - edited Hi, I'm not sure if this is was bbborro was referencing: pipeline { agent { docker { image 'alpine:latest' } } environment { PATHX = "${env.WORKSPACE}:${env.PATH}" PATH = "${env.WORKSPACE}:${env.PATH}" } stages { stage( 'Test' ) { steps { echo "PATH : ${env.PATH}" echo "PATHX: ${env.PATHX}" sh "printenv | grep PATH" } } } } Which prints: [Pipeline] echo PATH : /jenkins_slave/workspace/test:/usr/local/bin:/usr/bin:/bin:/usr/games [Pipeline] echo PATHX: /jenkins_slave/workspace/test:/jenkins_slave/workspace/test:/usr/local/bin:/usr/bin:/bin:/usr/games [Pipeline] sh [test] Running shell script + printenv + grep PATH PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin CLASSPATH= PATHX=/jenkins_slave/workspace/test:/jenkins_slave/workspace/test:/usr/local/bin:/usr/bin:/bin:/usr/games So both variables are expanded correctly, but PATH is not passed to the environment of sh. Could also be a problem in the sh step. I'm also confused with the order of expansion: whether (in the environment stage) i put PATHX first or second, it seems that PATH gets always set first, because PATHX has the WORKSPACE variable content always prefixed twice

          Hello, i managed to solve the problem. I think the problem was due to lack of knowledge from my part.

          Blanca Borro Escribano added a comment - Hello, i managed to solve the problem. I think the problem was due to lack of knowledge from my part.

          bbborro : I have the same issue , could you explain how you solve the problem ?

          Anthony Communier added a comment - bbborro : I have the same issue , could you explain how you solve the problem ?

          Anthony Communier the problem i had was that i was mixing the use of declarative pipeline with scripting pipeline. I finally used declarative pipeline for everything and found no problem.

          Blanca Borro Escribano added a comment - Anthony Communier the problem i had was that i was mixing the use of declarative pipeline with scripting pipeline. I finally used declarative pipeline for everything and found no problem.

          What does that mean for the mere mortals?  Is "steps" or "sh" the thing to avoid?

          Ilguiz Latypov added a comment - What does that mean for the mere mortals?  Is "steps" or "sh" the thing to avoid?

          Never mind, I see the magic recipe.  I still don't know the exact difference between "declarative" and "scripting", but I am mortal as well....

          the PATH+EXTRA system works for Scripted Pipeline. If it does not work for Declarative, please file a separate RFE in pipeline-model-definition-plugin. Workaround would I guess be something like (untested)

          environment {
            STUFF = "${MTI_HOME}/linux:${MTI_HOME}/bin:${QUARTUS_HOME}/bin:${DCP_LOC}/bin"
          }
          // …
          steps {
            withEnv(["PATH+EXTRA=$STUFF"]) {
              sh 'whatever'
            }
          }
          

          https://issues.jenkins-ci.org/browse/JENKINS-41339?focusedCommentId=357645&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-357645

           

          Ilguiz Latypov added a comment - Never mind, I see the magic recipe.  I still don't know the exact difference between "declarative" and "scripting", but I am mortal as well.... the PATH+EXTRA system works for Scripted Pipeline. If it does not work for Declarative, please file a separate RFE in pipeline-model-definition-plugin . Workaround would I guess be something like (untested) environment { STUFF = "${MTI_HOME}/linux:${MTI_HOME}/bin:${QUARTUS_HOME}/bin:${DCP_LOC}/bin" } // … steps { withEnv([ "PATH+EXTRA=$STUFF" ]) { sh 'whatever' } } https://issues.jenkins-ci.org/browse/JENKINS-41339?focusedCommentId=357645&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-357645  

          Ilguiz Latypov added a comment - - edited

          I figured the original issue appears fixed, and I am just confirming this for Jenkins 2.226 and current plugins. Sorry I just noticed Andrew Bayer's fix in this ticket.

          pipeline {
              agent {
                  label 'linux'
              }
          
              environment {
                  PATH="/a/new/bin:$PATH"
             }
             stages {
                  stage ('build') {
                      steps {
                          echo "PATH=$PATH"
                          echo "env.PATH=${env.PATH}"
                          sh "export -p | grep PATH"
                      }
                  }
              }
          }
          
           Running in Durability level: MAX_SURVIVABILITY
           [Pipeline] Start of Pipeline
           [Pipeline] node
           Running on Jenkins in /var/jenkins_home/workspace/System/Test_JENKINS-45916
           [Pipeline] {
           [Pipeline] withEnv
           [Pipeline] {
           [Pipeline] stage
           [Pipeline] { (build)
           [Pipeline] echo
           PATH=/a/new/bin:/usr/local/nvm/versions/node/v10.15.0/bin:/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
           [Pipeline] echo
           env.PATH=/a/new/bin:/usr/local/nvm/versions/node/v10.15.0/bin:/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
           [Pipeline] sh
           + export -p
           + grep PATH
           declare -x GEM_PATH="file:/var/jenkins_home/plugins/ruby-runtime/WEB-INF/lib/stapler-jruby-1.209.jar!/gem/"
           declare -x PATH="/a/new/bin:/usr/local/nvm/versions/node/v10.15.0/bin:/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
           [Pipeline] }
           [Pipeline] // stage
           [Pipeline] }
           [Pipeline] // withEnv
           [Pipeline] }
           [Pipeline] // node
           [Pipeline] End of Pipeline
           Finished: SUCCESS
          

          Ilguiz Latypov added a comment - - edited I figured the original issue appears fixed, and I am just confirming this for Jenkins 2.226 and current plugins. Sorry I just noticed Andrew Bayer's fix in this ticket. pipeline { agent { label 'linux' } environment { PATH= "/a/ new /bin:$PATH" } stages { stage ( 'build' ) { steps { echo "PATH=$PATH" echo "env.PATH=${env.PATH}" sh "export -p | grep PATH" } } } } Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Running on Jenkins in /var/jenkins_home/workspace/System/Test_JENKINS-45916 [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] stage [Pipeline] { (build) [Pipeline] echo PATH=/a/new/bin:/usr/local/nvm/versions/node/v10.15.0/bin:/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin [Pipeline] echo env.PATH=/a/new/bin:/usr/local/nvm/versions/node/v10.15.0/bin:/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin [Pipeline] sh + export -p + grep PATH declare -x GEM_PATH="file:/var/jenkins_home/plugins/ruby-runtime/WEB-INF/lib/stapler-jruby-1.209.jar!/gem/" declare -x PATH="/a/new/bin:/usr/local/nvm/versions/node/v10.15.0/bin:/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS

          Ilguiz Latypov added a comment - - edited

          The environment clause appears to be ignored by plugins executing code such as the following,

          EnvVars vars = build.getEnvironment(listener); 
          

          My current work-around is to define hard-coded default values for the pipeline "parameters", even for the agent "master" which runs on the same machine with the server. (If Jenkins server needs to run its own checkout of, say, a pipeline library, then it will fail to use Git on the server using the agent-targeting PATH).

              agent {
                  label 'master'
              }
              
              parameters {
                  string(name: 'FORTIFY_HOME', defaultValue: "/MYTOOLDIR/fortify", 
                      description: 'A work-around to the plugin using FORTIFY_HOME of the Jenkins server instead of that of the agent')
                  string(name: 'PATH', defaultValue: 
                      "/MYTOOLDIR/fortify/bin:/usr/local/FOO/bin:/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", 
                      description: 'A work-around to the plugin using PATH of the Jenkins server instead of that of the agent')
              }
          

          I wish Jenkins in general its plugins in particular gave precendence to the agent's environment. I did not try to find the exact API to fetch that in the context of the plugin running on the server but aiming the given "agent".

          Ilguiz Latypov added a comment - - edited The environment clause appears to be ignored by plugins executing code such as the following, EnvVars vars = build.getEnvironment(listener); My current work-around is to define hard-coded default values for the pipeline "parameters", even for the agent "master" which runs on the same machine with the server. (If Jenkins server needs to run its own checkout of, say, a pipeline library, then it will fail to use Git on the server using the agent-targeting PATH). agent { label 'master' } parameters { string(name: 'FORTIFY_HOME' , defaultValue: "/MYTOOLDIR/fortify" , description: 'A work-around to the plugin using FORTIFY_HOME of the Jenkins server instead of that of the agent' ) string(name: 'PATH' , defaultValue: "/MYTOOLDIR/fortify/bin:/usr/local/FOO/bin:/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" , description: 'A work-around to the plugin using PATH of the Jenkins server instead of that of the agent' ) } I wish Jenkins in general its plugins in particular gave precendence to the agent's environment. I did not try to find the exact API to fetch that in the context of the plugin running on the server but aiming the given "agent".

          citizenkahn added a comment -

          Folks, I didn't notice this workaround but it does seem viable.  I use script vs declarative so you might need to tweak this a bit.

           

          Yes, withEnv will not work within the inside closure.  Normally, the path variable will not be passed via a docker ENV method as that would usually be a bad idea given host OS and image OS are likely not the same.

           

          So, if we know what the images path is, then we can augment the path via the docker run args field `.inside(...)` to achieve the end of path modification.   It does mean that I must know or obtain the original path from the image.  I own the image and the pipeline code so I can easily do that.  If not, then I'd run an ephemeral docker container, and get the path from that via shell + returnStdout

           

          // hardcoded because I own the image or have determined what the image's path normally is
          def knownOriginalPath="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
          
          // my change to the path - in this case, a custom tool location used because I don't want to have to modify a bunch of existing pipeline code.
          def pathModification="/ansible2.10/bin"
          def desiredPath="${pathModification}:${knownOriginalPath}"
          
          
              docker.withRegistry(...) {
                 // for now, the plugin seems to respect the integrity of the inside option string.
                  docker.image(..)inside("-e PATH=${desiredPath}") {
                      sh 'echo PATH = $PATH'        
                  }
              } 

          citizenkahn added a comment - Folks, I didn't notice this workaround but it does seem viable.  I use script vs declarative so you might need to tweak this a bit.   Yes, withEnv will not work within the inside closure.  Normally, the path variable will not be passed via a docker ENV method as that would usually be a bad idea given host OS and image OS are likely not the same.   So, if we know what the images path is, then we can augment the path via the docker run args field `.inside(...)` to achieve the end of path modification.   It does mean that I must know or obtain the original path from the image.  I own the image and the pipeline code so I can easily do that.  If not, then I'd run an ephemeral docker container, and get the path from that via shell + returnStdout   // hardcoded because I own the image or have determined what the image's path normally is def knownOriginalPath= "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" // my change to the path - in this case , a custom tool location used because I don't want to have to modify a bunch of existing pipeline code. def pathModification= "/ansible2.10/bin" def desiredPath= "${pathModification}:${knownOriginalPath}" docker.withRegistry(...) { // for now, the plugin seems to respect the integrity of the inside option string. docker.image(..)inside( "-e PATH=${desiredPath}" ) { sh 'echo PATH = $PATH' } }

            abayer Andrew Bayer
            jammurp James Murphy
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: