Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45916

Path is not getting set correctly in pipeline when there is a variable present

      After upgrading to latest Pipeline version and it's dependencies (we also upgraded the blue ocean plugins but don't think that caused an issue here), we noticed that the PATH was not getting set in our Jenkins Pipeline Jobs. After debugging found that if I set PATH to a string inside the "environment" section, as such:

       

          environment

      {         PATH="/a/new/bin:$PATH"    }

       

      The PATH was updated successfully without any issues. However if I introduce a variable, the path does not get updated, and defaults to what is on the agent that runs the job. Example:

      pipeline

      {       agent

      {  label 'docker'    }

          environment {
              PATH="/a/new/bin:${env.SCM_HOME}:$PATH"
              SCM_HOME="${env.SCM_HOME}"
         }
        stages {
          stage ('build') {
            steps

      {         echo "PATH=$PATH"         echo "SCM_HOME=$SCM_HOME"       }

          }
        }
      }

       

      OUTPUT:

       
      [Pipeline] {

      [Pipeline] withEnv

      [Pipeline] {

      [Pipeline] stage

      [Pipeline] { (build)

      [Pipeline] echoPATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin

      [Pipeline] echoSCM_HOME=/usr/local/bin/scmtools/

      [Pipeline] }

      [Pipeline] // stage

      [Pipeline] }

      [Pipeline] // withEnv

      [Pipeline] }

      [Pipeline] // node

      [Pipeline] End of PipelineFinished: SUCCESS
       

       

      Notice that SCM_HOME resolves fine outside of PATH and that /a/new/bin was not added to the path. However if I drop the variable it will set PATH fine.

          [JENKINS-45916] Path is not getting set correctly in pipeline when there is a variable present

          Andrew Bayer added a comment -

          bbborro - please provide a Jenkinsfile that can reproduce this.

          Andrew Bayer added a comment - bbborro - please provide a Jenkinsfile that can reproduce this.

          Michael Musenbrock added a comment - - edited

          Hi, I'm not sure if this is was bbborro was referencing:

          pipeline {
              agent {
                  docker {
                      image 'alpine:latest'
                  }
              }
          
              environment {
                  PATHX = "${env.WORKSPACE}:${env.PATH}"
                  PATH = "${env.WORKSPACE}:${env.PATH}"
              }
          
              stages {
                  stage('Test') {
                      steps {
                          echo "PATH : ${env.PATH}"
                          echo "PATHX: ${env.PATHX}"
                          sh "printenv | grep PATH"
                      }
                  }
              }
          }

          Which prints:

          [Pipeline] echo PATH : /jenkins_slave/workspace/test:/usr/local/bin:/usr/bin:/bin:/usr/games
          [Pipeline] echo PATHX: /jenkins_slave/workspace/test:/jenkins_slave/workspace/test:/usr/local/bin:/usr/bin:/bin:/usr/games
          [Pipeline] sh
          [test] Running shell script
          + printenv
          + grep PATH
          PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
          CLASSPATH=
          PATHX=/jenkins_slave/workspace/test:/jenkins_slave/workspace/test:/usr/local/bin:/usr/bin:/bin:/usr/games
          

          So both variables are expanded correctly, but PATH is not passed to the environment of sh. Could also be a problem in the sh step.

          I'm also confused with the order of expansion: whether (in the environment stage) i put PATHX first or second, it seems that PATH gets always set first, because PATHX has the WORKSPACE variable content always prefixed twice

          Michael Musenbrock added a comment - - edited Hi, I'm not sure if this is was bbborro was referencing: pipeline { agent { docker { image 'alpine:latest' } } environment { PATHX = "${env.WORKSPACE}:${env.PATH}" PATH = "${env.WORKSPACE}:${env.PATH}" } stages { stage( 'Test' ) { steps { echo "PATH : ${env.PATH}" echo "PATHX: ${env.PATHX}" sh "printenv | grep PATH" } } } } Which prints: [Pipeline] echo PATH : /jenkins_slave/workspace/test:/usr/local/bin:/usr/bin:/bin:/usr/games [Pipeline] echo PATHX: /jenkins_slave/workspace/test:/jenkins_slave/workspace/test:/usr/local/bin:/usr/bin:/bin:/usr/games [Pipeline] sh [test] Running shell script + printenv + grep PATH PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin CLASSPATH= PATHX=/jenkins_slave/workspace/test:/jenkins_slave/workspace/test:/usr/local/bin:/usr/bin:/bin:/usr/games So both variables are expanded correctly, but PATH is not passed to the environment of sh. Could also be a problem in the sh step. I'm also confused with the order of expansion: whether (in the environment stage) i put PATHX first or second, it seems that PATH gets always set first, because PATHX has the WORKSPACE variable content always prefixed twice

          Hello, i managed to solve the problem. I think the problem was due to lack of knowledge from my part.

          Blanca Borro Escribano added a comment - Hello, i managed to solve the problem. I think the problem was due to lack of knowledge from my part.

          bbborro : I have the same issue , could you explain how you solve the problem ?

          Anthony Communier added a comment - bbborro : I have the same issue , could you explain how you solve the problem ?

          Anthony Communier the problem i had was that i was mixing the use of declarative pipeline with scripting pipeline. I finally used declarative pipeline for everything and found no problem.

          Blanca Borro Escribano added a comment - Anthony Communier the problem i had was that i was mixing the use of declarative pipeline with scripting pipeline. I finally used declarative pipeline for everything and found no problem.

          What does that mean for the mere mortals?  Is "steps" or "sh" the thing to avoid?

          Ilguiz Latypov added a comment - What does that mean for the mere mortals?  Is "steps" or "sh" the thing to avoid?

          Never mind, I see the magic recipe.  I still don't know the exact difference between "declarative" and "scripting", but I am mortal as well....

          the PATH+EXTRA system works for Scripted Pipeline. If it does not work for Declarative, please file a separate RFE in pipeline-model-definition-plugin. Workaround would I guess be something like (untested)

          environment {
            STUFF = "${MTI_HOME}/linux:${MTI_HOME}/bin:${QUARTUS_HOME}/bin:${DCP_LOC}/bin"
          }
          // …
          steps {
            withEnv(["PATH+EXTRA=$STUFF"]) {
              sh 'whatever'
            }
          }
          

          https://issues.jenkins-ci.org/browse/JENKINS-41339?focusedCommentId=357645&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-357645

           

          Ilguiz Latypov added a comment - Never mind, I see the magic recipe.  I still don't know the exact difference between "declarative" and "scripting", but I am mortal as well.... the PATH+EXTRA system works for Scripted Pipeline. If it does not work for Declarative, please file a separate RFE in pipeline-model-definition-plugin . Workaround would I guess be something like (untested) environment { STUFF = "${MTI_HOME}/linux:${MTI_HOME}/bin:${QUARTUS_HOME}/bin:${DCP_LOC}/bin" } // … steps { withEnv([ "PATH+EXTRA=$STUFF" ]) { sh 'whatever' } } https://issues.jenkins-ci.org/browse/JENKINS-41339?focusedCommentId=357645&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-357645  

          Ilguiz Latypov added a comment - - edited

          I figured the original issue appears fixed, and I am just confirming this for Jenkins 2.226 and current plugins. Sorry I just noticed Andrew Bayer's fix in this ticket.

          pipeline {
              agent {
                  label 'linux'
              }
          
              environment {
                  PATH="/a/new/bin:$PATH"
             }
             stages {
                  stage ('build') {
                      steps {
                          echo "PATH=$PATH"
                          echo "env.PATH=${env.PATH}"
                          sh "export -p | grep PATH"
                      }
                  }
              }
          }
          
           Running in Durability level: MAX_SURVIVABILITY
           [Pipeline] Start of Pipeline
           [Pipeline] node
           Running on Jenkins in /var/jenkins_home/workspace/System/Test_JENKINS-45916
           [Pipeline] {
           [Pipeline] withEnv
           [Pipeline] {
           [Pipeline] stage
           [Pipeline] { (build)
           [Pipeline] echo
           PATH=/a/new/bin:/usr/local/nvm/versions/node/v10.15.0/bin:/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
           [Pipeline] echo
           env.PATH=/a/new/bin:/usr/local/nvm/versions/node/v10.15.0/bin:/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
           [Pipeline] sh
           + export -p
           + grep PATH
           declare -x GEM_PATH="file:/var/jenkins_home/plugins/ruby-runtime/WEB-INF/lib/stapler-jruby-1.209.jar!/gem/"
           declare -x PATH="/a/new/bin:/usr/local/nvm/versions/node/v10.15.0/bin:/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
           [Pipeline] }
           [Pipeline] // stage
           [Pipeline] }
           [Pipeline] // withEnv
           [Pipeline] }
           [Pipeline] // node
           [Pipeline] End of Pipeline
           Finished: SUCCESS
          

          Ilguiz Latypov added a comment - - edited I figured the original issue appears fixed, and I am just confirming this for Jenkins 2.226 and current plugins. Sorry I just noticed Andrew Bayer's fix in this ticket. pipeline { agent { label 'linux' } environment { PATH= "/a/ new /bin:$PATH" } stages { stage ( 'build' ) { steps { echo "PATH=$PATH" echo "env.PATH=${env.PATH}" sh "export -p | grep PATH" } } } } Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Running on Jenkins in /var/jenkins_home/workspace/System/Test_JENKINS-45916 [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] stage [Pipeline] { (build) [Pipeline] echo PATH=/a/new/bin:/usr/local/nvm/versions/node/v10.15.0/bin:/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin [Pipeline] echo env.PATH=/a/new/bin:/usr/local/nvm/versions/node/v10.15.0/bin:/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin [Pipeline] sh + export -p + grep PATH declare -x GEM_PATH="file:/var/jenkins_home/plugins/ruby-runtime/WEB-INF/lib/stapler-jruby-1.209.jar!/gem/" declare -x PATH="/a/new/bin:/usr/local/nvm/versions/node/v10.15.0/bin:/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS

          Ilguiz Latypov added a comment - - edited

          The environment clause appears to be ignored by plugins executing code such as the following,

          EnvVars vars = build.getEnvironment(listener); 
          

          My current work-around is to define hard-coded default values for the pipeline "parameters", even for the agent "master" which runs on the same machine with the server. (If Jenkins server needs to run its own checkout of, say, a pipeline library, then it will fail to use Git on the server using the agent-targeting PATH).

              agent {
                  label 'master'
              }
              
              parameters {
                  string(name: 'FORTIFY_HOME', defaultValue: "/MYTOOLDIR/fortify", 
                      description: 'A work-around to the plugin using FORTIFY_HOME of the Jenkins server instead of that of the agent')
                  string(name: 'PATH', defaultValue: 
                      "/MYTOOLDIR/fortify/bin:/usr/local/FOO/bin:/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", 
                      description: 'A work-around to the plugin using PATH of the Jenkins server instead of that of the agent')
              }
          

          I wish Jenkins in general its plugins in particular gave precendence to the agent's environment. I did not try to find the exact API to fetch that in the context of the plugin running on the server but aiming the given "agent".

          Ilguiz Latypov added a comment - - edited The environment clause appears to be ignored by plugins executing code such as the following, EnvVars vars = build.getEnvironment(listener); My current work-around is to define hard-coded default values for the pipeline "parameters", even for the agent "master" which runs on the same machine with the server. (If Jenkins server needs to run its own checkout of, say, a pipeline library, then it will fail to use Git on the server using the agent-targeting PATH). agent { label 'master' } parameters { string(name: 'FORTIFY_HOME' , defaultValue: "/MYTOOLDIR/fortify" , description: 'A work-around to the plugin using FORTIFY_HOME of the Jenkins server instead of that of the agent' ) string(name: 'PATH' , defaultValue: "/MYTOOLDIR/fortify/bin:/usr/local/FOO/bin:/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" , description: 'A work-around to the plugin using PATH of the Jenkins server instead of that of the agent' ) } I wish Jenkins in general its plugins in particular gave precendence to the agent's environment. I did not try to find the exact API to fetch that in the context of the plugin running on the server but aiming the given "agent".

          citizenkahn added a comment -

          Folks, I didn't notice this workaround but it does seem viable.  I use script vs declarative so you might need to tweak this a bit.

           

          Yes, withEnv will not work within the inside closure.  Normally, the path variable will not be passed via a docker ENV method as that would usually be a bad idea given host OS and image OS are likely not the same.

           

          So, if we know what the images path is, then we can augment the path via the docker run args field `.inside(...)` to achieve the end of path modification.   It does mean that I must know or obtain the original path from the image.  I own the image and the pipeline code so I can easily do that.  If not, then I'd run an ephemeral docker container, and get the path from that via shell + returnStdout

           

          // hardcoded because I own the image or have determined what the image's path normally is
          def knownOriginalPath="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
          
          // my change to the path - in this case, a custom tool location used because I don't want to have to modify a bunch of existing pipeline code.
          def pathModification="/ansible2.10/bin"
          def desiredPath="${pathModification}:${knownOriginalPath}"
          
          
              docker.withRegistry(...) {
                 // for now, the plugin seems to respect the integrity of the inside option string.
                  docker.image(..)inside("-e PATH=${desiredPath}") {
                      sh 'echo PATH = $PATH'        
                  }
              } 

          citizenkahn added a comment - Folks, I didn't notice this workaround but it does seem viable.  I use script vs declarative so you might need to tweak this a bit.   Yes, withEnv will not work within the inside closure.  Normally, the path variable will not be passed via a docker ENV method as that would usually be a bad idea given host OS and image OS are likely not the same.   So, if we know what the images path is, then we can augment the path via the docker run args field `.inside(...)` to achieve the end of path modification.   It does mean that I must know or obtain the original path from the image.  I own the image and the pipeline code so I can easily do that.  If not, then I'd run an ephemeral docker container, and get the path from that via shell + returnStdout   // hardcoded because I own the image or have determined what the image's path normally is def knownOriginalPath= "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" // my change to the path - in this case , a custom tool location used because I don't want to have to modify a bunch of existing pipeline code. def pathModification= "/ansible2.10/bin" def desiredPath= "${pathModification}:${knownOriginalPath}" docker.withRegistry(...) { // for now, the plugin seems to respect the integrity of the inside option string. docker.image(..)inside( "-e PATH=${desiredPath}" ) { sh 'echo PATH = $PATH' } }

            abayer Andrew Bayer
            jammurp James Murphy
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: