If I symlink /bin/sh to my toolchain path and change env to /bin/env, the output shows the shell using this path variable:

PATH=/path/to/toolchain/bin:$PATH - just like the groovy output.

Cannot reproduce on a pristine installation of Jenkins 2.40 with current releases of Pipeline and its dependencies (installed 10 minutes ago).

Started by user admin
[Pipeline] node
Running on master in /Users/danielbeck/JENKINS-41339-Home/workspace/pipe
[Pipeline] {
[Pipeline] echo
env.PATH=/Users/danielbeck/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/groovy/bin:/opt/X11/bin:/usr/local/MacGPG2/bin
[Pipeline] sh
[pipe] Running shell script
+ env
…

Let me guess, envinject is installed and you're doing something insane like undefining all node environment variables?

No envinject. This is a pretty vanilla Jenkins install, just pipeline and blue ocean. For fun, I created a digitalocean droplet, installed jenkins, and reproduced it. http://138.197.195.70:8080/ (L/P jenkinsbug:jenkinsbug)

Did you setup the path in the right place? Mine is under Manage Jenkins > Configure System > Global Properties > Environment variables

Same happening here:

• CentOS 7.1
• Jenkins ver. 2.42
• Durable Task plugin 1.13

[workspace] Running shell script
nohup: failed to run command ‘sh’: No such file or directory

It happens with any shell script call.
If I downgrade Durable Task plugin to 1.12 it, it works.

No, I did not specify a path in the global configuration, I missed that part.

Will investigate more later.

This seams to be an issue after JENKINS-40734

I can reproduce this as well :

Debian 8
Jenkins 2.32.1
Durable task plugin 1.13

Reverting to durable task plugin 1.12 fixes the issue.

Reproduced on 2.19.4 with newest Pipeline plugins.

If and only if I set a custom PATH for the node the sh step runs on, the PATH gets messed up. Does not affect freestyle jobs, and the Pipeline job works again immediately when removing the custom PATH.

Workarounds:

• Set the PATH as env var for the invoked Jenkins master/slave process rather than through Jenkins. In the case of SSH slaves, could probably be a launch command prefix. (untested)
• Set e.g. the PATH+whatever env var instead, and just set the addition to the PATH (e.g. /whatever). This will have the same effect as setting PATH to /whatever:$PATH. That's how Jenkins plugins add e.g. tools to the path. (confirmed)
• Downgrading reportedly also works.

Sending bat signal to jglick.

Using the PATH+WHATEVER=/something/bin syntax works even in 1.13, and is the syntax documented in inline help as what you should use.

I will check if the nonrecommended mode can be supported without regressing JENKINS-40734.

Hmm, this is the in-line help for global environment variables:

These key-value pairs apply for every build on every node. They can be used in 
Jenkins' configuration (as $key or ${key}) and will be added to the environment 
for processes launched from the build.

I see the + syntax is mentioned in the node environment variable settings. I guess it needs to be copied over, assuming it also works globally. I wrote JENKINS-41492 for this update.

jglick how would you set PATH to something like:

$JAVA_HOME/bin:/opt/tools/bin:/opt/nodejs/bin:/opt/python/bin:/opt/composer/bin:/opt/go/${GOVERSION}/go/bin:$PATH

This worked in 1.12 but I can't find the right combination of the plus sign syntax to get all of these variables to line up. The inline help isn't terribly clear. Even then, if I am reading this right I will have to create additional environment variables so I can use GOVERSION in the middle of the path?

me too.

Every time I put a sh step in a pipeline project it break with same error.

node {
    echo "$PATH"
    sh 'ls'
}

The PATH environment variable seems to be corrupted.

Started by user Guest
[Pipeline] node
Running on master in /var/lib/jenkins/jobs/xyz/workspace
[Pipeline] {
[Pipeline] echo
$PATH:/usr/local/bin
[Pipeline] sh
[workspace] Running shell script
nohup: failed to run command `sh': No such file or directory
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code -2
Finished: FAILURE

danielbeck

What does this mean?

Set e.g. the PATH+whatever env var instead, and just set the addition to the PATH (e.g. /whatever). This will have the same effect as setting PATH to /whatever:$PATH. That's how Jenkins plugins add e.g. tools to the path. (confirmed)

Can you please give an example?

octavian See the two comments after mine. Jesse provides an example, James links to an issue explaining where it's documented.

danielbeck

Sorry to bother you, but I didn't understand his example either.

So let's assume I want to add /usr/local/lib/node_modules/npm/bin to the path.

Normally, I would add it like:

PATH=/usr/local/lib/node_modules/npm/bin:$PATH

However, now I would have to add it like this:

PATH+npm=/usr/local/lib/node_modules/npm/bin

Is that correct?

jglick?

That's correct.

PATH+whatever=value means that value with be added to the beginning of PATH, separated by the path separator char.

I do not see a 1.14 plugin available, so I assume that this fix is in fact not released yet. As per the conversation in https://groups.google.com/forum/#!msg/jenkinsci-users/LN057YL_Xis/wyIBfAbfCwAJ, I downgraded the durable tasks plugin from 1.13 to 1.12 and the pipeline plugins to try to work around this killer bug. Interestingly, this apparently does not require a reboot (not sure if that is another bug or not). However, after reverting the plugin, I still cannot run simple shell commands as part of the pipeline. How do I get back to working pipelines?

How do I get back to working pipelines?

Try actually restarting. Downgrading needs a restart.

Confirmed, restart needed after downgrading.

how would you set PATH to something like

Fix your launcher, ~/.profile, etc. Avoid doing this from Jenkins.

The fix was released in workflow-durable-task-step.

If PATH+whatever=value prepends how do I append something to PATH?

how do I append something to PATH

I don't think there's a good answer right now. What's the use case?

Again I do not recommend using Jenkins to modify system environment variables like PATH on nodes to begin with.

Part of the problem with the "downgrade" workaround is that if you've updated other plugins that depend on durable-task:1.13, those other plugins then do not get reloaded on restart, and therefore you cannot downgrade them in the plugin manager.

Or am I missing something?

It especially doesn't help when archives.jenkins-ci.org goes down and then you're just stuck.

@jglick you may not recommend it, but sometimes it is almost necessary. Getting AIX/HPUX/Solaris to do a git clone, or just running a jenkins slave over ssh without setting the path can be difficult. You can try to set the shell profile correctly, but java can do screwy things with that as well, so you have to set the path in the java process, not just the shell to get it to run. Aix is probably the worst in this case.

You do not need to downgrade anything. Upgrade workflow-durable-task-step.

That doesn't actually fix it, it just spits out a warning regarding this issue and fails the job.

Well I have test coverage demonstrating the fix working at least in the case I was able to reproduce based on initial comments. Your situation may be a different bug.

Again, if you must adjust PATH from node properties—rather than, say, fixing your computer setup, or setting environment variables for the whole agent JVM in the launcher—prefer the documented and supported PATH+SOMETHING=/something syntax.

I experienced this and the way I fixed this was to remove the PATH setting in my global config.

My global config was setting PATH to

/var/lib/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/NodeJS_4.4.1/bin:$PATH

I didn't actually need /var/lib/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/NodeJS_4.4.1/bin in my PATH anymore, and when I removed it, the error message went away.

BTW, rolling back did NOT fix the error. Only removing the PATH env var setting fixed the error.

mcsf I also just re-experienced this a few days ago after updating Jenkins and all my plugins again. 

Using 'PATH+name=/x/y/z' still works for me in my Jenkinsfile, but I had to remove any form of it from my global config.

Well, this time it seems to be problem with workflow-job (Pipeline: Job), after downgrade to previous version it works as expected.

Reopened according to the feedback. The issue is not "Fixed" according to the change, because it only adds diagnostics for a single variable (PATH). All other variable expressions do not get even diagnostics.

The issue should be either fixed in a wider way or closed as Won't fix with the notification in the update center for Durable Task 

oleg_nenashev no the diagnostics in ShellStep were not considered part of the fix; this was just an aside. The fix was the behavioral change in ExecutorStepExecution, which fixed a reproducible bug, by expanding variables when a node block is entered. There may be other bugs which can be reproduced in some manner.

jglick

I'm trying to get clarity on the using `PATH+name=/x/y/z` syntax. I thought I knew a lot of things but I'm not sure how to use that to set the path properly (not sure what the + does in terms of PATH bash setting). Do you have a good example of /how/ to do this?  The inline documentation is not particularly clear. I'd happily accept a new best practice way that works.

I can definitely add my 2 cents that in 4 years of Jenkins development and 15,000+ build jobs, programmatically overloading the PATH to achieve what I want across build environments is not just common, but necessary in a multi-tenant build environment world. So I definitely don't agree that "you should never rewrite the path in code" is some kind of best practice.  This change has seriously caused us pain and resulted in even more assorted script hacks to work around that are far less elegant than just exporting a new PATH.

Overriding $PATH is fine, e.g.

withEnv(["PATH+mytool=/opt/my-tool-1.2.3/bin"]) {
  sh 'mytool'
}

which is shorthand for

withEnv(["PATH=/opt/my-tool-1.2.3/bin:$PATH"]) {
  sh 'mytool'
}

Just avoid doing so from node definitions if possible.

@jglick Does "PATH+mytool=" also work for the global environment vars?  And also is the implication that I have to do this for every executable which may be launched from that PATH extension?  For example if 2 executables are in /foo/bin do I need to add 2 PATH vars PATH+executable1="/foo/bin" and PATH+executeable2="/foo/bin"  ?

I honestly cant believe this plugin has broken my installation more than once over something so trivial.  

rendion, no, it doesn't also work for global environment vars. no, you do not have to add an entry for every executable in a directory (that's not how PATH works). sorry something broke you. see https://stackoverflow.com/a/44380495/6090676 for examples of how to add things to the PATH. please use the mailing list or stackoverflow for questions about usage in the future and file specific and reproducible bugs (including Jenkinsfile, what you expected to happen, what actually happened) as appropriate. thank you!

(Thats not how PATH works) <-- correct so why does the syntax require a key at all PATH+stupiduselessconfusingkey=

i documented the stupiduselessconfusingkey at https://stackoverflow.com/a/44380495/6090676 (it's not required) and corrected my answer about whether it also works for global environment vars (it doesn't, sorry).

How do I modify PATH in Global Jenkins settings? Is this even possible not only for specific pipelines, but for the whole Jenkins with 'PATH+name=/x/y/z' syntax?

As oleg_nenashev pointed out, this doesn't only affect the PATH but other Global environment variables as well.

If I were to define for example Global environment variable NODE_TMP=/tmp/${NODE_NAME}

Then Create Freestyle project and Pipeline one with only one step that does env

Then run both projects on master:

In Console Output of the Freestyle build, I would see NODE_TMP=/tmp/master

while in Console Output of the Pipeline it would be NODE_TMP=/tmp/${NODE_NAME}

What's the status of this bug?  It is being fixed?

AFAIK no

FWIW I'm seeing this one too, and this was exceptionally confusing since I have two different build pipelines which are largely the same and both built from blue ocean.

The first one didn't have this issue, with an environment like this:

environment {
    GOPATH = "${WORKSPACE}/go"
    PATH = "${PATH}:${WORKSPACE}/go:${WORKSPACE}/go/bin"
    BUILD_ROOT = "${WORKSPACE}/go/src/github.com/<things>l"
  }

The second one, which used the same PATH variable, did.

I can accept broken but pls consistency.

So really there is just this hack used by Jenkins available ? It's not even bash syntax and provides only prepend option.

The old truth I learned working with Jenkins for 4 years is up to date - keep Jenkins installation and configuraion minimal and do everything you can by yourself in scripts (not by plugins). Jenkins and pipeline should only be glue, not the build system itself.

Disagree Dawid.  Having used Jenkins for 11 years.  Use the plugins as much as possible.

This is crazy. Is there no sane way to do this in a declarative pipeline?

I would like to do this rather basic thing in a declarative pipeline in a Jenkins file:

environment {
  GOPATH = "$WORKSPACE/gopath/bin"
  PATH = "$GOPATH/bin:$PATH"
}

But for some inexplicable reason we can't modify environment variables like one does in every single other utility in the world? Fine, I'll write a script. But clearly looking at buildbot keeps being shoved forcibly up my priority list.

lyda EnvInject plugin does not support Pipeline, and there is no short-term plans to do so. Your comment is not related to this ticket IMHO, Declarative Pipeline has its own implementation. AFAIK there is a ticket for the case you raise (CC abayer svanoort)

I am really confused by all of this.

First, I'm running 2.138.2 LTS (I know, I need to update), with all of the latests plugin version for pipeline (and no EnvInject plugin).  I have tried the following, with no success:

1st:

environment {
  DCP_VERSION = "1.2-alpha"
  DCP_LOC = "/opt/dcp/${DCP_VERSION}"
  QUARTUS_HOME = "/opt/inteldevstack/intelFPGA_pro/quartus" 
  MTI_HOME = "/opt/altera/17.1/modelsim_ase"
  TBB_HOME = "/opt/intel/tbb"
  PATH = "${MTI_HOME}/linux:${MTI_HOME}/bin:${QUARTUS_HOME}/bin:${DCP_LOC}/bin:${PATH}"
  LD_LIBRARY_PATH = '${LD_LIBRARY_PATH}:${TBB_HOME}/lib/intel64_lin/gcc4.7'
}

Produced this error:

nohup: failed to run command 'sh': No such file or directory

2nd (based on googling the sh command error)

environment {
  DCP_VERSION = "1.2-alpha"
  DCP_LOC = "/opt/dcp/${DCP_VERSION}"
  QUARTUS_HOME = "/opt/inteldevstack/intelFPGA_pro/quartus" 
  MTI_HOME = "/opt/altera/17.1/modelsim_ase"
  TBB_HOME = "/opt/intel/tbb"
  PATH+EXTRA = "${MTI_HOME}/linux:${MTI_HOME}/bin:${QUARTUS_HOME}/bin:${DCP_LOC}/bin"
  LD_LIBRARY_PATH = '${LD_LIBRARY_PATH}:${TBB_HOME}/lib/intel64_lin/gcc4.7'
}

Now I get this error:
(PATH + EXTRA) is a binary expression, but it should be a variable expression at line: 11 column: 19. File: WorkflowScript @ line 11, column 19.
PATH+EXTRA="${MTI_HOME}/linux:${MTI_HOME}/bin:${QUARTUS_HOME}/bin:${DCP_LOC}/bin"

Note that neither of these solutions are actually documented in the pipeline guide (https://jenkins.io/doc/book/pipeline/syntax/), and BOTH options are fully acceptable and copied from the built-in syntax generator.

So, what is the right way to do this? I can't set global paths as a lot of these paths change depending on the job (for example, MTI_HOME will change depending on simulator used - future script development task)

Update:  I got around the first issue by setting Jenkins global shell to /bin/bash.  Not cross platform, but works for now.

gruemaster the PATH+EXTRA system works for Scripted Pipeline. If it does not work for Declarative, please file a separate RFE in pipeline-model-definition-plugin. Workaround would I guess be something like (untested)

environment {
  STUFF = "${MTI_HOME}/linux:${MTI_HOME}/bin:${QUARTUS_HOME}/bin:${DCP_LOC}/bin"
}
// …
steps {
  withEnv(["PATH+EXTRA=$STUFF"]) {
    sh 'whatever'
  }
}