It happens due to this class: https://github.com/jenkinsci/envinject-plugin/blob/master/src/main/java/org/jenkinsci/plugins/envinject/service/EnvironmentVariablesNodeLoader.java#L65-L68 , likely it happens due to this commit: https://github.com/jenkinsci/envinject-plugin/commit/21877ef1b473abb90c62034cd4cc941623692838 . And this commit is perfectly valid.

1) Check Remoting version on your AIX hosts. It should be 3.7

2) If the version is up to date, try removing the JAR cache in "~/.jenkins" and reconnecting the agent

If the issue happens, likely you have a machine being shared between two Jenkins masters of different versions. In such case use the -jarCache parameter to specify different cache locations

1. It is 3.7.
2. I tried that as well, still the same error

I only have 1 master. A quick note that I do have a   "Prefix Start Slave Command"  which has

export LIBPATH=$LIBPATH:/opt/freeware/lib;

Only because AIX is a pain, and I can't run curl/git without it, even though the same line is in my .profile.

The same error is occuring for me on zOS.  I only have one Jenkins master.  Jenkins: 2.46.2, Env Injector Plugin: 2.1
Agent version is 3.7.  With the ~/jenkins/cache/jars directory cleaned up, I get errors like those included below when the agent is restarted and then the same error occurs as above when the job is initiated.  I also tried removing the jars directory and then the agent complains with "Root directory not writable: /u/dlbuild/.jenkins/cache/jars",

May 24, 2017 6:36:57 PM hudson.remoting.JarCacheSupport$1 run WARNING: Failed to resolve a jar 2ebd67217364ba864c9c382142164236 java.io.IOException: Failed to write to /u/dlbuild/.jenkins/cache/jars/2E/BD67217364BA864C9C382142164236.jar at hudson.remoting.FileSystemJarCache.retrieve(FileSystemJarCache.java:133) at hudson.remoting.JarCacheSupport$1.run(JarCacheSupport.java:64) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:482) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:345) at java.util.concurrent.FutureTask.run(FutureTask.java:177) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110) at java.lang.Thread.run(Thread.java:769) Caused by: java.io.IOException: Root directory does not exist at java.nio.file.Files.createDirectories(Files.java:716) at hudson.remoting.Util.mkdirs(Util.java:230) at hudson.remoting.FileSystemJarCache.createTempJar(FileSystemJarCache.java:158) at hudson.remoting.FileSystemJarCache.retrieve(FileSystemJarCache.java:93)

tparker1 How do you restart the the agent? With entire JVM? If yes, would be great to have a full stacktrace

Sorry for many low-level questions just in case. As you may guess I have neither AIX nor IBM mainframe machine right now, so I cannot reproduce the issue on my own. Have not seen such issues on the platforms I have now though maybe it happens there as well

I have aix,solaris sparc/x86, hpux, centos 6/7, sles10, and windows. Sadly aix was the only one to fail for me. I rolled back for now.

I have aix-7.1, zlinux-SLES11sp4, windows-2012 and linux-SLES11sp4 which are all okay after the upgrade.  Only zOS is failing for me;  it is using USS (Unix System Services) so it is treated like a unix system.  The way it is invoked:

cd "/u/dlbuild/jenkins" && /usr/lpp/java/J7.0/bin/java -Dfile.encoding=ISO8859-1 -Djava.io.tmpdir=/u/dlbuild/tmp -jar slave.jar -text Running in text mode

A snippet from the Agent log for it looks like this :

[05/24/17 12:01:10] [SSH] Opening SSH connection to xxxx.raleigh.ibm.com:22.
[05/24/17 12:01:10] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
[05/24/17 12:01:11] [SSH] Authentication successful.
[05/24/17 12:01:11] [SSH] The remote users environment is:
@="sh"
ERRNO="0"
HOME="/u/dlbuild"
IFS="     
"
LINENO="0"
LOGNAME="DLBUILD"
MAIL="/usr/mail/DLBUILD"
MAILCHECK="600"
OPTIND="1"
PATH="/bin"
PPID="882"
PS1="\$ "
PS2="> "
PS3="#? "
PS4="+ "
PWD="/u/dlbuild"
RANDOM="10945"
SECONDS="0"
SHELL="/bin/sh"
SSH_CLIENT="xxxx 35173 22"
SSH_CONNECTION="xxxx 35173 yyyy 22"
USER="DLBUILD"
[05/24/17 12:01:11] [SSH] Starting sftp client.
[05/24/17 12:01:11] [SSH] Copying latest slave.jar...
[05/24/17 12:01:14] [SSH] Copied 719,269 bytes.
Expanded the channel window size to 4MB
[05/24/17 12:01:14] [SSH] Starting slave process: cd "/u/dlbuild/jenkins" && /usr/lpp/java/J7.0/bin/java -Dfile.encoding=ISO8859-1 -Djava.io.tmpdir=/u/dlbuild/tmp -jar slave.jar -text
Running in text mode
<===[JENKINS REMOTING CAPACITY]===><===[HUDSON TRANSMISSION BEGINS]===channel started
May 24, 2017 7:01:22 PM hudson.remoting.JarCacheSupport$1 run
WARNING: Failed to resolve a jar 033560cffbe8012d0fca4763c9237a01
java.io.IOException: Failed to write to /u/dlbuild/.jenkins/cache/jars/03/3560CFFBE8012D0FCA4763C9237A01.jar
    at hudson.remoting.FileSystemJarCache.retrieve(FileSystemJarCache.java:133)
    at hudson.remoting.JarCacheSupport$1.run(JarCacheSupport.java:64)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:482)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:345)
    at java.util.concurrent.FutureTask.run(FutureTask.java:177)
    at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
    at java.lang.Thread.run(Thread.java:769)
Caused by: java.io.IOException: Root directory does not exist
    at java.nio.file.Files.createDirectories(Files.java:716)
    at hudson.remoting.Util.mkdirs(Util.java:230)
    at hudson.remoting.FileSystemJarCache.createTempJar(FileSystemJarCache.java:158)
    at hudson.remoting.FileSystemJarCache.retrieve(FileSystemJarCache.java:93)
    ... 6 more

Slave.jar version: 3.7
This is a Unix agent

May 24, 2017 7:01:27 PM hudson.remoting.JarCacheSupport$1 run
WARNING: Failed to resolve a jar 14bce94dd1f2619f3ddac05c3ead7634
java.io.IOException: Failed to write to /u/dlbuild/.jenkins/cache/jars/14/BCE94DD1F2619F3DDAC05C3EAD7634.jar
    at hudson.remoting.FileSystemJarCache.retrieve(FileSystemJarCache.java:133)
    at hudson.remoting.JarCacheSupport$1.run(JarCacheSupport.java:64)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:482)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:345)
    at java.util.concurrent.FutureTask.run(FutureTask.java:177)
    at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
    at java.lang.Thread.run(Thread.java:769)
Caused by: java.io.IOException: Root directory does not exist
    at java.nio.file.Files.createDirectories(Files.java:716)
    at hudson.remoting.Util.mkdirs(Util.java:230)
    at hudson.remoting.FileSystemJarCache.createTempJar(FileSystemJarCache.java:158)
    at hudson.remoting.FileSystemJarCache.retrieve(FileSystemJarCache.java:93)
    ... 6 more

Looks like you are setting variables on startup as well, which is similar to what I have to do when I start my aix slave. So possibly due to that?

My other nodes' configurations don't pass in any additional  variables beyond the Port and the Java Path under Advanced.

However, under Node Properties, I am defining some env variables.

The build error seems to be complaining about loading those ??

I wonder if the 'Prepare Jobs Environment', properties file would work better.

I downgraded to 2.0 and it is working

2.1 was also failing on a window 2003 server slave with the same error (same Jenkins Master) but I see that it is using slave.jar 3.4.1

I see this issue, too, on AIX and Linux on z Systems. I have seen similar bugs in the past and it has been a compatibility issue with the IBM JDK. I have not yet tested this, but maybe another one has time to give it a try. IBM publishes its Java SDK for Linux on Intel, too: https://developer.ibm.com/javasdk/downloads/sdk8/

Reverted Envinject to 2.0, which is doing fine on all machines.

I could decouple this anonymous class to a nested class, which should definitely cause whatever checks to pass since it will be a new class. OTOH it will be rather a hack, because I still do not understand what could cause it. Maybe some crappy core dependency overlap due to the EnvInject lib

No response from the reporter. Unassigning

You never gave me anything to try, what response were you looking for?

I am seeing this on my AIX agents too, all my other agents (Windows, SunOS, HP, Red Hat & Suse) are ok.

• Jenkins: 2.60.2
• envinject : 2.1.3

Also had the same issue on AIX build agents.  No issues with RHEL, Windows, iOS build agents.

• Jenkins v2.46.2
• EnvInject v2.1.3

This was resolved by rolling back to version 2.0 of the EnvInject plugin.

In our case, when the issue was reported then the version of java was "1.8.0" , and now it has been upgraded to version "1.8.0_144" on the AIX slave node, to resolve the issue.