-
Task
-
Resolution: Unresolved
-
Major
-
Jenkins 2.121.2 on a CentOS 7
In order to install some internal tools within /opt/tools through a Jenkins job, I have created a /opt/tools directory belonging to jenkins:jenkins and where user jenkins only (the user running the slave) has rwx rights.
Trying something like :
stages {
steps('xyz') {
dir('/opt/tools') {
sh "pwd"
}
}
}
Fails with an exception ending with :
java.nio.file.AccessDeniedException: /opt/tools@tmp at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) at java.nio.file.Files.createDirectory(Files.java:674) at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) at java.nio.file.Files.createDirectories(Files.java:767) at hudson.FilePath.mkdirs(FilePath.java:3098) at hudson.FilePath.access$900(FilePath.java:209) at hudson.FilePath$Mkdirs.invoke(FilePath.java:1216) at hudson.FilePath$Mkdirs.invoke(FilePath.java:1212) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2913) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
It appears that Jenkins tries to create a tools@tmp directory at the same level as tools. Yet, there is absolutely no reason for the tools root directory to be writable for any user.
And as far as /opt is concerned here, for sure it must not be writable for anybody else than root.
Additionnally, such @tmp directory is not removed once the build is achieved. Even though it seems that the directory is empty, I think that Jenkins should remove it to give back a clean environment.
[JENKINS-52750] dir step creates a <dirname>@tmp directory at <dirname> level.
In the scenario where you have a pipeline where you have 2 stages, each of them executed in a docker container where the USER of the container is "root". You would have the following problem.
Imagine that Stage A executes something like: sh ("mkdir -p foo/bar"). This will generate the directories foo and bar with user and group permissions of "root".
Stage B executes using dir("foo/bar") the following command : sh ("touch somefile.txt").
Due to the fact that dir() tries to create a @tmp directory within "foo/bar" the pipeline will fail prompting something like this:
Exception: java.nio.file.AccessDeniedException: </path/to/the/workspace...>/foo/bar/@tmp
I omit the workspace path as it is not relevant. Why is this @tmp creation actually needed?
Why is this @tmp creation actually needed?
The @tmp directory is created by the sh step to store things like the log output and return code of the script (see all uses of controlDir in FileMonitoringTask.java), and is created as a sibling to the workspace directory, since the agent is assumed to have write access to the workspace and its parent directory. The problem is that the dir step actually changes the workspace for everything inside of its body, so the shell step ends up creating a temp directory adjacent to the directory specified by dir, and if your permissions are such that creating directories next to the one specified by dir fails, the sh step will fail as well.
I don't know if there is a way for FileMonitoringController to use the original workspace rather than the directory specified by dir when creating the temp directory, but if so that might be a way to fix this (I think only the current workspace is available, although maybe we could change the dir step to provide the original value in some way. I wonder if looking at what the parallel step does for the workspace would be useful here.). As a workaround you should be able to use cd in your sh script instead of the dir step.
So, if I correctly understand the answer above, the point is that Jenkins needs some space to record its own stuff when a job is running. IMHA (but perhaps there are some cases I don't imagine where it is not possible ?) the better place to do that is the build workspace, whatever the current working directory is. At least, jenkins is sure to be able (i.e. have rwx rights) to create its temp directory.
I guess that it should not be too difficult for a jenkins code to retrieve the aboslute path of the build workspace.
Besides, as a jenkins admin, I wouldn't still see this temporary directory once the build is achieved. Please, let this place as clean as you found it and sweep your own dust ! 
. Especially given that this temp dir looks to be empty in most cases when the build is over.
Best Regards
Philip Garrett
added a comment - The behavior is both undocumented (as far as I can tell), and surprising, since normally "change current directory" would not be a mutating operation.
Also, using the name of the directory requested by the user compounds the confusion, because it's easy to miss the @tmp suffix, or think it's an artifact of the exception rendering. I didn't really notice the @tmp suffix until after I had been fighting with this problem for a while, believing Jenkins was trying to create a directory that already existed.
Philip Garrett
added a comment - The behavior is both undocumented ( as far as I can tell ), and surprising, since normally "change current directory" would not be a mutating operation.
Also, using the name of the directory requested by the user compounds the confusion, because it's easy to miss the @tmp suffix, or think it's an artifact of the exception rendering. I didn't really notice the @tmp suffix until after I had been fighting with this problem for a while, believing Jenkins was trying to create a directory that already existed. Do not use dir for this purpose. Simply start your sh script with cd /opt/tools.
A have a pipeline which builds and creates installer/.zip archive. I'm using "dir" to change directory to a folder and execute python script there. The problem I encountered is that now the installer/zip contains additional empty folder@tmp.
Edit: I've read the explanation that Devin wrote again, and I see that it's not "dir" that created the folder@tmp, but rather the "sh" step (or in my case - the "bat" step probably).
Is that the only step that does it?
It's a bit problematic. Now I'm affraid to use "dir", because who knows how will it affect other steps? And it's a very useful step...
If "sh"/"bat" is the only such step, maybe it could create the temp folder in some location for temporary files, if jenkins provides such thing. If not, maybe jenkins could be extended to provide it.
Piotr Krukowiecki
added a comment - - edited A have a pipeline which builds and creates installer/.zip archive. I'm using "dir" to change directory to a folder and execute python script there. The problem I encountered is that now the installer/zip contains additional empty folder@tmp.
Edit: I've read the explanation that Devin wrote again, and I see that it's not "dir" that created the folder@tmp, but rather the "sh" step (or in my case - the "bat" step probably).
Is that the only step that does it?
It's a bit problematic. Now I'm affraid to use "dir", because who knows how will it affect other steps? And it's a very useful step...
If "sh"/"bat" is the only such step, maybe it could create the temp folder in some location for temporary files, if jenkins provides such thing. If not, maybe jenkins could be extended to provide it. 
Warren Humphreys
added a comment - jglick ‘Simply’ adding a sh “cd /opt/tools” does not work in a declarative pipeline. Subsequent sh steps will have their current directory reset back to the workspace directory. This is frustrating and not documented.
Warren Humphreys
added a comment - jglick ‘Simply’ adding a sh “cd /opt/tools” does not work in a declarative pipeline. Subsequent sh steps will have their current directory reset back to the workspace directory. This is frustrating and not documented. Adjacent sh steps should be coalesced, and if possible their content externalized to a versioned script so that you are left with only a short one-line sh argument. Of course on occasion there is a reason for there to be intervening steps of other types.
Hi all,
The point is not only related to sh something, it is a more generic issue. Have I done the same with python, ruby or groovy step, I would have the same.
The point is actually that when a dir() occurs, a @tmp folder is created behind the scene, and this is potentially a problem in some cases.
Any update here. I consider this a bug because it makes pipelines fail with strange errors and I have to add code that cleans up the mess. Also `dir()` is simply not relly usable then.
Patrick
added a comment - I've got the same problem with dir() in my Pipeline script. Is there any solution available yet?
Patrick
added a comment - I've got the same problem with dir() in my Pipeline script. Is there any solution available yet? This is indeed a bug and I don't even manage to make the "cd " workaround work:
The following pipeline:
pipeline {
agent {label 'ansible'}
stages {
stage('playbook') {
steps {
sh '''
cd /git/it/ansible/infrastructure
pwd
ls -al
./vault_pass
ansible-playbook --check docker.yml
'''
}
}
}
}
yields that result:
+ cd /git/it/ansible/infrastructure + pwd /git/it/ansible/infrastructure + ls -al total 8 drwxrwxrwx 1 root root 4096 Nov 18 19:26 . drwxrwxrwx 1 root root 4096 Nov 18 19:25 .. drwxrwxrwx 1 root root 4096 Dec 14 2020 .git -rwxrwxrwx 1 root root 234 Nov 20 2019 .gitignore -rwxrwxrwx 1 root root 218 Nov 20 2019 .project -rwxrwxrwx 1 root root 3354 Nov 20 2019 Vagrantfile -rwxrwxrwx 1 root root 131 Nov 20 2019 all.yml -rwxrwxrwx 1 root root 380 Nov 20 2019 ansible.cfg -rwxrwxrwx 1 root root 720 Nov 20 2019 ansible.yml -rwxrwxrwx 1 root root 126 Nov 20 2019 aptgetclean.yml -rwxrwxrwx 1 root root 61 Nov 20 2019 common.yml -rwxrwxrwx 1 root root 513 Nov 20 2019 docker.yml drwxrwxrwx 1 root root 4096 Nov 20 2019 files drwxrwxrwx 1 root root 4096 Nov 20 2019 group_vars drwxrwxrwx 1 root root 4096 Nov 20 2019 host_vars drwxrwxrwx 1 root root 4096 Nov 20 2019 inventories -rwxrwxrwx 1 root root 94 Nov 20 2019 monitoring.yml drwxrwxrwx 1 root root 4096 Nov 20 2019 public_keys -rwxrwxrwx 1 root root 227 Nov 20 2019 requirements.yml drwxrwxrwx 1 root root 4096 Nov 20 2019 roles drwxrwxrwx 1 root root 4096 Nov 20 2019 scripts drwxrwxrwx 1 root root 4096 Nov 20 2019 templates -rwxrwxrwx 1 root root 2172 Nov 20 2019 updater-rsync.yml -rwxrwxrwx 1 root root 44 Nov 18 19:56 vault_pass -rwxrwxrwx 1 root root 64 Nov 20 2019 weblate.yml + ./vault_pass /home/workspace/ansible@tmp/durable-ac15a4f4/script.sh: line 1: ./vault_pass: not found
EDIT: this formatting language is difficult to handle....
Joel
added a comment - - edited This is indeed a bug and I don't even manage to make the "cd " workaround work:
The following pipeline:
pipeline {
agent {label 'ansible' }
stages {
stage( 'playbook' ) {
steps {
sh '''
cd /git/it/ansible/infrastructure
pwd
ls -al
./vault_pass
ansible-playbook --check docker.yml
'''
}
}
}
}
yields that result:
+ cd /git/it/ansible/infrastructure
+ pwd
/git/it/ansible/infrastructure
+ ls -al
total 8
drwxrwxrwx 1 root root 4096 Nov 18 19:26 .
drwxrwxrwx 1 root root 4096 Nov 18 19:25 ..
drwxrwxrwx 1 root root 4096 Dec 14 2020 .git
-rwxrwxrwx 1 root root 234 Nov 20 2019 .gitignore
-rwxrwxrwx 1 root root 218 Nov 20 2019 .project
-rwxrwxrwx 1 root root 3354 Nov 20 2019 Vagrantfile
-rwxrwxrwx 1 root root 131 Nov 20 2019 all.yml
-rwxrwxrwx 1 root root 380 Nov 20 2019 ansible.cfg
-rwxrwxrwx 1 root root 720 Nov 20 2019 ansible.yml
-rwxrwxrwx 1 root root 126 Nov 20 2019 aptgetclean.yml
-rwxrwxrwx 1 root root 61 Nov 20 2019 common.yml
-rwxrwxrwx 1 root root 513 Nov 20 2019 docker.yml
drwxrwxrwx 1 root root 4096 Nov 20 2019 files
drwxrwxrwx 1 root root 4096 Nov 20 2019 group_vars
drwxrwxrwx 1 root root 4096 Nov 20 2019 host_vars
drwxrwxrwx 1 root root 4096 Nov 20 2019 inventories
-rwxrwxrwx 1 root root 94 Nov 20 2019 monitoring.yml
drwxrwxrwx 1 root root 4096 Nov 20 2019 public_keys
-rwxrwxrwx 1 root root 227 Nov 20 2019 requirements.yml
drwxrwxrwx 1 root root 4096 Nov 20 2019 roles
drwxrwxrwx 1 root root 4096 Nov 20 2019 scripts
drwxrwxrwx 1 root root 4096 Nov 20 2019 templates
-rwxrwxrwx 1 root root 2172 Nov 20 2019 updater-rsync.yml
-rwxrwxrwx 1 root root 44 Nov 18 19:56 vault_pass
-rwxrwxrwx 1 root root 64 Nov 20 2019 weblate.yml
+ ./vault_pass
/home/workspace/ansible@tmp/durable-ac15a4f4/script.sh: line 1: ./vault_pass: not found
EDIT: this formatting language is difficult to handle....
Hatter
added a comment - I'm surprised to see this issue is still not fixed yet in 2023... I don't think the AccessDeniedException is unexpected.
dir seems to expect a relative path as argument.
Ref: https://www.jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#dir-change-current-directory
Harikishen Hosamana
added a comment - - edited I don't think the AccessDeniedException is unexpected.
dir seems to expect a relative path as argument.
Ref: https://www.jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#dir-change-current-directory Joel
added a comment - I fail to see how AccessDeniedException is to be expected.
It is at least not mentioned in the documentation, let alone be intuitive that whenever we change directory a @tmp folder will be generated as sibling (and therefore the parent must be writable).
What I expected, is that when I use the dir directive to change folder, I would actually be in that folder.
Joel
added a comment - I fail to see how AccessDeniedException is to be expected.
It is at least not mentioned in the documentation, let alone be intuitive that whenever we change directory a @tmp folder will be generated as sibling (and therefore the parent must be writable).
What I expected, is that when I use the dir directive to change folder, I would actually be in that folder. We ran into this very same issue some days ago. Actually I am puzzled we haven't had this trouble earlier already.
According to my experiments there is a (subtile) difference on wether the directory does exist on checkout(seems working) or wether it was created during the build in another step (causing troubles).
Any suggestions for workarounds available?
The dir step is generally used for workspace-relative paths where the agent is assumed to have full control over the directory, and Jenkins will use sibling directories for various things such as temp directories or shared libraries. If you want to install things in the absolute directory /opt/tmp on the agent, I would do it in a script with the sh step so that you have full control over what will happen and so that the Pipeline does not run with /opt/tmp as its workspace directory at any point.
You might be able to use the Workspace Cleanup Plugin to clear out the temp directory, but to take a step back, I think it would make more sense to just use an agent that already has those tools installed in the first place, i.e. some kind of container or pre-built VM image, so that you aren't changing your agent configuration inside of Jenkins at all.