The dir step is generally used for workspace-relative paths where the agent is assumed to have full control over the directory, and Jenkins will use sibling directories for various things such as temp directories or shared libraries. If you want to install things in the absolute directory /opt/tmp on the agent, I would do it in a script with the sh step so that you have full control over what will happen and so that the Pipeline does not run with /opt/tmp as its workspace directory at any point.

You might be able to use the Workspace Cleanup Plugin to clear out the temp directory, but to take a step back, I think it would make more sense to just use an agent that already has those tools installed in the first place, i.e. some kind of container or pre-built VM image, so that you aren't changing your agent configuration inside of Jenkins at all.

In the scenario where you have a pipeline where you have 2 stages, each of them executed in a docker container where the USER of the container is "root". You would have the following problem.

Imagine that Stage A executes something like: sh ("mkdir -p foo/bar"). This will generate the directories foo and bar with user and group permissions of "root".

Stage B executes using dir("foo/bar") the following command : sh ("touch somefile.txt"). 

Due to the fact that dir() tries to create a @tmp directory within "foo/bar" the pipeline will fail prompting something like this:

Exception: java.nio.file.AccessDeniedException: </path/to/the/workspace...>/foo/bar/@tmp

I omit the workspace path as it is not relevant. Why is this @tmp creation actually needed?

Why is this @tmp creation actually needed?

The @tmp directory is created by the sh step to store things like the log output and return code of the script (see all uses of controlDir in FileMonitoringTask.java), and is created as a sibling to the workspace directory, since the agent is assumed to have write access to the workspace and its parent directory. The problem is that the dir step actually changes the workspace for everything inside of its body, so the shell step ends up creating a temp directory adjacent to the directory specified by dir, and if your permissions are such that creating directories next to the one specified by dir fails, the sh step will fail as well.

I don't know if there is a way for FileMonitoringController to use the original workspace rather than the directory specified by dir when creating the temp directory, but if so that might be a way to fix this (I think only the current workspace is available, although maybe we could change the dir step to provide the original value in some way. I wonder if looking at what the parallel step does for the workspace would be useful here.). As a workaround you should be able to use cd in your sh script instead of the dir step.

So, if I correctly understand the answer above, the point is that Jenkins needs some space to record its own stuff when a job is running. IMHA (but perhaps there are some cases I don't imagine where it is not possible ?) the better place to do that is the build workspace, whatever the current working directory is. At least, jenkins is sure to be able (i.e. have rwx rights) to create its temp directory.

I guess that it should not be too difficult for a jenkins code to retrieve the aboslute path of the build workspace.

Besides, as a jenkins admin, I wouldn't still see this temporary directory once the build is achieved. Please, let this place as clean as you found it and sweep your own dust ! . Especially given that this temp dir looks to be empty in most cases when the build is over.

Best Regards

The behavior is both undocumented (as far as I can tell), and surprising, since normally "change current directory" would not be a mutating operation.
 
Also, using the name of the directory requested by the user compounds the confusion, because it's easy to miss the @tmp suffix, or think it's an artifact of the exception rendering. I didn't really notice the @tmp suffix until after I had been fighting with this problem for a while, believing Jenkins was trying to create a directory that already existed.