This just happened to us as well.

Not only are the names super long with that UUID like value on them, but the first character of the branch name is missing too.

If you look at Tyrel's example, his directory name is "ine-job_my-branch...." – that was probably supposed to be "line" or something else. We are also seeing the first character removed.

gregcovertsmith Yeah, characters are being removed from the beginning of the name. It's not always just the first character, though; it seems to depend on the length of the build name. Longer names have more characters removed.

We do have a workaround, but it's annoying: Inside the node element of our Jenkinsfile we wrap the entire build also with:

ws("workspace/${env.JOB_NAME}") {
  // ...
}

I would even vote for raising this to severe.

I don't know what is worse – giant long workspace names, or randomly removing characters from the workspace names.

I did go through and try to downlevel all of the plugins I had just updated – the problem never went away (I did it one by one – unfortunately, I was a couple of weeks behind, so there were a lot of updates)

I just reached a point where my Jenkins install wouldn't start (too many cross dependencies), and I had to move back up to latest to get it started again.

Indeed, this was done to fix JENKINS-30744. Encoding the name gets rid of %2F which was there before in place of / in a branch name. For our organization, %2F wasn't causing any problems (Maven doesn't seem to care) and I prefer it to this messy situation.

I vote for an option like with Matrix projects "Use custom child workspace". If multibranch projects had such an option it could be used to solve this issue and the feature branch disk space problem at the same time.

The default value could be "${PROJECT_NAME}/${BRANCH_NAME}" and people with / in their branch names would simply get subdirectories (probably called features or releases) no one would get %2F in their folder names. Those with larger workspaces could set it to "${PROJECT_NAME}" and enjoy saving lots of gigabytes of disk space.

Hi,
This change was very bad for all Python development. Super-long shebang paths like this cause all standard Python entry points in virtualenv to fail due to the Linux operating system shebang length. This includes 'pip' and 'easy_install' !
I appreciate fixing the '%2f' was needed, but this is not the best way to do it I think. It's taken days and days fixing every single call to a subprocess with the standard workaround for this which is to run the programs as 'venv/bin/python venv/bin/entrypoint <args>'.

Another issue are directory names that start with -. It makes using commands more difficult (e.g. du -sch * needs to be du -sch – *).

Just trying to collect the problems as I see it:

1. The path names are nearly useless for "humans". I'm not too upset about that on its own, since ideally Jenkins would manage everything in a smart way on its own.
2. The path names may start with - which is a problem for various reasons.
3. The path names are very long and may cause issues with Windows and even Linux.
4. A lot of the old workspace cleanup methodologies no longer work (e.g. by matching the path with the job name). There are lots of shell and groovy scripts that try to check if a workspace could possibly be in use anymore and remove it if the job doesn't exist. This was already broken in pipeline builds, but it is much more obvious now.
5. Forcing users (e.g. Jenkinsfile authors) to manage their workspace by hand seems wrong; they shouldn't be forced to use ws() and even forcing them to use deleteDir() makes me uncomfortable.
6. node() doesn't have the same sticky properties as a regular build. So now you'll see every job on every slave because it is random.

It feels like workspaces should overall be smarter...

I feel like the default workspace for node() should be special and should be removed as soon as the build is finished (or shortly after). If ws() is specified then it should follow the old-style workspace rules.

Otherwise the maintenance becomes hard for the various slaves, unless you're recycling them.

I upgraded Jenkins from 2.22 to 2.24 today, and I simultaneously updated all plugins. After that we started seeing this issue, and it is causing all of our Python builds to fail.

The issue is that Pip inside a virtualenv cannot handle long path names, because it is based on a script that uses a shebang. The length of a shebang is hard-coded in the kernel to 128 characters, which was not long enough for the workspaces that Jenkins now generates. See also https://github.com/pypa/pip/issues/1773.

To try and mitigate the problem I downgraded Jenkins to 2.22 and the multibranch plugin to 2.8. This did not resolve the issue. (After downgrading more plugins, I ended up in a state of incompatible plugins, which caused the organization (of the GitHub plugin) with all repositories and branches to become corrupt in Jenkins. I had to add a new organisation which caused a rebuild of all branches of all repositories, and which reset all branch numbers. Fortunately we were using Jenkinsfiles so at least the build configuration was preserved.)

At this point I still haven’t been able to revert back to our old configuration, but at least there exists a workaround for the Python/Pip issue: instead of invoking virtualenv/bin/pip, invoke virtualenv/bin/python -m pip.

After seeing the effects this can cause I'm changing it to Critical. It's almost a blocker...

I feel like the default workspace for node() should be special and should be removed as soon as the build is finished (or shortly after).

NO! I don't want to git clone 700MB repositories for twice for every build! I also need to compile code on multiple slaves, then copy stuff around to other slaves and then package stuff up in installers. Local workspaces must not be modified in between.

I'm pretty sure most git users would be upset about having to clone the repository for every build.

Hi all,
here's a workaround I use for this issue on Linux. I have this in a CPSWorkFlowLib var, lets call it 'safeNode.groovy':

def call(label=null, Closure body) {
    node(label) {
        def path = pwd()
        def branchName = env.BRANCH_NAME
        if (branchName) {
                path = path.split('/')
                def workspaceRoot = path[0..<-1].join('/')
                def currentWs = path[-1]
                // Here is where we make branch names safe for directories - 
                // the most common bad character is '/' in 'feature/add_widget'
		// which gets replaced with '%2f', so JOB_NAME will be 
                // ${PROJECT_NAME}%2f${BRANCH_NAME}
		def newWorkspace = env.JOB_NAME.replace('/','-')
		newWorkspace = newWorkspace.replace('%2f', '-')
		newWorkspace = newWorkspace.replace('%2F', '-')
                // Add on the '@n' suffix if it was there
                if (currentWs =~ '@') {
                    newWorkspace = "${newWorkspace}@${currentWs.split('@')[-1]}"
                }
                path = "${workspaceRoot}/${newWorkspace}"
	}
        ws(path) {
                body()
        }
}  

Now in your pipeline code you can use it like:

safeNode {
    sh "pip install pytest"
}

NO! I don't want to git clone 700MB repositories for twice for every build! I also need to compile code on multiple slaves, then copy stuff around to other slaves and then package stuff up in installers. Local workspaces must not be modified in between.

I totally agree that there needs to be a solution for things like 2.7gb git repositories (we have some) and lots of downloaded dependencies. I'm just saying that the default node() behavior isn't the right place to handle this.

I would assume that anyone who needs that ability would use something like the external workspace plugin to do sane caching of data or using something like node('restriction') and ws() to ensure they don't keep getting new (empty) workspaces.

With the current model, if you have 10 slaves, each with 4 executors, you have the possibility of needing to clone that repository up to 40 times, with a minimum of 10 (one per slave) per node() invocation.

Interestingly, this problem has come up before and been the subject of a plugin specifically for it: https://wiki.jenkins-ci.org/display/JENKINS/Short+Workspace+Path+Plugin

I also have been having this issue. But I have just setup Jenkins to automate a MSBuild for our team. Pipeline jobs work fine but Multipath does not work.

Currently our deepest file on our repository is 197 characters deep. leaving me just 63 for the file path. 52 are consumed by the supposed fix for %2 and that leaves me 11 chars C:\ consumes 3\ and then I can't handle even the git branch name. So this totally breaks my code. I know I am running close to the edge and will have a conversation with my devs about it but currently its a waste of 52 chars. I think I could live with 8

I also have been having this issue. My workaround is to set the workspace with "ws('D:\...')".

Additionally it seems that the workspace directory of the global settings is not considered. In my global setting I have specified a path like "D:\jenkins\workspaces", but the multibranch pipeline always creates a path under "C:\jenkins\workspace". With the workaround I could change the path for the job, but not for the script itself.

There seems to be another issue introduced with this update. Multi branch pipeline now ignores the workspace setting.
We have the workspace on a separate drive, but all of a sudden Jenkins creates the workspace in the install directory.
This still works correctly for single pipeline projects.

Jenkins is running on Windows Server 2012 R2.
Workspaces are created in "C:\Program Files (x86)\Jenkins\workspace".
[Manage Jenkins -> Configure System -> Advanced... -> Workspace Root Directory] is set to "D:\workspace\${ITEM_FULL_NAME}"

This issue causes major problems in my msbuild projekts on windows. When using some NuGET dependencies you are practically guaranteed to exceed the path length limit due to these strange strings at the end of the directory names.

I am using the safeNode implementation of eeaston.

Here is a mostly platform independent implementation:

import java.util.regex.Pattern

def call(String label = null, Closure body) {
    node(label) {
        String path = pwd()
        String branchName = env.BRANCH_NAME
        if (branchName) {
            path = path.split(Pattern.quote(File.separator))
            def workspaceRoot = path[0..<-1].join(File.separator)
            def currentWs = path[-1]
            // Here is where we make branch names safe for directories -
            // the most common bad character is '/' in 'feature/add_widget'
            // which gets replaced with '%2f', so JOB_NAME will be
            // ${PR}}OJECT_NAME}%2f${BRANCH_NAME}
            String newWorkspace = env.JOB_NAME.replace('/', '-')
            newWorkspace = newWorkspace.replace(File.separator, '-')
            newWorkspace = newWorkspace.replace('%2f', '-')
            newWorkspace = newWorkspace.replace('%2F', '-')
            // Add on the '@n' suffix if it was there
            if (currentWs =~ '@') {
                newWorkspace = "${newWorkspace}@${currentWs.split('@')[-1]}"
            }
            path = "${workspaceRoot}${File.separator}${newWorkspace}"
        }
        ws(path) {
            body()
        }
    }
}

This is a terrible issue and broke our builds completely on install. Can we have our paths back?

Those following this issue might try to enable the new long file name support in Windows 10 Anniversary Edition, where Microsoft has now removed the path limitation if you turn that support on.

I ran into a very interesting problem with that setting enabled / on that new version of Windows: JENKINS-39179

Thought I would link here, in case anyone tries to go that route.

MSBuild does not support long file path names FYI Git does support long file paths but will break powershell and other command unless you use the //? access paths which almost no one does normally.

This i a feature breaking bug that should be fixed sooner than later.

The only fix I found is to revert multibranch and related plugins to the version before september updates. This is a must fix for build to work in windows environments as well compatibility with previous builds.

I do understand why people are upset by this change.

However, if you want the original behavior, you can set the below as an option on your jenkins java vm setup (for example, in /etc/default/jenkins)

-Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0

Setting this back to the original behavior has generally satisfied us: I hope this option is not removed. But for those which this changed caused a problem, the above is a pretty OK solution (IMO)

Also, I can confirm that msbuild does not work with long file names, even in those versions of Windows 10 where it has been enabled at the filesystem level.

Hi, when activating -Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0 (running on Windows platform), I get 3 folders created for every branch:

• projectname_branchname-C5MAL5QJSBT3VC7KWLCI3CTGCJVURIOQAGAQA2TXJGASFJP5755A
• projectname_branchname-C5MAL5QJSBT3VC7KWLCI3CTGCJVURIOQAGAQA2TXJGASFJP5755A@script
• projectname_branchname-C5MAL5QJSBT3VC7KWLCI3CTGCJVURIOQAGAQA2TXJGASFJP5755A@tmp
  when Jenkins pipeline is running Jenkinsfile, shell scripts are executed in the 1st one which is empty, while the code has been checked out in the one ending with @script
  Is this a side effect of this or related to another defect ? Thanks

gregcovertsmith nim

How do it add this setting on a windows Jenkins install?

-Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0

Do I add it to the jenkins.xml file?

<arguments>-Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0</arguments>

I got it to work by typing the following in to the  groovy script console, but how do I make sure its set on start up?

jenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0

We run Jenkins as a service on Windows and there the parameter must be added to <arguments> in jenkins.xml.

That makes our arguments tag look like this:

<arguments>-Xrs -Xmx256m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0 -jar "%BASE%\jenkins.war" --httpPort=8080 --webroot="%BASE%\war"</arguments>

(the rest was already in there)

gstapviadata thanks, I couldn't find the info anywhere, much appreciated!

Hi everyone!
I`ve got problem with Jenkins paths and cmake:

CMake Error at /opt/tools/common.toolchain.cmake:449 (add_custom_target):
  The target name
  "_var_jenkins_home_workspace_n_feature-SW_EN-208-jenkins-G4FWLCUHTMHI23I7XTJIZIYQ4LYV2G7DLKU6TNRUTURI4LDBWIIA@2_deps_atlas-bld_cygwin64_utst_iris_text-2d_texture_texture-fonts_guikit_atlas1_atlas1"
  is reserved or not valid for certain CMake features, such as generator
  expressions, and may result in undefined behavior.

I think that it`s because there is "@" in directory name. So cmake fails the build
How can I make Jenkins to stop using "@" in names, anybody?

UPD:

Option -Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0 helps me. Thanks.

I have the same problem as mamelnikov had: the "@" sign jenkins adds to workspace paths of parallel nodes in a pipeline breaks build tools. Setting

jenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0

did not help. Any way to tell jenkins not to use the "@" sign, but e.g. underscore instead?

For our setup, setting jenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0 had some side effects when using the Pipeline Maven plugin. In particular, we were seeing:

[ERROR] The specified user settings file does not exist: c:\hudson\workspace\UDP\tcdl\hotfixcleanF8.1.
1cleanFCRQ-6100@tmp\withMaven19741b98\settings.xml
ERROR: [withMaven] WARNING Exception parsing the logs generated by the Jenkins Maven Event Spy c:\hudson\workspace\UDP\tcdl\hotfix%2F8.1.1%2FCRQ-6100@tmp\withMaven19741b98\maven-spy-20170904-153723-126.log, ignore file.  Please report a bug associated for the component 'pipeline-maven-plugin' at https://issues.jenkins-ci.org 

I must be missing something, but it seems that all issues that are related to this are close as 'duplicate'!

None of them have a real solution, just some workarounds

As the above commenter noted, none of the issues regarding this have actually been addressed. They have only been marked as duplicates of one another, and closed. I'm reopening this one so that a resolution can be found.

Facing the same issue - even on non multi-branch pipeline there are created workspace directories with suffixes like:

@2

@2tmp

@script

@script@tmp

@tmp

A (hacky) workaround for this, using scripted pipelines:

def wrappedWorkspace(body) {
    node {
        if(env.WORKSPACE.contains("@")) {
            ws (env.WORKSPACE.replace("@", "__")) {
                body()
            }
        }else{
            body()
        }
    }
}

Thanks pleemann, tried on Jenkins 2.91 but it still does create a new workspace with @3 suffix instead of the one specified in ws().

This issue seems to occur when you do have any aborted job on your queue.

What I did to stop it was to find out which build number was aborted then I removed the respective folder then I restarted Jenkins. After that Jenkins started to recreate the path in right way, without '@' or or numbers

eduardofesilva in my case I don't have any (manually) aborted jobs but I do have failed jobs from previous runs.

cb_tes_global try to use the same approach it might works as well and let me know if that woked for you

Just a note to anyone who might be working on this: This isn't only about the "@" in the folder names, it is a general problem that the folder names are extremely long and unwieldy. All logs tend to become near unreadable as soon as they mention a file path.

This is causing my builds to fail because the foldername starts with a - and a command called by dpkg takes the folder as argument but thinks that the folder is actually another flag. Example: dpkg-source --before-build -Foldername

We have the same problem as hermain these folder names should really start with the proper git name and then end with random characters... rather than the current behavior which truncates the git repo name first

I got here because I would like to have two different multibranch pipeline jobs share the same workspace for a same branch. If I call ws(env.BRANCH_NAME) in both of my Jenkinsfile, it does the trick but when the branch is deleted, the multibranch pipeline scan will only delete the default workspace and not the custom workspace leaving the sources and the build artifacts on the node.

Most other job types have a customWorkspace api that let the user define what the workspace should be. i understand that it can't be that simple for a multi-branch pipeline but maybe a custom workspace name template would work.

deubeuliou, how is your problem related to this bug? If it is not please open another ticket.

On Linux this also creates the problem that shebangs that then include this path get too long and thus scripts (e.g. in python virtualenvs in that build location) cannot be executed.

Why is this hash code behind the folder name? is that collision between branches? So, Why does not  jenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0 have any problems? Please let me know.... 

Hi!
I found a palliative solution to the problem. I added the Allocate Workspace.  It works on multibranch pipeline, on any node.

Ex.:
def wsDir = "/some/path/${env.BRANCH_NAME}"
ws (wsDir) {
// some block
}

dalmosantos _> Another possible solution is to utilize the 'customWorkspace' element as mentioned earlier. It would look something like: 

agent{
  node{
    label 'my-node-label'
    customWorkspace "MyFixedLocation/MyPipelineName_${BRANCH_NAME}"
  }
}

Using 

customWorkspace "${JENKINS_HOME}/Workspace/${URLDecoder.decode(JOB_NAME)}/${BUILD_NUMBER}"

Due to this change got some issues with c++ msbuild prebuild triggered scripts but managed to fix those

After upgrade to this crufty new version of pipeline, all multibranch pipelines using older (eg no declarative pipeline stuffs like pipeline{} agent{} etc) syntax have now a useless name for the git repo, and first part of the name is truncated instead of the ending. Setting jenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0 has no noticeable effect. Combining new and old syntax does not appear to get beyond carving a workspace, and attempting to set workspace via customWorkspace doesn't work at all.

Who approved this disruptive myopic change? Way to break everything...

An acceptable workaround would be a flag to opt out of the workspace name mangling in either the Jenkinsfile or the multibranch configuration.

We are using custom workspace but this is causing trouble with third party tools which do not know that custom workspaces exists.