-
Bug
-
Resolution: Won't Fix
-
Critical
-
Windows 2012 x64, Java 1.8, Jenkins 2.7.4, Pipeline: Multibranch 2.9, Pipeline: API 2.4, Linux Python
-
Powered by SuggestiMate
Apparently after a recent update of plugins, our Multibranch Pipeline jobs started putting their workspaces in mangled locations. They used to be, for example, in D:\jenkins\cm-jenkins-09\my-pipline-job\my-branch-name and now they're like D:\jenkins\cm-jenkins-09\ine-job_my-banch-name-5P5URKHPJXWGGVIWDQOJBB3N7RJECAQJGFTDCVOPY3PABO7LNTIQ
There are several problems with this:
- They're difficult to read with all that garbage at the end, and the beginning of the name sometimes cut off
- They're not organized within the parent job like they used to be
- This is causing files within them to have extremely long paths such that Windows won't actually let us delete them (Jenkins, CMD, and Windows Explorer all fail to delete them)
Let me know if I can provide any other information to help solve this. This is a private Jenkins install so I can't point you to it.
- duplicates
-
JENKINS-38506 Invalid workspace path for pipeline jobs on jenkins master
-
- Resolved
-
-
JENKINS-34564 Give the ability to choose how the multibranch subprojects will be named.
-
- Resolved
-
- is duplicated by
-
JENKINS-49251 Pipeline in Multi-branch mode creates too long workspace folder name
-
- Resolved
-
-
JENKINS-40072 Multibranch creates multiple new directories
-
- Closed
-
-
JENKINS-48176 Reduce length of workspace directory name for multibranch jobs
-
- Resolved
-
- relates to
-
JENKINS-2111 removing a job (including multibranch/org folder branches/repos) does not remove the workspace
-
- Resolved
-
-
JENKINS-54640 Workspace folders are not unique
-
- Closed
-
-
JENKINS-52911 Higly voted critical issue JENKINS-38706 remains unassigned in JIRA
-
- Resolved
-
[JENKINS-38706] Workspace directory names mangled in multibranch pipeline
Also, I can confirm that msbuild does not work with long file names, even in those versions of Windows 10 where it has been enabled at the filesystem level.
Hi, when activating -Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0 (running on Windows platform), I get 3 folders created for every branch:
- projectname_branchname-C5MAL5QJSBT3VC7KWLCI3CTGCJVURIOQAGAQA2TXJGASFJP5755A
- projectname_branchname-C5MAL5QJSBT3VC7KWLCI3CTGCJVURIOQAGAQA2TXJGASFJP5755A@script
- projectname_branchname-C5MAL5QJSBT3VC7KWLCI3CTGCJVURIOQAGAQA2TXJGASFJP5755A@tmp
when Jenkins pipeline is running Jenkinsfile, shell scripts are executed in the 1st one which is empty, while the code has been checked out in the one ending with @script
Is this a side effect of this or related to another defect ? Thanks
How do it add this setting on a windows Jenkins install?
-Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0
Do I add it to the jenkins.xml file?
<arguments>-Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0</arguments>
I got it to work by typing the following in to the groovy script console, but how do I make sure its set on start up?
jenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0
We run Jenkins as a service on Windows and there the parameter must be added to <arguments> in jenkins.xml.
That makes our arguments tag look like this:
<arguments>-Xrs -Xmx256m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0 -jar "%BASE%\jenkins.war" --httpPort=8080 --webroot="%BASE%\war"</arguments>
(the rest was already in there)
gstapviadata thanks, I couldn't find the info anywhere, much appreciated!
Hi everyone!
I`ve got problem with Jenkins paths and cmake:
CMake Error at /opt/tools/common.toolchain.cmake:449 (add_custom_target): The target name "_var_jenkins_home_workspace_n_feature-SW_EN-208-jenkins-G4FWLCUHTMHI23I7XTJIZIYQ4LYV2G7DLKU6TNRUTURI4LDBWIIA@2_deps_atlas-bld_cygwin64_utst_iris_text-2d_texture_texture-fonts_guikit_atlas1_atlas1" is reserved or not valid for certain CMake features, such as generator expressions, and may result in undefined behavior.
I think that it`s because there is "@" in directory name. So cmake fails the build
How can I make Jenkins to stop using "@" in names, anybody?
UPD:
Option -Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0 helps me. Thanks.
I have the same problem as mamelnikov had: the "@" sign jenkins adds to workspace paths of parallel nodes in a pipeline breaks build tools. Setting
jenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0
did not help. Any way to tell jenkins not to use the "@" sign, but e.g. underscore instead?
For our setup, setting jenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0 had some side effects when using the Pipeline Maven plugin. In particular, we were seeing:
[ERROR] The specified user settings file does not exist: c:\hudson\workspace\UDP\tcdl\hotfixcleanF8.1. 1cleanFCRQ-6100@tmp\withMaven19741b98\settings.xml ERROR: [withMaven] WARNING Exception parsing the logs generated by the Jenkins Maven Event Spy c:\hudson\workspace\UDP\tcdl\hotfix%2F8.1.1%2FCRQ-6100@tmp\withMaven19741b98\maven-spy-20170904-153723-126.log, ignore file. Please report a bug associated for the component 'pipeline-maven-plugin' at https://issues.jenkins-ci.org
I must be missing something, but it seems that all issues that are related to this are close as 'duplicate'!
None of them have a real solution, just some workarounds
As the above commenter noted, none of the issues regarding this have actually been addressed. They have only been marked as duplicates of one another, and closed. I'm reopening this one so that a resolution can be found.
Facing the same issue - even on non multi-branch pipeline there are created workspace directories with suffixes like:
@2
@2tmp
@script
@script@tmp
@tmp
A (hacky) workaround for this, using scripted pipelines:
def wrappedWorkspace(body) { node { if(env.WORKSPACE.contains("@")) { ws (env.WORKSPACE.replace("@", "__")) { body() } }else{ body() } } }
Thanks pleemann, tried on Jenkins 2.91 but it still does create a new workspace with @3 suffix instead of the one specified in ws().
This issue seems to occur when you do have any aborted job on your queue.
What I did to stop it was to find out which build number was aborted then I removed the respective folder then I restarted Jenkins. After that Jenkins started to recreate the path in right way, without '@' or or numbers
eduardofesilva in my case I don't have any (manually) aborted jobs but I do have failed jobs from previous runs.
cb_tes_global try to use the same approach it might works as well and let me know if that woked for you
Just a note to anyone who might be working on this: This isn't only about the "@" in the folder names, it is a general problem that the folder names are extremely long and unwieldy. All logs tend to become near unreadable as soon as they mention a file path.
This is causing my builds to fail because the foldername starts with a - and a command called by dpkg takes the folder as argument but thinks that the folder is actually another flag. Example: dpkg-source --before-build -Foldername
We have the same problem as hermain these folder names should really start with the proper git name and then end with random characters... rather than the current behavior which truncates the git repo name first
I got here because I would like to have two different multibranch pipeline jobs share the same workspace for a same branch. If I call ws(env.BRANCH_NAME) in both of my Jenkinsfile, it does the trick but when the branch is deleted, the multibranch pipeline scan will only delete the default workspace and not the custom workspace leaving the sources and the build artifacts on the node.
Most other job types have a customWorkspace api that let the user define what the workspace should be. i understand that it can't be that simple for a multi-branch pipeline but maybe a custom workspace name template would work.
deubeuliou, how is your problem related to this bug? If it is not please open another ticket.
On Linux this also creates the problem that shebangs that then include this path get too long and thus scripts (e.g. in python virtualenvs in that build location) cannot be executed.
Why is this hash code behind the folder name? is that collision between branches? So, Why does not jenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0 have any problems? Please let me know....
Hi!
I found a palliative solution to the problem. I added the Allocate Workspace. It works on multibranch pipeline, on any node.
Ex.:
def wsDir = "/some/path/${env.BRANCH_NAME}"
ws (wsDir) {
// some block
}
dalmosantos _> Another possible solution is to utilize the 'customWorkspace' element as mentioned earlier. It would look something like:
agent{ node{ label 'my-node-label' customWorkspace "MyFixedLocation/MyPipelineName_${BRANCH_NAME}" } }
Using
customWorkspace "${JENKINS_HOME}/Workspace/${URLDecoder.decode(JOB_NAME)}/${BUILD_NUMBER}"
Due to this change got some issues with c++ msbuild prebuild triggered scripts but managed to fix those
After upgrade to this crufty new version of pipeline, all multibranch pipelines using older (eg no declarative pipeline stuffs like pipeline{} agent{} etc) syntax have now a useless name for the git repo, and first part of the name is truncated instead of the ending. Setting jenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0 has no noticeable effect. Combining new and old syntax does not appear to get beyond carving a workspace, and attempting to set workspace via customWorkspace doesn't work at all.
Who approved this disruptive myopic change? Way to break everything...
An acceptable workaround would be a flag to opt out of the workspace name mangling in either the Jenkinsfile or the multibranch configuration.
We are using custom workspace but this is causing trouble with third party tools which do not know that custom workspaces exists.
Cant use the workaround in my case. I don't use docker, I build directly on a build server VM (Windows)
agent{ node{ label 'my-node-label' customWorkspace "MyFixedLocation/MyPipelineName_${BRANCH_NAME}" } }
In my case, I can't specify label field, so not possible to specify customWorkspace. Any other workarounds please.
this completely breaks my windows builds because of the 260 limit. Why can't you use the github short hash instead?
This is also a huge issue here. Where it is blocking us from properly using the multibranch pipeline feature.
Also huge issue for us. The mangled and shortened names often produce paths that start with a dash character, and that breaks many configure and build scripts in a variety of projects.
The fact that this is an almost two year old issue that is still unresolved is pretty ridiculous.
svanoort abayer rsandell Hi. Is it in the scope for the Pipeline team? godskalk has recently raised JENKINS-52911 to get some attention to it
Yes the current behavior is intentional. Certainly it has some problems that affect some people; the previous behavior also had serious problems that affected lots of people.
Rather than applying further tweaks, the basic assumption—that a workspace path on an agent (relative to the agent’s “filesystem root”) must be a function of the job’s full name so that it is guaranteed to be unique—should be discarded, in favor of a system of allocating short, legible workspace paths per job × agent and explicitly tracking them.
This could be integrated with proper workspace cleanup (for “permanent, as opposed to elastic, agents), which Jenkins has never done, which is why I proposed such a rewrite in JENKINS-2111.
We just ran into a problem where the name of our git branch is "feature--XXX" and Jenkins decided the name was too long, so stripped feature. Leaving a directory that starts with "--".
13:22:18 [CentOS 6: Main RPM Build] Found files starting with hyphen
13:22:18 [CentOS 6: Main RPM Build] error: Bad exit status from /var/tmp/rpm-tmp.mTAEkR (%install)
I'm adding some tweaks to the implementation in a PR.
- Each agent's system properties will also be checked for a local override of path max. This will allow windows agents to have a shorter path max without affecting *nix agents
- I am adding a second implementation that uses the NameMangler. As NameMangler should always produce a safe name for all OSes (but it might be too long on windows if your SCM checkout is too deep... nothing we can do there) it is a simple boolean flag to enable/disable. And as it is opinionated it can only be set globally. System property will be jenkins.branch.WorkspaceLocatorV2Impl.ENABLE I suspect that the NameMangler version is the one people actually want, so before I default it on I would appreciate some real world feed-back
stephenconnolly I didn't see anything in that PR addressing a possible leading "-" - are you assuming the path length should fix "most" cases? That's the problem I had as well as others previously (e.g. dener). Otherwise, kudos and thanks!
This is just beating around the bush. See JENKINS-2111 for the proper permanent solution to handling inelastic (“dumb”) build agents.
Unless in that corresponding XML file you're holding a mapping of what the paths are today to some shorter friendly your name that is then used in actuality I'm still not sure how that resolves the issue of the workspace having a path that has a 'mangled' name. I agree that having better logic around work spaces and the ability to clean them up automatically is a good thing but I don't think it's directly related to this nor will one automatically solve the other (depending on the implementation it could)
in that corresponding XML file you're holding a mapping of what the paths are today to some shorter friendly your name
This. Read what I wrote.
I did read what you wrote, hence my question - nothing in this:
Each node (master, agent) should just pay attention to when a workspace is used. (If in core, via WorkspaceList; otherwise, perhaps via WorkspaceListener.) Then record a workspaces.xml, a sibling of workspace/, with a list of records: relative workspace path, Item.fullName, timestamp.
seems to say that you are intending to 'clean' or 'modify' or 'change' the paths from what they are today to anything nicer - just that you want to maintain a map of job to workspace for cleanup purposes down the road.
Sorry, that was a corollary that should have been made explicit: if you are maintaining a persistent mapping of this kind, then the need to include some manner of hash in the workspace path disappears. (You would still need to translate dangerous characters to something more neutral like _.)
WORKAROUND
- For users using python virtualenv, try using
virtualenv --relocatable venv
... BEFORE calling "pip"
That changes the shebang line to:
#!/usr/bin/env python27
... or whatever version of python you are using, but significantly shorter.
kpop, there is no way you need over 50 base-64 characters to avoid name clashes. Git manages with just 7 hex-characters for thousands of commits.
godskalk, I agree: less is certainly better, but name clashes should be avoided.
My workaround for this was to define a jobName and then use the jobName in the workspace. The jobName var was also to account for feature branches and builds around several different microservices that might also have the same Jira id attached to them.
Our feature branch work is all short lived and named feature/jira-12345-some-description. Our services are named my-service-name. The below lines result in a 'my-service-name-jira-12345' workspace name. 99% of the time we don't need to look at workspaces anyway, but this makes the console log readable as well as making it possible to write directory cleanup patterns based on naming.
def jobName = JOB_NAME.replaceAll(/\/\w+%2F/,'-').toLowerCase()
ansiColor( 'xterm' ) {
timestamps {try {node() {
ws( "workspace/${jobName}" ) {
stage( 'Pull Github Repository' ) {
This issue became obsolete with the release of the fix of JENKINS-2111.
I am using Jenkins 2.150.2 version but still hitting this issue?
I also install short workspace plugins without any luck.
Is this truly fixed?
suemiao please do not reopen. If necessary, file a separate (but linked) bug report with complete steps to reproduce from scratch as well as FINE logs from jenkins.branch.WorkspaceLocatorImpl.
I do understand why people are upset by this change.
However, if you want the original behavior, you can set the below as an option on your jenkins java vm setup (for example, in /etc/default/jenkins)
Setting this back to the original behavior has generally satisfied us: I hope this option is not removed. But for those which this changed caused a problem, the above is a pretty OK solution (IMO)