Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60969

@libs pseudo-workspace collision due to branch name truncation

      I've spot an issue with unwanted interaction between different, independent jobs.

      Here are the details: Our Jenkins instance (2.176.1 LTS) has some global pipeline libraries configured. My task was some kind of structural migration on a bunch of different projects. So I created a corresponding branch on each of these projects, did the change there and pushed it. So all the build jobs started at the same time and surprisingly most of them failed with a strange error during initial jobs steps like this:

      ...
      remote: Compressing objects: 100% (6/6), done.        
      remote: Total 8 (delta 0), reused 0 (delta 0)        
      error: cannot lock ref 'refs/remotes/origin/bugfix/ESTA-3146-yla': Unable to create '/var/jenkins_home/workspace/rate-build-description-to-json_5@libs/pipeline-helper/.git/refs/remotes/origin/bugfix/ESTA-3146-yla.lock': File exists.
      
      Another git process seems to be running in this repository, e.g.
      ...
      

      After digging around I figured out, that the pipeline helper libs are cloned on the Jenkins master using a folder named like the job a/o branch, which is suffixed with `_<buildnumber>@libs`. Additionally the folder name is stripped to 37 characters by removing characters from the beginning of the name. This leads to the real cause: Because of the identical branch name on each Git repository, the unique part of the folder names got lost and so the jobs try to use the same folder during their initialization steps respectively the cloning of the pipeline helper libs. As soon as the simultaneous running jobs have a differnt build number, it works as expected.

      The branch was named "feature/KIHUB-7882-separate-build-description-to-json" on all modified projects respectively Git repositories and so at the master it looks like this right now:

      $ cd /var/jenkins_home/workspace
      $ ls -1
      ...
      rate-build-description-to-json_2@libs
      rate-build-description-to-json_3@libs
      rate-build-description-to-json_4@libs
      rate-build-description-to-json_5@libs
      rate-build-description-to-json_6@libs
      rate-build-description-to-json_7@libs
      rate-build-description-to-json_8@libs
      rate-build-description-to-json_9@libs
      ...
      $
      

      So for me there are two questions and two possible solutions:
      1. Why is the folder name stripped?
      2. Why is the folder structure "flat" at this point and not like on the build jobs themself?

      I think there should be at least one folder level in between like this pseudo code example:

      /var/jenkins_home/workspace/OrgaScannerA/JobA-A/rate-build-description-to-json_3@libs
      /var/jenkins_home/workspace/OrgaScannerA/JobA-B/rate-build-description-to-json_3@libs
      /var/jenkins_home/workspace/OrgaScannerB/JobB-A/rate-build-description-to-json_3@libs
      /var/jenkins_home/workspace/OrgaScannerB/JobB-B/rate-build-description-to-json_3@libs
      

      Or the folder name itself should be unique:

      /var/jenkins_home/workspace/OrgaScannerA-JobA-A-rate-build-description-to-json_3@libs
      /var/jenkins_home/workspace/OrgaScannerA-JobA-B-rate-build-description-to-json_3@libs
      /var/jenkins_home/workspace/OrgaScannerB-JobB-A-rate-build-description-to-json_3@libs
      /var/jenkins_home/workspace/OrgaScannerB-JobB-B-rate-build-description-to-json_3@libs
      

          [JENKINS-60969] @libs pseudo-workspace collision due to branch name truncation

          Jesse Glick added a comment -

          try -Djenkins.branch.WorkspaceLocatorImpl.MODE=ENABLED

          Jesse Glick added a comment - try -Djenkins.branch.WorkspaceLocatorImpl.MODE=ENABLED

          Daniel Estermann added a comment - - edited

          It doesn't seem to change anything, the folder names in ${JENKINS_HOME}/workspace look the same to me:

          Without -Djenkins.branch.WorkspaceLocatorImpl.MODE=ENABLED

          With -Djenkins.branch.WorkspaceLocatorImpl.MODE=ENABLED

          swf What do you think?

          Daniel Estermann added a comment - - edited It doesn't seem to change anything, the folder names in ${JENKINS_HOME}/workspace look the same to me: Without -Djenkins.branch.WorkspaceLocatorImpl.MODE=ENABLED With -Djenkins.branch.WorkspaceLocatorImpl.MODE=ENABLED swf What do you think?

          Yves Schumann added a comment -

          jglick: What functionality is enabled, if this option is used?

          Yves Schumann added a comment - jglick : What functionality is enabled, if this option is used?

          Jesse Glick added a comment -

          The workspace management system introduced in JENKINS-2111 is enabled for all projects, not merely multibranch. I had thought this would affect the pseudo-workspaces used by workflow-cps-global-lib, but on closer inspection that seems to not be the issue. Possibly a bug in WorkspaceLocatorImpl. Would need to dig into a reproduction test case. Is /var/jenkins_home/workspace/workspaces.txt not tracking the various branch projects?

          Jesse Glick added a comment - The workspace management system introduced in JENKINS-2111 is enabled for all projects, not merely multibranch. I had thought this would affect the pseudo-workspaces used by workflow-cps-global-lib , but on closer inspection that seems to not be the issue. Possibly a bug in WorkspaceLocatorImpl . Would need to dig into a reproduction test case. Is /var/jenkins_home/workspace/workspaces.txt not tracking the various branch projects?

          Daniel Estermann added a comment - - edited

          I've just checked /var/jenkins_home/workspace/workspaces.txt on my test instance. It contains indeed the following entries:

          <path_to_project>/<branch>
          <folder>
          

          But IMHO it doesn't help against different jobs working on the same git folder.

          Daniel Estermann added a comment - - edited I've just checked /var/jenkins_home/workspace/workspaces.txt on my test instance. It contains indeed the following entries: <path_to_project>/<branch> <folder> But IMHO it doesn't help against different jobs working on the same git folder.

          Jesse Glick added a comment -

          Not sure offhand. Bug somewhere.

          Jesse Glick added a comment - Not sure offhand. Bug somewhere.

          jglick May I hand over this issue to you?

          Daniel Estermann added a comment - jglick May I hand over this issue to you?

          Jesse Glick added a comment -

          No, I am not maintaining these plugins, sorry.

          Jesse Glick added a comment - No, I am not maintaining these plugins, sorry.

          Alexander added a comment -

          according to the web presented information for the plugins...

          branch-api-plugin maintainers are:
          jglick recena abayer svanoort rsandell dnusbaum carroll bitwiseman (bitwiseman) kshultz jtaboada 
           
          workflow-cps-global-lib-plugin is marked deprecated - maintainers are:

          jglick kktest5 abayer svanoort dnusbaum rsandell carroll bitwiseman kshultz jtaboada 

          Alexander added a comment - according to the web presented information for the plugins... branch-api-plugin maintainers are: jglick recena abayer svanoort rsandell dnusbaum carroll bitwiseman (bitwiseman) kshultz jtaboada     workflow-cps-global-lib-plugin is marked deprecated - maintainers are: jglick kktest5 abayer svanoort dnusbaum rsandell carroll bitwiseman kshultz jtaboada  

          Jesse Glick added a comment -

          Whatever might be going on here, JENKINS-70869 would offer a workaround.

          Jesse Glick added a comment - Whatever might be going on here, JENKINS-70869 would offer a workaround.

            smasher Daniel Estermann
            swf Yves Schumann
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: