Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50975

Concurrent Groovy Shared Library syncs on different jobs use same workspace root

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major

      We're encountering an issue where multiple jobs syncing a groovy shared library concurrently results in the wrong sync root being used for some of the jobs. For example, job1 and job2 are triggered at the same time and use the p4 plugin for syncing the groovy shared library. job1 will have the correct workspace root (e.g.  /var/lib/jenkins/jobs/job1/workspace%40libs/LIBRARYNAME) for the groovy shared library sync, but job2 will sync the groovy shared library to the same workspace root as job1 (e.g.  /var/lib/jenkins/jobs/job1/workspace%40libs/LIBRARYNAME). When job2 loads the groovy shared library, it references what the workspace root should be for job2 (e.g.  /var/lib/jenkins/jobs/job2/workspace%40libs/LIBRARYNAME), which was not sync'd to, and causes job2 to run based off of an old groovy shared library sync.

        1. job1.txt
          4 kB
        2. job2.txt
          4 kB
        3. error_logs.txt
          2 kB

          [JENKINS-50975] Concurrent Groovy Shared Library syncs on different jobs use same workspace root

          Kurt Routley added a comment -

          Kurt Routley added a comment - stuartrowe mentioned it could be due to https://github.com/jenkinsci/workflow-cps-global-lib-plugin/blob/e080b501af942c15325fd17f974bec9cc04142c2/src/main/java/org/jenkinsci/plugins/workflow/libs/SCMSourceRetriever.java  , specifically the doRetrive method. It could also be a knock-on from https://issues.jenkins-ci.org/browse/JENKINS-40408    

          Paul Allen added a comment -

          Resolved in 1.8.12

          Paul Allen added a comment - Resolved in 1.8.12

          Sven Schubert added a comment -

          This bug still exists with version 1.8.14 of the P4 plugin, and Pipeline 2.5:  We have one shared library configured in Jenkins. If the Jenkins tries to start multiple jobs (A, B, C) at the same time (e.g. because an upstream project has been built and multiple jobs depend on this project), only the first job (A) will be able to load the shared library. The other jobs (B, C) get confused about the workspace when trying to load the shared library, e.g. job B tries to find the shared library (pipeline-commons) within the workspace of another job and not within his own workspace (error_logs.txt).

          If the jobs are not started at the same time, everything works fine. 

          Environment:

          • Pipeline: 2.5
          • Pipeline: Groovy (workflow-cps): 2.54 
          • Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.9

          Sven Schubert added a comment - This bug still exists with version 1.8.14 of the P4 plugin, and Pipeline 2.5:  We have one shared library configured in Jenkins. If the Jenkins tries to start multiple jobs (A, B, C) at the same time (e.g. because an upstream project has been built and multiple jobs depend on this project), only the first job (A) will be able to load the shared library. The other jobs (B, C) get confused about the workspace when trying to load the shared library, e.g. job B tries to find the shared library (pipeline-commons) within the workspace of another job and not within his own workspace ( error_logs.txt ). If the jobs are not started at the same time, everything works fine.  Environment: Pipeline: 2.5 Pipeline: Groovy (workflow-cps): 2.54  Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.9

          Paul Allen added a comment - - edited

          I have recently made changes to Global Library workspaces for multiple instances JENKINS-53922
          https://ci.jenkins.io/job/Plugins/job/p4-plugin/job/master/281/

          I may need to add ${EXECUTOR_NUMBER} to the Global Library Workspace name formatter:

          In file GlobalLibraryScmScurce.java:43

          setFormat("jenkins-lib-${NODE_NAME}-${JOB_NAME}" + id);
          
          setFormat("jenkins-lib-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}" + id);
          

          Did you want to try this out and build it yourself or I can submit a change for you to try on the nightly build?

           

          (Just re-reading your comment and logs - Build 281 might be enough without the EXECUTOR_NUMBER)

          Paul Allen added a comment - - edited I have recently made changes to Global Library workspaces for multiple instances JENKINS-53922 https://ci.jenkins.io/job/Plugins/job/p4-plugin/job/master/281/ I may need to add ${EXECUTOR_NUMBER } to the Global Library Workspace name formatter: In file GlobalLibraryScmScurce.java:43 setFormat( "jenkins-lib-${NODE_NAME}-${JOB_NAME}" + id); setFormat( "jenkins-lib-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}" + id); Did you want to try this out and build it yourself or I can submit a change for you to try on the nightly build?   (Just re-reading your comment and logs - Build 281 might be enough without the EXECUTOR_NUMBER)

          Prem Gangana added a comment -

          Hi p4paul,

          I have tested this with the build 281. It is using 
          "jenkins-lib-${NODE_NAME}-${JOB_NAME}" + id
          as client name now, but i would suggest replacing / with _ in id instead of . (dot). So this is unique across different jobs due to JOB_NAME in there.

          However, the issue will surface if a given job has two builds triggered at / almost at the same time, since the jenkins appends @<number> to the build workspace, and so the client root dirs will be different. I would suggest adding WORKSPACE base name some where in there.

           

          Prem Gangana added a comment - Hi p4paul , I have tested this with the build 281. It is using  "jenkins-lib-${NODE_NAME}-${JOB_NAME}" + id as client name now, but i would suggest replacing / with _ in id instead of . (dot). So this is unique across different jobs due to JOB_NAME in there. However, the issue will surface if a given job has two builds triggered at / almost at the same time, since the jenkins appends @<number> to the build workspace, and so the client root dirs will be different. I would suggest adding WORKSPACE base name some where in there.  

          Paul Allen added a comment -

          Happy to change . to {{_ }}but was curious to know why?  Do you have dots in your JOB_NAME?

          If the same Job is started concurrently then the EXECUTOR_NUMBER should be enough, this is the @1 or @2 that Jenkins adds to the directory. I think WORKSPACE would also work, but we could end up with very long names.  I had considered using a SHA1 of the NODE_NAME + JOB_NAME + depot path + EXECUTOR, but it becomes harder to debug if problems appear later on.

          Paul Allen added a comment - Happy to change .  to {{_ }}but was curious to know why?  Do you have dots in your JOB_NAME? If the same Job is started concurrently then the EXECUTOR_NUMBER should be enough, this is the @1 or @2 that Jenkins adds to the directory. I think WORKSPACE would also work, but we could end up with very long names.  I had considered using a SHA1 of the NODE_NAME + JOB_NAME + depot path + EXECUTOR, but it becomes harder to debug if problems appear later on.

          Paul Allen added a comment -

          Turns out EXECUTOR_NUMBER and WORKSPACE are not define when the client workspace is created, Jenkins sets these variables later.

          Plan B is to use a UUID and clean up (delete the client) after syncing the Library.

          Paul Allen added a comment - Turns out EXECUTOR_NUMBER and WORKSPACE are not define when the client workspace is created, Jenkins sets these variables later. Plan B is to use a UUID and clean up (delete the client) after syncing the Library.

          Prem Gangana added a comment - - edited

          > Having - is for uniformity sake since you were having jenkins-lib${NODE_NAME}-${JOB_ NAME}. 

          -> Deleting the client is actually what i was going to ask you about. Cleaning up the client is needed here, otherwise the clients are going to get stacked up.

          -> How about using BUILD_NUMBER ?

           

          Prem Gangana added a comment - - edited > Having - is for uniformity sake since you were having jenkins-lib${NODE_NAME}-${JOB_ NAME}.  -> Deleting the client is actually what i was going to ask you about. Cleaning up the client is needed here, otherwise the clients are going to get stacked up. -> How about using BUILD_NUMBER ?  

          Paul Allen added a comment -

          I have tried 'Plan B' a UUID and delete. 

          I have also exposed the delete option for Manual Workspaces (visible in FreeStyle and Pipeline 'checkout' steps).  Not too sure if I will release it like this as deleting the client for normal 'checkout' steps may effect polling.

          https://ci.jenkins.io/job/Plugins/job/p4-plugin/job/master/284/

          Paul Allen added a comment - I have tried 'Plan B' a UUID and delete.  I have also exposed the delete option for Manual Workspaces (visible in FreeStyle and Pipeline 'checkout' steps).  Not too sure if I will release it like this as deleting the client for normal 'checkout' steps may effect polling. https://ci.jenkins.io/job/Plugins/job/p4-plugin/job/master/284/

          Paul Allen added a comment -

          Ready for release.

          Paul Allen added a comment - Ready for release.

          Paul Allen added a comment -

          Released in 1.9.3

          Paul Allen added a comment - Released in 1.9.3

          Jay Spang added a comment - - edited

          I know this issue is old, but I think it's repro'ing for me on Jenkins 2.164.3 with P4 Plugin 1.9.7.

          Here is an example pipeline that repros the error. Note that it DYNAMICALLY loads a pipeline.

          library(
              identifier: 'dynamically-loaded-library@now', retriever: legacySCM(
                  [
                      $class: 'PerforceScm', credential: 'p4creds',
                      populate: [
                          $class: 'AutoCleanImpl', 
                          delete: true, 
                          modtime: false, 
                          parallel: [enable: false, minbytes: '1024', minfiles: '1', threads: '4'], 
                          pin: '', 
                          quiet: true, 
                          replace: true, 
                          tidy: false
                      ], 
                      workspace: [
                          $class: 'ManualWorkspaceImpl', 
                          charset: 'none', 
                          name: "jenkins-${env.JOB_NAME}-dynamically-loaded-library",
                          pinHost: false, 
                          spec: [allwrite: true, clobber: false, compress: false, line: 'LOCAL', locked: false, modtime: false, rmdir: false, streamName: '', 
                              view: "//depot/build-library/... //jenkins-${env.JOB_NAME}-dynamically-loaded-library/..."
                          ]
                      ]
                  ]
              )
          )
          
          node() {
              stage('Echo') {
                  echo("If you hit 'Build Now' rapidly enough, a build wiill fail before it even gets here.")
              }
          }
          
          

          If you hit "Build Now" once, it works great. If you rapidly hit it a few times, several builds will fail with this error.

          P4 Task: reverting all pending and shelved revisions.
          ... p4 revert d:\JenkinsHome\jobs\stream_name\workspac___
           -
          p4 revert d:\JenkinsHome\jobs\stream_name\jobs\stream_name\workspace_libs\dynamically-loaded-library_2/...
          
          ERROR: P4: Task Exception: hudson.AbortException: P4JAVA: Error(s):
          Path 'd:\JenkinsHome\jobs\stream_name\workspace_libs\dynamically-loaded-library_2/...' is not under client's root 'd:\JenkinsHome\jobs\stream_name\workspace_libs\dynamically-loaded-library'.
          

          It seems like when two builds are run in a VERY close timeframe, the plugin will append "_2" to the library's foldername, but re-uses the old p4 workspace.

          EDIT: Edited my original comment to be more concise, with a working example.

          Jay Spang added a comment - - edited I know this issue is old, but I think it's repro'ing for me on Jenkins 2.164.3 with P4 Plugin 1.9.7. Here is an example pipeline that repros the error. Note that it DYNAMICALLY loads a pipeline. library( identifier: 'dynamically-loaded-library@now' , retriever: legacySCM( [ $class: 'PerforceScm' , credential: 'p4creds' , populate: [ $class: 'AutoCleanImpl' , delete: true , modtime: false , parallel: [enable: false , minbytes: '1024' , minfiles: '1' , threads: '4' ], pin: '', quiet: true , replace: true , tidy: false ], workspace: [ $class: 'ManualWorkspaceImpl' , charset: 'none' , name: "jenkins-${env.JOB_NAME}-dynamically-loaded-library" , pinHost: false , spec: [allwrite: true , clobber: false , compress: false , line: 'LOCAL' , locked: false , modtime: false , rmdir: false , streamName: '', view: " //depot/build-library/... //jenkins-${env.JOB_NAME}-dynamically-loaded-library/..." ] ] ] ) ) node() { stage( 'Echo' ) { echo( "If you hit 'Build Now' rapidly enough, a build wiill fail before it even gets here." ) } } If you hit "Build Now" once, it works great. If you rapidly hit it a few times, several builds will fail with this error. P4 Task: reverting all pending and shelved revisions. ... p4 revert d:\JenkinsHome\jobs\stream_name\workspac___ - p4 revert d:\JenkinsHome\jobs\stream_name\jobs\stream_name\workspace_libs\dynamically-loaded-library_2/... ERROR: P4: Task Exception: hudson.AbortException: P4JAVA: Error(s): Path 'd:\JenkinsHome\jobs\stream_name\workspace_libs\dynamically-loaded-library_2/...' is not under client 's root ' d:\JenkinsHome\jobs\stream_name\workspace_libs\dynamically-loaded-library'. It seems like when two builds are run in a VERY close timeframe, the plugin will append "_2" to the library's foldername, but re-uses the old p4 workspace. EDIT: Edited my original comment to be more concise, with a working example.

          Hi Jay, 

          I'm currently trying to reproduce your issue, so far I have been unable to. If possible could you share your library file?

          Thanks,

          Matthew

          Matthew Smeeth added a comment - Hi Jay,  I'm currently trying to reproduce your issue, so far I have been unable to. If possible could you share your library file? Thanks, Matthew

          Jay Spang added a comment - - edited

          To be extra clear, it only repros for me with dynamically loaded libraries.

          If you import your library with...

          library 'static-global-library@now'

          it doesn't happen. But if you load your library with...

          library(identifier: 'dynamically-loaded-library@now', retriever: legacySCM( // ... ) )

          it happens. Just click "Build Now" as rapidly as you you can 4-5 times.

          I can still send you a library if you'd like, but the failure happens before it even loads, so its contents don't matter.

          Jay Spang added a comment - - edited To be extra clear, it only repros for me with dynamically loaded libraries. If you import your library with... library ' static -global-library@now' it doesn't happen. But if you load your library with... library(identifier: 'dynamically-loaded-library@now' , retriever: legacySCM( // ... ) ) it happens. Just click "Build Now" as rapidly as you you can 4-5 times. I can still send you a library if you'd like, but the failure happens before it even loads, so its contents don't matter.

          Jay Spang added a comment -

          Hey msmeeth, I was wondering if you had any luck repro'ing the error? The key is to use "dynamically loaded" libraries (where you specify the `retriever` property). You also have to hit "Build Now" very rapidly to get it to repro.

          I hypothesize that p4groovy isn't changing the workspace root when it decides to append "_2" to the folder name. Since it works properly when specifying global libraries (that were previously loaded into the Jenkins global config), maybe it uses a slightly different code path.

          Jay Spang added a comment - Hey msmeeth , I was wondering if you had any luck repro'ing the error? The key is to use "dynamically loaded" libraries (where you specify the `retriever` property). You also have to hit "Build Now" very rapidly to get it to repro. I hypothesize that p4groovy isn't changing the workspace root when it decides to append "_2" to the folder name. Since it works properly when specifying global libraries (that were previously loaded into the Jenkins global config), maybe it uses a slightly different code path.

          Siri added a comment -

          We are facing the below error as well , is there any update on this ticket?

          Jenkins version : 2.361.4

          P4 plugin version : 1.13.3

           

          Error :

          ERROR: P4: Task Exception: hudson.AbortException: P4JAVA: Error(s): Path '/data/home/ldmqa/jenkins/workspace/Trigger_CD/...' is not under client's root '/data/home/ldmqa/jenkins/workspace/Trigger_CD%402'.  

          Siri added a comment - We are facing the below error as well , is there any update on this ticket? Jenkins version : 2.361.4 P4 plugin version : 1.13.3   Error : ERROR: P4: Task Exception: hudson.AbortException: P4JAVA: Error(s): Path '/data/home/ldmqa/jenkins/workspace/Trigger_CD/...' is not under client 's root ' /data/home/ldmqa/jenkins/workspace/Trigger_CD%402'.

            p4paul Paul Allen
            kroutley Kurt Routley
            Votes:
            2 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: