Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50975

Concurrent Groovy Shared Library syncs on different jobs use same workspace root

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major

      We're encountering an issue where multiple jobs syncing a groovy shared library concurrently results in the wrong sync root being used for some of the jobs. For example, job1 and job2 are triggered at the same time and use the p4 plugin for syncing the groovy shared library. job1 will have the correct workspace root (e.g.  /var/lib/jenkins/jobs/job1/workspace%40libs/LIBRARYNAME) for the groovy shared library sync, but job2 will sync the groovy shared library to the same workspace root as job1 (e.g.  /var/lib/jenkins/jobs/job1/workspace%40libs/LIBRARYNAME). When job2 loads the groovy shared library, it references what the workspace root should be for job2 (e.g.  /var/lib/jenkins/jobs/job2/workspace%40libs/LIBRARYNAME), which was not sync'd to, and causes job2 to run based off of an old groovy shared library sync.

        1. error_logs.txt
          2 kB
        2. job1.txt
          4 kB
        3. job2.txt
          4 kB

          [JENKINS-50975] Concurrent Groovy Shared Library syncs on different jobs use same workspace root

          Paul Allen added a comment -

          Turns out EXECUTOR_NUMBER and WORKSPACE are not define when the client workspace is created, Jenkins sets these variables later.

          Plan B is to use a UUID and clean up (delete the client) after syncing the Library.

          Paul Allen added a comment - Turns out EXECUTOR_NUMBER and WORKSPACE are not define when the client workspace is created, Jenkins sets these variables later. Plan B is to use a UUID and clean up (delete the client) after syncing the Library.

          Prem Gangana added a comment - - edited

          > Having - is for uniformity sake since you were having jenkins-lib${NODE_NAME}-${JOB_ NAME}. 

          -> Deleting the client is actually what i was going to ask you about. Cleaning up the client is needed here, otherwise the clients are going to get stacked up.

          -> How about using BUILD_NUMBER ?

           

          Prem Gangana added a comment - - edited > Having - is for uniformity sake since you were having jenkins-lib${NODE_NAME}-${JOB_ NAME}.  -> Deleting the client is actually what i was going to ask you about. Cleaning up the client is needed here, otherwise the clients are going to get stacked up. -> How about using BUILD_NUMBER ?  

          Paul Allen added a comment -

          I have tried 'Plan B' a UUID and delete. 

          I have also exposed the delete option for Manual Workspaces (visible in FreeStyle and Pipeline 'checkout' steps).  Not too sure if I will release it like this as deleting the client for normal 'checkout' steps may effect polling.

          https://ci.jenkins.io/job/Plugins/job/p4-plugin/job/master/284/

          Paul Allen added a comment - I have tried 'Plan B' a UUID and delete.  I have also exposed the delete option for Manual Workspaces (visible in FreeStyle and Pipeline 'checkout' steps).  Not too sure if I will release it like this as deleting the client for normal 'checkout' steps may effect polling. https://ci.jenkins.io/job/Plugins/job/p4-plugin/job/master/284/

          Paul Allen added a comment -

          Ready for release.

          Paul Allen added a comment - Ready for release.

          Paul Allen added a comment -

          Released in 1.9.3

          Paul Allen added a comment - Released in 1.9.3

          Jay Spang added a comment - - edited

          I know this issue is old, but I think it's repro'ing for me on Jenkins 2.164.3 with P4 Plugin 1.9.7.

          Here is an example pipeline that repros the error. Note that it DYNAMICALLY loads a pipeline.

          library(
              identifier: 'dynamically-loaded-library@now', retriever: legacySCM(
                  [
                      $class: 'PerforceScm', credential: 'p4creds',
                      populate: [
                          $class: 'AutoCleanImpl', 
                          delete: true, 
                          modtime: false, 
                          parallel: [enable: false, minbytes: '1024', minfiles: '1', threads: '4'], 
                          pin: '', 
                          quiet: true, 
                          replace: true, 
                          tidy: false
                      ], 
                      workspace: [
                          $class: 'ManualWorkspaceImpl', 
                          charset: 'none', 
                          name: "jenkins-${env.JOB_NAME}-dynamically-loaded-library",
                          pinHost: false, 
                          spec: [allwrite: true, clobber: false, compress: false, line: 'LOCAL', locked: false, modtime: false, rmdir: false, streamName: '', 
                              view: "//depot/build-library/... //jenkins-${env.JOB_NAME}-dynamically-loaded-library/..."
                          ]
                      ]
                  ]
              )
          )
          
          node() {
              stage('Echo') {
                  echo("If you hit 'Build Now' rapidly enough, a build wiill fail before it even gets here.")
              }
          }
          
          

          If you hit "Build Now" once, it works great. If you rapidly hit it a few times, several builds will fail with this error.

          P4 Task: reverting all pending and shelved revisions.
          ... p4 revert d:\JenkinsHome\jobs\stream_name\workspac___
           -
          p4 revert d:\JenkinsHome\jobs\stream_name\jobs\stream_name\workspace_libs\dynamically-loaded-library_2/...
          
          ERROR: P4: Task Exception: hudson.AbortException: P4JAVA: Error(s):
          Path 'd:\JenkinsHome\jobs\stream_name\workspace_libs\dynamically-loaded-library_2/...' is not under client's root 'd:\JenkinsHome\jobs\stream_name\workspace_libs\dynamically-loaded-library'.
          

          It seems like when two builds are run in a VERY close timeframe, the plugin will append "_2" to the library's foldername, but re-uses the old p4 workspace.

          EDIT: Edited my original comment to be more concise, with a working example.

          Jay Spang added a comment - - edited I know this issue is old, but I think it's repro'ing for me on Jenkins 2.164.3 with P4 Plugin 1.9.7. Here is an example pipeline that repros the error. Note that it DYNAMICALLY loads a pipeline. library( identifier: 'dynamically-loaded-library@now' , retriever: legacySCM( [ $class: 'PerforceScm' , credential: 'p4creds' , populate: [ $class: 'AutoCleanImpl' , delete: true , modtime: false , parallel: [enable: false , minbytes: '1024' , minfiles: '1' , threads: '4' ], pin: '', quiet: true , replace: true , tidy: false ], workspace: [ $class: 'ManualWorkspaceImpl' , charset: 'none' , name: "jenkins-${env.JOB_NAME}-dynamically-loaded-library" , pinHost: false , spec: [allwrite: true , clobber: false , compress: false , line: 'LOCAL' , locked: false , modtime: false , rmdir: false , streamName: '', view: " //depot/build-library/... //jenkins-${env.JOB_NAME}-dynamically-loaded-library/..." ] ] ] ) ) node() { stage( 'Echo' ) { echo( "If you hit 'Build Now' rapidly enough, a build wiill fail before it even gets here." ) } } If you hit "Build Now" once, it works great. If you rapidly hit it a few times, several builds will fail with this error. P4 Task: reverting all pending and shelved revisions. ... p4 revert d:\JenkinsHome\jobs\stream_name\workspac___ - p4 revert d:\JenkinsHome\jobs\stream_name\jobs\stream_name\workspace_libs\dynamically-loaded-library_2/... ERROR: P4: Task Exception: hudson.AbortException: P4JAVA: Error(s): Path 'd:\JenkinsHome\jobs\stream_name\workspace_libs\dynamically-loaded-library_2/...' is not under client 's root ' d:\JenkinsHome\jobs\stream_name\workspace_libs\dynamically-loaded-library'. It seems like when two builds are run in a VERY close timeframe, the plugin will append "_2" to the library's foldername, but re-uses the old p4 workspace. EDIT: Edited my original comment to be more concise, with a working example.

          Hi Jay, 

          I'm currently trying to reproduce your issue, so far I have been unable to. If possible could you share your library file?

          Thanks,

          Matthew

          Matthew Smeeth added a comment - Hi Jay,  I'm currently trying to reproduce your issue, so far I have been unable to. If possible could you share your library file? Thanks, Matthew

          Jay Spang added a comment - - edited

          To be extra clear, it only repros for me with dynamically loaded libraries.

          If you import your library with...

          library 'static-global-library@now'

          it doesn't happen. But if you load your library with...

          library(identifier: 'dynamically-loaded-library@now', retriever: legacySCM( // ... ) )

          it happens. Just click "Build Now" as rapidly as you you can 4-5 times.

          I can still send you a library if you'd like, but the failure happens before it even loads, so its contents don't matter.

          Jay Spang added a comment - - edited To be extra clear, it only repros for me with dynamically loaded libraries. If you import your library with... library ' static -global-library@now' it doesn't happen. But if you load your library with... library(identifier: 'dynamically-loaded-library@now' , retriever: legacySCM( // ... ) ) it happens. Just click "Build Now" as rapidly as you you can 4-5 times. I can still send you a library if you'd like, but the failure happens before it even loads, so its contents don't matter.

          Jay Spang added a comment -

          Hey msmeeth, I was wondering if you had any luck repro'ing the error? The key is to use "dynamically loaded" libraries (where you specify the `retriever` property). You also have to hit "Build Now" very rapidly to get it to repro.

          I hypothesize that p4groovy isn't changing the workspace root when it decides to append "_2" to the folder name. Since it works properly when specifying global libraries (that were previously loaded into the Jenkins global config), maybe it uses a slightly different code path.

          Jay Spang added a comment - Hey msmeeth , I was wondering if you had any luck repro'ing the error? The key is to use "dynamically loaded" libraries (where you specify the `retriever` property). You also have to hit "Build Now" very rapidly to get it to repro. I hypothesize that p4groovy isn't changing the workspace root when it decides to append "_2" to the folder name. Since it works properly when specifying global libraries (that were previously loaded into the Jenkins global config), maybe it uses a slightly different code path.

          Siri added a comment -

          We are facing the below error as well , is there any update on this ticket?

          Jenkins version : 2.361.4

          P4 plugin version : 1.13.3

           

          Error :

          ERROR: P4: Task Exception: hudson.AbortException: P4JAVA: Error(s): Path '/data/home/ldmqa/jenkins/workspace/Trigger_CD/...' is not under client's root '/data/home/ldmqa/jenkins/workspace/Trigger_CD%402'.  

          Siri added a comment - We are facing the below error as well , is there any update on this ticket? Jenkins version : 2.361.4 P4 plugin version : 1.13.3   Error : ERROR: P4: Task Exception: hudson.AbortException: P4JAVA: Error(s): Path '/data/home/ldmqa/jenkins/workspace/Trigger_CD/...' is not under client 's root ' /data/home/ldmqa/jenkins/workspace/Trigger_CD%402'.

            p4paul Paul Allen
            kroutley Kurt Routley
            Votes:
            2 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: