Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59657

Concurrent runs of a pipeline job that uses a custom workspaces cause p4sync to fail occassionally

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • p4-plugin
    • p4-plugin: 1.9.6
      workflow-durable-task-step: 2.29
      Jenkins: 2.168

      I have a number of jobs that are named differently, and do similar but different operations, but have the same perforce mapping and run the same setup.

      I wanted to share a workspace amongst all those jobs, so that i can benefit from the p4sync and setup caching. 

      I wrote my jobs like this:

      node('myNodeLabel') {
          ws('C:\\MyCustomWorkspace') {
              stage('Prepare node') {
                def p4WsName = 'custom-workspace-${NODE_NAME}'
                p4sync credential: 'myCred',
                 changelog: false,
                 populate: autoClean(replace: true, quiet: true, delete:false),
                  workspace: [
                    $class: 'ManualWorkspaceImpl',
                    charset: 'utf8',
                    pinHost: false,
                    name: p4WsName,
                    spec: [
                      allwrite: true,
                      backup: false,
                      clobber: true,
                      compress: false,
                      line: 'LOCAL',
                      locked: false,
                      modtime: false,
                      rmdir: true,
                      streamName: '',
                      type: 'WRITABLE',
                      view: """\
                        //depot/... //${p4WsName}/..."""
                    ]
                  ]
              }
              // Do stuff
          }
      }
       

      When 2 instances of the job run on the same node, jenkins correctly allocates second workspace at C:\MyCustomWorkspace@2, syncs it etc.

      After the concurrent runs have finished and I run the job again, there is a high chance that the job will fail during the p4sync step with an error message that's indicating that the p4 root is set incorrectly:
      > Path 'C:\MyCustomWorkspace/...' is not under client's root 'C:\MyCustomWorkspace%402'.
      More complete console log of the p4 operations:

      P4 Task: reverting all pending and shelved revisions.
      ... p4 revert C:\MyCustomWorkspace/...
       +
      ... rm [abandoned files]
      duration: (15ms)
      
      P4 Task: cleaning workspace to match have list.
      ... p4 reconcile -w -f -e -d C:\MyCustomWorkspace/...
       +
      duration: 0m 10s
      
      P4 Task: syncing files at change: XXXXXXX
      ... p4 sync -q C:\MyCustomWorkspace/...@XXXXXXX
       +
      P4 Task: attempt: 1
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // ws
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      [BFA] Scanning build for known causes...
      [BFA] No failure causes found
      [BFA] Done. 0s
      ERROR: P4: Task Exception: com.perforce.p4java.exception.P4JavaException: com.perforce.p4java.exception.P4JavaException: hudson.AbortException: P4JAVA: Error(s):
      Path 'C:\MyCustomWorkspace/...' is not under client's root 'C:\MyCustomWorkspace%402'.
      

      Repro steps:
      1. Create a pipeline job using the code I pasted above and pin it on a single node.
      2. Change the //Do stuff to something like a 1 min sleep
      3. Start 2 runs of the job, which should run on the same node, and verify that jenkins had created 2 different workspaces. You might want to wait until the p4 sync of the first run has finished before starting the second one.
      4. After the 2 runs have finished, start a 3rd run
      5. The 3rd run will probably fail.

      Note: I tried creating my custom workspace inside my jenkins workspace root, and experienced the same issue.

          [JENKINS-59657] Concurrent runs of a pipeline job that uses a custom workspaces cause p4sync to fail occassionally

          The error message I saw is similar to the one described in https://issues.jenkins-ci.org/browse/JENKINS-50975

          Vasiliki Siakka added a comment - The error message I saw is similar to the one described in https://issues.jenkins-ci.org/browse/JENKINS-50975

          Karl Wirth added a comment - - edited

          Hi vsiakka - Thanks for raisining this.

          Sharing a workspace between jobs is not supported because the root will change for each when job when Jenkins decides to use a different location as you have found. We recommend that the workspace naming convention is:

          jenkins-${NODE_NAME}${JOB_NAME}${EXECUTOR_NUMBER}

            https://www.perforce.com/manuals/jenkins/Content/P4Jenkins/variable-expansion.html#Workspace_name

          This ties that one workspace to the job, executor and node it's running on so in theory is unique.

          Please also note that Jenkins has a bug in EXECUTOR_NUMBER (JENKINS-48882) that will sometime cause multiple concurrent jobs on the same node and different executors to fail. We have defensive code to try and work around the bug but sometimes we cannot avoid that failure. 

            

           

           

          Karl Wirth added a comment - - edited Hi vsiakka - Thanks for raisining this. Sharing a workspace between jobs is not supported because the root will change for each when job when Jenkins decides to use a different location as you have found. We recommend that the workspace naming convention is: jenkins-${NODE_NAME}${JOB_NAME}${EXECUTOR_NUMBER}   https://www.perforce.com/manuals/jenkins/Content/P4Jenkins/variable-expansion.html#Workspace_name This ties that one workspace to the job, executor and node it's running on so in theory is unique. Please also note that Jenkins has a bug in EXECUTOR_NUMBER ( JENKINS-48882 ) that will sometime cause multiple concurrent jobs on the same node and different executors to fail. We have defensive code to try and work around the bug but sometimes we cannot avoid that failure.        

          Thank you for the reply p4karl. Would you consider supporting this in the future, should I change this to a feature request?

          Vasiliki Siakka added a comment - Thank you for the reply p4karl . Would you consider supporting this in the future, should I change this to a feature request?

          Karl Wirth added a comment -

          Hi vsiakka

          Sorry for the delay. I was going through old tickets and noticed I missed your update.

          What part would you like to be an enhancement request? The fact that two build jobs cannot use the same workspace concurrently is fundamental to the way Perforce tracks what is in the workspace (it keeps a 'have' list). Also you wouldn't be able to trust the build results if potentially it had been built from different versions of the software.

          Thanks in advance,

          Karl

          Karl Wirth added a comment - Hi vsiakka Sorry for the delay. I was going through old tickets and noticed I missed your update. What part would you like to be an enhancement request? The fact that two build jobs cannot use the same workspace concurrently is fundamental to the way Perforce tracks what is in the workspace (it keeps a 'have' list). Also you wouldn't be able to trust the build results if potentially it had been built from different versions of the software. Thanks in advance, Karl

            p4karl Karl Wirth
            vsiakka Vasiliki Siakka
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: