Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67549

P4 preview populate option updates have list when triggered by swarm review

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • p4-plugin
    • P4 Plugin 1.11.6
      Jenkins 2.328 (Windows)
      Java 8

      When pipeline job is triggered by swarm, checkout perforce step with preview populate option is updating the client have list. There are unshelve and revert -k commands, related to swarm review, executed on the client, which are basically updating both workspace files and client have list.

      Here is the log from the end of checkout perforce with preview option:

      ...
      [2021-12-28T15:39:54.217Z] P4 Task: syncing files at change: 233853
      [2021-12-28T15:39:54.217Z] P4 Task: skipping sync.
      [2021-12-28T15:39:54.217Z] duration: (0ms)
      [2021-12-28T15:39:54.217Z] 
      [2021-12-28T15:39:54.217Z] P4 Task: unshelve review: 234089
      [2021-12-28T15:39:54.219Z] (p4):cmd:... p4 unshelve -f -s234089 -cdefault
      [2021-12-28T15:39:54.332Z] //sandbox_streams/development/project.txt - also opened by robot.obibok@robot.obibok_idk300_sandbox_streams
      [2021-12-28T15:39:54.332Z] 
      [2021-12-28T15:39:54.338Z] (p4):stop:12
      [2021-12-28T15:39:54.338Z] //sandbox_streams/development/project.txt
      [2021-12-28T15:39:54.338Z] ... duration: (121ms)
      [2021-12-28T15:39:54.339Z] (p4):cmd:... p4 revert -k C:\J_noAV\.agents\IDK447\workspace\pelines-test_sandbox_development/..___
      [2021-12-28T15:39:54.355Z] (p4):stop:13
      

      Full log from checkout step attached as p4preview.log
      The documentation for p4 preview populate option clearly states:

      No file sync or cleanup is attempted, and the Workspace "have list" is not updated.  

      In our use case, we use p4 plugin with preview option only to set up P4 env vars and then do all the p4 actions manually if needed. This unexpected behavior makes us to implement workarounds in case of swarm triggered builds, as the client have list must not be modified.

        1. config.xml
          4 kB
        2. consoleText.txt
          15 kB
        3. Jenkinsfile
          1 kB
        4. p4preview.log
          14 kB
        5. swarm.PNG
          swarm.PNG
          120 kB

          [JENKINS-67549] P4 preview populate option updates have list when triggered by swarm review

          Karl Wirth added a comment -

          Hi plawniczuk - Can you please send me the job definition in Jenkins and your Jenkinsfile. I'm trying to decide if we are falling back to implict sync. If they contain confidential information please send them to support@perforce.com for my attention.

          Thanks in advance,

          Karl

          Karl Wirth added a comment - Hi plawniczuk - Can you please send me the job definition in Jenkins and your Jenkinsfile. I'm trying to decide if we are falling back to implict sync. If they contain confidential information please send them to support@perforce.com for my attention. Thanks in advance, Karl

          Job configuration in Jenkins config.xml

          Jenkinsfile Jenkinsfile

          Full console log: consoleText.txt

          Swarm test configuration 

          Piotr Ławniczuk added a comment - Job configuration in Jenkins  config.xml Jenkinsfile  Jenkinsfile Full console log:  consoleText.txt Swarm test configuration 

          Karl Wirth added a comment -

          plawniczuk - Thanks for the data. I was able to reproduce this and the problem is the implicit unshelve caused by 'Build Review' assumes you have given control of the workspace to Jenkins and is seperate to the sync step which as you show above does skip the sync correctly.

          Can you please expand on what setting the variables does for you please and why you then perform operations outside the build job against the same workspace.

           

          Note for QA - Reproduction steps:

          (1) Create a static workspace on the Jenkins build machine and sync and open for edit files. For example:

          $ p4 client -o
          Client:	STATIC
          Update:	2022/01/17 14:11:06
          Access:	2022/01/17 14:15:47
          Owner:	super
          Description:
          Root:	/var/lib/jenkins/workspace/STATIC
          Options:	noallwrite noclobber nocompress unlocked nomodtime normdir
          SubmitOptions:	submitunchanged
          LineEnd:	local
          View:
          	//depot/project1/static/main/... //STATIC/...
          

          (2) Submit a Jenkinsfile as '//depot/project1/static/main/Jenkinsfile' ensuring your Perforce credential is called 'JenkinsCredential':

          pipeline {
              agent {
                  node {
                      label 'master'
                  }
              }
              triggers {
                  p4Trigger()
              }
              options {
                  buildDiscarder logRotator(numToKeepStr: '31')
                  disableResume()
                  disableConcurrentBuilds()
                  timestamps()
                  skipDefaultCheckout()
              }
              stages {
                  stage('Setup Pipeline'){
                      steps {
                          checkout perforce(
                              credential: 'JenkinsCredential',
                              populate: previewOnly(
                                  quiet: true
                              ),
                              workspace: staticSpec(
                                  charset: 'utf8',
                                  name: 'STATIC',
                                  pinHost: false
                              )
                          )
                          //SetupPipeline()
                      }
                  }
                  stage('Test'){
                      steps {
                          echo 'Test'
                      }
                  }
              }
          }
          
          

          (3) In Jenkins create a pipeline job with Jenkinsfile from SCM.

          (4) Build job manually once. No files in workspace are changed.

          (5) Shelve a file for path '//depot/project1/static/main/...'

          p4 edit //depot/project1/static/main/f1
          p4 shelve //depot/project1/static/main/f1
          

          (6) In Jenkins using 'Build Review' , specify shelve changelist number from step (5) under 'review' and 'shelved' under 'status'.

          (7) Result is the preview sync runs then shelve is unshelved with 'p4 revert':

          14:15:47  P4 Task: syncing files at change: 286   # Correct behaviour of preview sync.
          14:15:47  P4 Task: skipping sync.
          14:15:47  duration: (0ms)
          14:15:47  
          14:15:47  P4 Task: unshelve review: 287
          14:15:47  (p4):cmd:... p4 unshelve -f -s287 -cdefault     # This needs to update have list
          14:15:47  (p4):stop:5
          14:15:47  //depot/project1/static/main/f286
          14:15:47  ... duration: (10ms)
          14:15:47  (p4):cmd:... p4 revert -k /var/lib/jenkins/workspace/STATIC/...     # This reverts manually opened files.
          14:15:47  (p4):stop:6
          14:15:47  P4: saving built changes.
          
          

           

           

           

           

           

          Karl Wirth added a comment - plawniczuk - Thanks for the data. I was able to reproduce this and the problem is the implicit unshelve caused by 'Build Review' assumes you have given control of the workspace to Jenkins and is seperate to the sync step which as you show above does skip the sync correctly. Can you please expand on what setting the variables does for you please and why you then perform operations outside the build job against the same workspace.   Note for QA - Reproduction steps: (1) Create a static workspace on the Jenkins build machine and sync and open for edit files. For example: $ p4 client -o Client: STATIC Update: 2022/01/17 14:11:06 Access: 2022/01/17 14:15:47 Owner: super Description: Root: / var /lib/jenkins/workspace/STATIC Options: noallwrite noclobber nocompress unlocked nomodtime normdir SubmitOptions: submitunchanged LineEnd: local View: //depot/project1/ static /main/... //STATIC/... (2) Submit a Jenkinsfile as '//depot/project1/static/main/Jenkinsfile' ensuring your Perforce credential is called 'JenkinsCredential': pipeline { agent { node { label 'master' } } triggers { p4Trigger() } options { buildDiscarder logRotator(numToKeepStr: '31' ) disableResume() disableConcurrentBuilds() timestamps() skipDefaultCheckout() } stages { stage( 'Setup Pipeline' ){ steps { checkout perforce( credential: 'JenkinsCredential' , populate: previewOnly( quiet: true ), workspace: staticSpec( charset: 'utf8' , name: 'STATIC' , pinHost: false ) ) //SetupPipeline() } } stage( 'Test' ){ steps { echo 'Test' } } } } (3) In Jenkins create a pipeline job with Jenkinsfile from SCM. (4) Build job manually once. No files in workspace are changed. (5) Shelve a file for path '//depot/project1/static/main/...' p4 edit //depot/project1/ static /main/f1 p4 shelve //depot/project1/ static /main/f1 (6) In Jenkins using 'Build Review' , specify shelve changelist number from step (5) under 'review' and 'shelved' under 'status'. (7) Result is the preview sync runs then shelve is unshelved with 'p4 revert': 14:15:47 P4 Task: syncing files at change: 286 # Correct behaviour of preview sync. 14:15:47 P4 Task: skipping sync. 14:15:47 duration: (0ms) 14:15:47 14:15:47 P4 Task: unshelve review: 287 14:15:47 (p4):cmd:... p4 unshelve -f -s287 -cdefault # This needs to update have list 14:15:47 (p4):stop:5 14:15:47 //depot/project1/ static /main/f286 14:15:47 ... duration: (10ms) 14:15:47 (p4):cmd:... p4 revert -k / var /lib/jenkins/workspace/STATIC/... # This reverts manually opened files. 14:15:47 (p4):stop:6 14:15:47 P4: saving built changes.          

          We operate on rather large workspaces, around ~700GB of source data, and we do all p4 related commands manually inside the pipeline script, just to have total control of the client/have list. We use p4 preview step usually just to get the current head CL number for the client, and then environment variables can be easily used with other scripts on the machine. I guess it's a bit unusual way to handle Jenkins builds but it works with our custom scripts and libs. 

          Piotr Ławniczuk added a comment - We operate on rather large workspaces, around ~700GB of source data, and we do all p4 related commands manually inside the pipeline script, just to have total control of the client/have list. We use p4 preview step usually just to get the current head CL number for the client, and then environment variables can be easily used with other scripts on the machine. I guess it's a bit unusual way to handle Jenkins builds but it works with our custom scripts and libs. 

          Karl Wirth added a comment -

          Discussed with development. Will be investigated during an upcoming sprint.

          Karl Wirth added a comment - Discussed with development. Will be investigated during an upcoming sprint.

          We also use preview sync followed by a revert/reconcile/sync script to avoid performance issues we hit with the "auto cleanup and sync" mode.

          Robert Cowsill added a comment - We also use preview sync followed by a revert/reconcile/sync script to avoid performance issues we hit with the "auto cleanup and sync" mode.

          Karl Wirth added a comment -

          rcowsill - Thanks for the feedback. If you havent already can you also vote up the job. I'll make sure the developers see this.

          Karl Wirth added a comment - rcowsill - Thanks for the feedback. If you havent already can you also vote up the job. I'll make sure the developers see this.

            Unassigned Unassigned
            plawniczuk Piotr Ławniczuk
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: