Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42411

Pipeline P4_CHANGELIST populated from Jenkinsfile changelist, not from repo p4sync mappings in Jenkinsfile

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Blocker Blocker
    • p4-plugin, pipeline
    • None
    • Jenkins 2.48, P4 Plugin 1.4.14

      I am noticing an odd issue where our P4_CHANGELIST number is populated per the last set of changes to the Jenkinfs file, but not for any 'p4sync' calls in the Jenkinsfile itself.

      As an example, here's the initial Jenkinsfile sync snippet, notice the changelist "961423". From here on out, the P4_CHANGELIST variable will have that as it's changelist number.

      [... snip ...]
      P4 Task: reverting all pending and shelved revisions.
      ... p4 revert /home/jenkins/jobs/build-tip/workspace%40script/... +
      ... rm [abandoned files]
      duration: (106ms)
      
      ... p4 sync /home/jenkins/jobs/build-tip/workspace%40script/...#none +
      ... rm -rf /home/jenkins/jobs/build-tip/workspace%40script
      
      P4 Task: syncing files at change: 961423
      ... p4 sync -p /home/jenkins/jobs/build-tip/workspace%40script/...@961423 +
      duration: (158ms)
      
      P4 Task: saving built changes.
      Found last change 961423 on syncID jenkins-build-tip
      ... p4 client -o jenkins-build-tip +
      ... p4 info +
      ... p4 client -o jenkins-build-tip +
      ... p4 info +
      ... done
      [... snip ...]
      

      A little farther in my build, I have my actual repo mappings. Here's the sync log from it:

      [... snip ...]
      P4 Task: reverting all pending and shelved revisions.
      ... p4 revert /home/jenkins/workspace/build-tip/... +
      ... rm [abandoned files]
      duration: (160ms)
      
      ... p4 sync /home/jenkins/workspace/build-tip/...#none +
      ... rm -rf /home/jenkins/workspace/build-tip
      
      P4 Task: syncing files at change: 968942
      ... p4 sync -p /home/jenkins/workspace/build-tip/...@968942 +
      duration: 9m 8s
      
      P4 Task: saving built changes.
      Found last change 968924 on syncID jenkins-build-01
      ... p4 client -o jenkins-build-01 +
      ... p4 info +
      ... p4 changes //jenkins-build-01/...@968924,968942 +
      ... p4 changes -l -m1 @968942 +
      ... p4 user -o user1 +
      ... p4 describe -s 968942 +
      ... p4 fixes -c968942 +
      ... p4 changes -l -m1 @968936 +
      ... p4 user -o user2 +
      ... p4 describe -s 968936 +
      ... p4 fixes -c968936 +
      ... done
      [... snip ...]
      

      In the case above, I would expect P4_CHANGELIST to be 968942, as it is the newest changelist for the repo at that point in the Jenkinsfile. If I were to have a workflow step right after my p4sync call that did something like this:

      sh "echo ${env.P4_CHANGELIST}"
      

      I would want the output to be '968942', but right now, it outputs '961423'.

          [JENKINS-42411] Pipeline P4_CHANGELIST populated from Jenkinsfile changelist, not from repo p4sync mappings in Jenkinsfile

          Paul Allen added a comment -

          There are two syncs occurring: one to fetch the Jenkinsfile and the other to fetch your source as described in the Jenkinsfile by 'p4sync' or 'checkout' DSL.  You MUST use different workspace names as the workspace names form part of the syncID used to record what was synced in a build.

          I normally postfix my Jenkinsfile workspace with '-script' and limit the view to just include the Jenkinsfile.  The sync stage in the script I leave the Workspace name to the default format value. Now I get two unique 'syncID's in the build XML data and can differentiate the changelists when computing polling or change reporting.

          Environment variables have always been problematic in Jenkins and depend heavily on scope, once set Jenkins seems to be reluctant to update them.  Please double check your workspace naming convention and if it is still a problem I'll take a closer look.

          Paul Allen added a comment - There are two syncs occurring: one to fetch the Jenkinsfile and the other to fetch your source as described in the Jenkinsfile by 'p4sync' or 'checkout' DSL.  You MUST use different workspace names as the workspace names form part of the syncID used to record what was synced in a build. I normally postfix my Jenkinsfile workspace with '-script' and limit the view to just include the Jenkinsfile.  The sync stage in the script I leave the Workspace name to the default format value. Now I get two unique 'syncID's in the build XML data and can differentiate the changelists when computing polling or change reporting. Environment variables have always been problematic in Jenkins and depend heavily on scope, once set Jenkins seems to be reluctant to update them.  Please double check your workspace naming convention and if it is still a problem I'll take a closer look.

          Kyle Mott added a comment - - edited

          That is what I am doing now. Here's the project checkout definition for Jenkinsfile:

          Workspace Name: jenkins-project-tip-jenkinsfile
          View Mapping: //depot/project/tip/Jenkinsfile //jenkins-project-tip-jenkinsfile/Jenkinsfile
          

          And here's what a checkout DSL looks like from that above Jenkinsfile (there's a couple, all with unique workspace client names). Note that the var $p4DepotPath is "//depot/project/tip" below:

                          checkout(
                              [
                                  $class: 'PerforceScm',
                                  credential: "${p4CredentialsUUID}",
                                  populate:
                                      [
                                          $class: 'ForceCleanImpl',
                                          have: false,
                                          pin: '',
                                          quiet: false,
                                          parallel: [ enable: false, minbytes: '1024', minfiles: '1', path: '/usr/bin/p4', threads: '4' ]
                                      ],
                                  workspace:
                                      [
                                          $class: 'ManualWorkspaceImpl',
                                          charset: 'none',
                                          name: "jenkins-${env.NODE_NAME}-${env.JOB_NAME}-build-${env.BUILD_NUMBER}",
                                          pinHost: true,
                                          spec:
                                              [
                                                  allwrite: false,
                                                  backup: false,
                                                  clobber: false,
                                                  compress: false,
                                                  line: 'LOCAL',
                                                  locked: false,
                                                  modtime: false,
                                                  rmdir: false,
                                                  serverID: '',
                                                  streamName: '',
                                                  type: 'WRITABLE',
                                                  view: "${p4DepotPath}... //jenkins-${env.NODE_NAME}-${env.JOB_NAME}-build-${env.BUILD_NUMBER}/..."
                                              ]
                                      ]
                              ]
                          )
          

          p4paul, should I be doing something different?

          Kyle Mott added a comment - - edited That is what I am doing now. Here's the project checkout definition for Jenkinsfile: Workspace Name: jenkins-project-tip-jenkinsfile View Mapping: //depot/project/tip/Jenkinsfile //jenkins-project-tip-jenkinsfile/Jenkinsfile And here's what a checkout DSL looks like from that above Jenkinsfile (there's a couple, all with unique workspace client names). Note that the var $p4DepotPath is "//depot/project/tip" below: checkout( [ $class: 'PerforceScm' , credential: "${p4CredentialsUUID}" , populate: [ $class: 'ForceCleanImpl' , have: false , pin: '', quiet: false , parallel: [ enable: false , minbytes: '1024' , minfiles: '1' , path: '/usr/bin/p4' , threads: '4' ] ], workspace: [ $class: 'ManualWorkspaceImpl' , charset: 'none' , name: "jenkins-${env.NODE_NAME}-${env.JOB_NAME}-build-${env.BUILD_NUMBER}" , pinHost: true , spec: [ allwrite: false , backup: false , clobber: false , compress: false , line: 'LOCAL' , locked: false , modtime: false , rmdir: false , serverID: '', streamName: '', type: 'WRITABLE' , view: "${p4DepotPath}... //jenkins-${env.NODE_NAME}-${env.JOB_NAME}-build-${env.BUILD_NUMBER}/..." ] ] ] ) p4paul , should I be doing something different?

          Paul Allen added a comment -

          SyncID's are derived from the workspace name and need to be unique across different workspaces (as the change number represents a different view), but unique across build history (so we can find them).  The 'env.BUILD_NUMBER' will break this as each build will create a unique SyncID and we won't be able to find the previous build change.

          To avoid this the code looks to replace BUILD_NUMBER, but I noticed recently users are adding 'env.' which will prevent the auto-replace. You could remove the 'env.BUILD_NUMBER' from your workspace or manually set the hidden field 'syncID':

          syncID: "jenkins-${env.NODE_NAME}-${env.JOB_NAME}"
          

          https://github.com/jenkinsci/p4-plugin/blob/master/WORKFLOW.md#advanced-polling

          Paul Allen added a comment - SyncID's are derived from the workspace name and need to be unique across different workspaces (as the change number represents a different view), but unique across build history (so we can find them).  The 'env.BUILD_NUMBER' will break this as each build will create a unique SyncID and we won't be able to find the previous build change. To avoid this the code looks to replace BUILD_NUMBER, but I noticed recently users are adding 'env.' which will prevent the auto-replace. You could remove the 'env.BUILD_NUMBER' from your workspace or manually set the hidden field 'syncID': syncID: "jenkins-${env.NODE_NAME}-${env.JOB_NAME}" https://github.com/jenkinsci/p4-plugin/blob/master/WORKFLOW.md#advanced-polling

          Kyle Mott added a comment -

          p4paul assuming I leave the mapping for the Jenkinsfile as-is:

          Workspace Name: jenkins-project-tip-script
          View Mapping: //depot/project/tip/Jenkinsfile //jenkins-project-tip-script/Jenkinsfile
          

          The only thing I need to do now is in each of my checkout DSL calls in the Jenkinsfile, adjust my workspace name to the following, without env.BUILD_NUMBER:

          name: "jenkins-${env.NODE_NAME}-${env.JOB_NAME}-build",
          

          And that should bring in changelists for that sync step for each build? We have multiple in our Jenkinsfile, one for build (above) and one for test (on a different node)--changelists should be available across both checkout() DSL steps in the build with the above adjustments?

          Kyle Mott added a comment - p4paul assuming I leave the mapping for the Jenkinsfile as-is: Workspace Name: jenkins-project-tip-script View Mapping: //depot/project/tip/Jenkinsfile //jenkins-project-tip-script/Jenkinsfile The only thing I need to do now is in each of my checkout DSL calls in the Jenkinsfile, adjust my workspace name to the following, without env.BUILD_NUMBER: name: "jenkins-${env.NODE_NAME}-${env.JOB_NAME}-build" , And that should bring in changelists for that sync step for each build? We have multiple in our Jenkinsfile, one for build (above) and one for test (on a different node)--changelists should be available across both checkout() DSL steps in the build with the above adjustments?

          Paul Allen added a comment - - edited

          Yes leave the Jenkinsfile workspace and view as-is.

          Within your Jenkins file:
          1. Try to use different workspace names for different syncs (especially if the view maps different files).

          • e.g. steps for sync source, sync assets, sync tests should all use different named workspaces.

          2. Ensure that subsequent builds use the same name for the workspace for the corresponding step (or override the syncID).

          • e.g. don't use a build ID, UUID or date/time in the workspace name.
             

          Paul Allen added a comment - - edited Yes leave the Jenkinsfile workspace and view as-is. Within your Jenkins file: 1. Try to use different workspace names for different syncs (especially if the view maps different files). e.g. steps for sync source, sync assets, sync tests should all use different named workspaces. 2. Ensure that subsequent builds use the same name for the workspace for the corresponding step (or override the syncID). e.g. don't use a build ID, UUID or date/time in the workspace name.  

          Kyle Mott added a comment -

          p4paul I think I got this resolved using your suggestions, thanks. No ghost builds are triggered, and each build has the correct changesets listed (from what I can tell). Feel free to close, if I run into this again, I will re-open.

          Kyle Mott added a comment - p4paul I think I got this resolved using your suggestions, thanks. No ghost builds are triggered, and each build has the correct changesets listed (from what I can tell). Feel free to close, if I run into this again, I will re-open.

          Paul Allen added a comment -

          Hi Kyle, thank you for letting me know the issues is resolves.
          I will close it as 'Not A Defect'.

          Paul Allen added a comment - Hi Kyle, thank you for letting me know the issues is resolves. I will close it as 'Not A Defect'.

          Thomas Murphy added a comment -

          Using jenkins 2.134 and P4 plugin 1.8.14, I have the problem described in the title of this bug. Using the option to pull a jenkinsfile from SCM, I find the $P4_CHANGELIST variable inside the jenkinsfile code is set to the most recent changelist for the pipeline script itself, and not for the overall project (that gets pulled in the P4 Sync stage of the pipeline). The logs show the P4 plugin does successfully pull the latest changelist (for the project) when it pulls the code in P4 Sync, but the P4_CHANGLIST variable is not reset to this changelist, and continues to identify the last change to the pipeline script. Based on the information in this bug (and the latest P4 Plugin documentation), I am using a different job name in the jenkins UI where I ask to pull the pipeline from SCM (jenkins-${NODE_NAME}${JOB_NAME}-script) and in the P4 Sync stage (jenkins${NODE_NAME}-${JOB_NAME}). 

           

          Thomas Murphy added a comment - Using jenkins 2.134 and P4 plugin 1.8.14, I have the problem described in the title of this bug. Using the option to pull a jenkinsfile from SCM, I find the $P4_CHANGELIST variable inside the jenkinsfile code is set to the most recent changelist for the pipeline script itself, and not for the overall project (that gets pulled in the P4 Sync stage of the pipeline). The logs show the P4 plugin does successfully pull the latest changelist (for the project) when it pulls the code in P4 Sync, but the P4_CHANGLIST variable is not reset to this changelist, and continues to identify the last change to the pipeline script. Based on the information in this bug (and the latest P4 Plugin documentation), I am using a different job name in the jenkins UI where I ask to pull the pipeline from SCM (jenkins-${NODE_NAME} ${JOB_NAME}-script) and in the P4 Sync stage (jenkins ${NODE_NAME}-${JOB_NAME}).   

          David added a comment -

          Hello, just wanted to add that I'm seeing similar behavior to the commented before me. I'm using the pipeline from SCM option with a different workspace, and during the build it's setting P4_CHANGELIST to the value in the first pipeline-only sync, and not getting updated when the subsequent sync pulls the code down. I think this is due to that P4_CHANGELIST variable being set only in the 'buildEnvironment' function, which I think only gets called once when setting up the environment. We can't think of a good in-jenkins way to fix this, and will workaround the issue in our own scripts for now.

          David added a comment - Hello, just wanted to add that I'm seeing similar behavior to the commented before me. I'm using the pipeline from SCM option with a different workspace, and during the build it's setting P4_CHANGELIST to the value in the first pipeline-only sync, and not getting updated when the subsequent sync pulls the code down. I think this is due to that P4_CHANGELIST variable being set only in the 'buildEnvironment' function, which I think only gets called once when setting up the environment. We can't think of a good in-jenkins way to fix this, and will workaround the issue in our own scripts for now.

          Raghu added a comment - - edited

          Hi, I am also seeing similar issue and it's kind of blocker, We needed to do some hacks to keep going for now but it is not efficient.

          Here is how our setup is, We have pipeline jobs and  two p4 clients for each job one p4 client is for getting jenkinsfile from p4(using pipeline from SCM option ) and at this point P4_CHANGELIST  is getting set to latest CL of that particular jenkinsfile.

           

          We have 2nd p4 client which we use for getting actual build related code and here even after syncing code from 2nd p4 client, Still P4_CHANGELIST has value of latest CL of that jenkinsfile but not from the latest CL from 2nd p4 client which we use for getting actual build related code.

           

          We needed this P4_CHANGELIST for further stages as part of our build flow and it is also blocking us from enabling percl builds.

           

          Raghu added a comment - - edited Hi, I am also seeing similar issue and it's kind of blocker, We needed to do some hacks to keep going for now but it is not efficient. Here is how our setup is, We have pipeline jobs and  two p4 clients for each job one p4 client is for getting jenkinsfile from p4(using pipeline from SCM option ) and at this point P4_CHANGELIST  is getting set to latest CL of that particular jenkinsfile.   We have 2nd p4 client which we use for getting actual build related code and here even after syncing code from 2nd p4 client, Still P4_CHANGELIST has value of latest CL of that jenkinsfile but not from the latest CL from 2nd p4 client which we use for getting actual build related code.   We needed this P4_CHANGELIST for further stages as part of our build flow and it is also blocking us from enabling percl builds.  

            Unassigned Unassigned
            kmott Kyle Mott
            Votes:
            5 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: