Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-72761

Provide better populate options for cloud nodes

      When working with cloud nodes (where the node may only exist for a single build before being replaced), the Auto cleanup and sync and Sync only populate options can lead to errors where files are not appropriately synced from Perforce Helix.

      • With Auto cleanup and sync, use of p4 resolve prior to p4 sync can lead to spurious errors "cannot clobber writable file" because the havelist doesn't match what exists in the workspace (because the workspace is empty from being on a brand new node).
      • With Sync only, Perforce relies on the inaccurate havelist and only syncs new files, leading to all other files being missing from the build.

      The only option that works reliably for us right now is the "not recommended" Forced clean and sync. This, however, is overkill, because I don't actually need to wipe out the workspace because I already know it to be empty.

      It would be very nice to have an option to "Flush and sync only" that runs only flush -f and sync -pf. Thereby, the inaccurate havelist could be bypassed entirely without the extra overhead of reset, sync #none, and rm.

          [JENKINS-72761] Provide better populate options for cloud nodes

          Latonya added a comment - - edited

          Hello,

          I found that there is a feature request for a “Flush and sync only” option in the Perforce Jenkins plugin. This option would run p4 flush -f and p4 sync -pf to bypass the havelist and sync only the files that are needed for the build. However, this feature has not been implemented yet, and there is no ETA for when it will be available.

          As a workaround, you could try to use the Graph force clean and sync option, which is similar to the Forced clean and sync option, but uses the p4 clean -f https://www.beballplayers.com/ command instead of p4 sync #none and rm. This command is faster and more efficient, as it only removes the files that are not in sync with the depot. You can also use the p4 clean -f -e option to exclude files that are opened for edit.

          I hope this helps you improve your build performance and avoid errors. 

          Best regards,
          Latonya

          Latonya added a comment - - edited Hello, I found that there is a feature request for a “Flush and sync only” option in the Perforce Jenkins plugin. This option would run p4 flush -f and p4 sync -pf to bypass the havelist and sync only the files that are needed for the build. However, this feature has not been implemented yet, and there is no ETA for when it will be available. As a workaround, you could try to use the Graph force clean and sync option, which is similar to the Forced clean and sync option, but uses the p4 clean -f https://www.beballplayers.com/ command instead of p4 sync #none and rm. This command is faster and more efficient, as it only removes the files that are not in sync with the depot. You can also use the p4 clean -f -e option to exclude files that are opened for edit. I hope this helps you improve your build performance and avoid errors.  Best regards, Latonya

          Andrew Ray added a comment -

          latonya86dodson "It only removes files..." There are no files to remove. That's why I'm not happy with the extra steps. Because we're launching a new container for every build, we know with certainty that the workspace directory will be empty. p4 sync -k #none is basically the same as p4 flush -f, so it would work just as well.

          Andrew Ray added a comment - latonya86dodson "It only removes files..." There are no files to remove. That's why I'm not happy with the extra steps. Because we're launching a new container for every build, we know with certainty that the workspace directory will be empty. p4 sync -k #none is basically the same as p4 flush -f , so it would work just as well.

          Karl Wirth added a comment -

          Hi isc_aray - Thanks for the suggestion. I see two usage cases here.

          (1) Never reuse same ephemeral node twice. Disk never kept.

          If it's a new container are you giving it a unique client workspace name each time? If you are not reusing container names then the suggested workspace name would be unique:

              jenkins-${NODE_NAME}${JOB_NAME}${EXECUTOR_NUMBER}

          So you would never reuse a workspace so there is no have entries to wipe.

          The problem comes if NODE_NAME is not unique. However if you always use "Sync only" and don't populate the have list then the workspace name can be reused as needed:

             https://www.perforce.com/manuals/jenkins/Content/P4Jenkins/populate-sync-only.html

          (2) Ephemeral node reused with a non empty disk. The disk may or may not be from the same build job.

          This is the one that is trickier when you have to force clean because it `may` be in an inconsistent state.

           

          However based on your last comment ("There are no files to remove."). I just wanted to confirm that you need a solution for (2).

           

          Karl Wirth added a comment - Hi isc_aray - Thanks for the suggestion. I see two usage cases here. (1) Never reuse same ephemeral node twice. Disk never kept. If it's a new container are you giving it a unique client workspace name each time? If you are not reusing container names then the suggested workspace name would be unique:     jenkins-${NODE_NAME} ${JOB_NAME} ${EXECUTOR_NUMBER} So you would never reuse a workspace so there is no have entries to wipe. The problem comes if NODE_NAME is not unique. However if you always use "Sync only" and don't populate the have list then the workspace name can be reused as needed:     https://www.perforce.com/manuals/jenkins/Content/P4Jenkins/populate-sync-only.html (2) Ephemeral node reused with a non empty disk. The disk may or may not be from the same build job. This is the one that is trickier when you have to force clean because it `may` be in an inconsistent state.   However based on your last comment ("There are no files to remove."). I just wanted to confirm that you need a solution for (2).  

          Andrew Ray added a comment -

          p4karl It's the first one. The problem arises if we have ever used a workspace populate option that does populate the have list, then we can never guarantee that Sync only is safe.

          Andrew Ray added a comment - p4karl It's the first one. The problem arises if we have ever used a workspace populate option that does populate the have list, then we can never guarantee that Sync only is safe.

            Unassigned Unassigned
            isc_aray Andrew Ray
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: