• Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Major Major
    • p4-plugin
    • None
    • Jenkins 1.554.2, perforce plugin 1.3.27, RHEL 5.5

      It would be much more efficient to have an option for Jenkins to delete the workspace at the end of the build job. While the Jenkins perforce plugin is really good about creating and managing workspaces, currently it ends up creating lots of clients/workspaces without cleaning any up. Since the perforce server has to track a lot of metadata on each client workspace (records on each of the files that are checked out), it can severely limit the scalability of the Jenkins/perforce plugin using workspace management.

      In a production atmosphere with hundreds to thousands of jobs that can run on different nodes, the perforce server can even run out of RAM loading all the client records into memory. This often leads to the perforce admin hunting down members of the devops team.

          [JENKINS-24186] Add option for deleting perforce clients

          Rob Petti added a comment -

          Unfortunately, clients need to be preserved in order for subsequent syncs to be more efficient. Deleting them at the end of the job would cause every run to sync everything from scratch, which is a massive waste of time and bandwidth. It's far more efficient to just keep them around.

          Rob Petti added a comment - Unfortunately, clients need to be preserved in order for subsequent syncs to be more efficient. Deleting them at the end of the job would cause every run to sync everything from scratch, which is a massive waste of time and bandwidth. It's far more efficient to just keep them around.

          Caleb Mayeux added a comment -

          True, unless you're using the force sync option and wiping your workspace at the end of each build anyway. Maybe it could be a checkbox under the force sync option?

          I have no idea how typical this workflow is, but ours relies on us wiping the workspace and force syncing each time. This is done because for a number of branches across a large pool of nodes, there isn't near enough storage space on each build slave to keep a sync'ed workspace on each (Dealing with a few hundred branches of 5-10GB workspace size in each, and a pool of a couple dozen nodes = 100s * (5-10)GB * ~ 24 = many terabytes of expensive SAN). Typical space/bandwidth tradeoff, with a local perforce proxy keeping the traffic across the WAN to a minimum. Additionally, it keeps builds from being contaminated by old artifacts.

          Also this proposed enhancement would probably be useful for jobs that are run only rarely.

          Caleb Mayeux added a comment - True, unless you're using the force sync option and wiping your workspace at the end of each build anyway. Maybe it could be a checkbox under the force sync option? I have no idea how typical this workflow is, but ours relies on us wiping the workspace and force syncing each time. This is done because for a number of branches across a large pool of nodes, there isn't near enough storage space on each build slave to keep a sync'ed workspace on each (Dealing with a few hundred branches of 5-10GB workspace size in each, and a pool of a couple dozen nodes = 100s * (5-10)GB * ~ 24 = many terabytes of expensive SAN). Typical space/bandwidth tradeoff, with a local perforce proxy keeping the traffic across the WAN to a minimum. Additionally, it keeps builds from being contaminated by old artifacts. Also this proposed enhancement would probably be useful for jobs that are run only rarely.

          Rob Petti added a comment -

          A few things:

          • You can very easily just run 'p4 client -df $P4CLIENT' as the first step in your job.
          • The plugin is already overloaded with options. We should only add new ones if they are absolutely necessary.
          • Someone else (possibly you) will need to implement it if we decide to go ahead with adding it.

          Rob Petti added a comment - A few things: You can very easily just run 'p4 client -df $P4CLIENT' as the first step in your job. The plugin is already overloaded with options. We should only add new ones if they are absolutely necessary. Someone else (possibly you) will need to implement it if we decide to go ahead with adding it.

          Caleb Mayeux added a comment -

          1) True, there is an very easy workaround adding the command as the last step or post build task. It'd just be cleaner inside the plugin (arguably the opposite if you're not using it and concerned about bloat).
          2) Yes, it has become an incredibly flexible plugin.
          3) Took a look at the code (guess I should've done that part first). I was originally thinking it'd be a checkbox under force sync, which would be pretty attractive. Looking at the SCM class though, the only obvious way I can see to do it would be to add it as a separate post-build step - similar to the way the labeling works (I'm assuming that's why the labeling portion isn't grouped with the rest of the plugin). A post-build step just for that option would be almost as annoying as having to add a generic step with the command line command, so I'm somewhat inclined to side with your reluctance on this feature.

          Well, thanks for the time and input! Perhaps we can leave this ticket open for a month or so and see if anyone else wants to weigh in, then close it out?

          Caleb Mayeux added a comment - 1) True, there is an very easy workaround adding the command as the last step or post build task. It'd just be cleaner inside the plugin (arguably the opposite if you're not using it and concerned about bloat). 2) Yes, it has become an incredibly flexible plugin. 3) Took a look at the code (guess I should've done that part first). I was originally thinking it'd be a checkbox under force sync, which would be pretty attractive. Looking at the SCM class though, the only obvious way I can see to do it would be to add it as a separate post-build step - similar to the way the labeling works (I'm assuming that's why the labeling portion isn't grouped with the rest of the plugin). A post-build step just for that option would be almost as annoying as having to add a generic step with the command line command, so I'm somewhat inclined to side with your reluctance on this feature. Well, thanks for the time and input! Perhaps we can leave this ticket open for a month or so and see if anyone else wants to weigh in, then close it out?

          Omar Ahmad added a comment -

          I have ran into the same issue. We also have many jobs and jenkins slave nodes with limited space, and so deleting codebases at the end of jobs is necessary. Also for release jobs (of which we have many), I prefer to run a brand new force sync to retrieve the code to be built.

          I am using the workspace-cleanup-plugin to delete job workspaces, however this is part of the solution, as it deletes the jenkins workspace, but does not safely cleanup the perforce workspace. It would be nice, if there was a post build action that could be activated to delete or at the least safely cleanup (i.e. revert any checked out files, delete any pending changelists, remove all workspace files) as a post build action that could be run before the workspace-cleanup-plugin deletes the jenkins workspace.

          Omar Ahmad added a comment - I have ran into the same issue. We also have many jobs and jenkins slave nodes with limited space, and so deleting codebases at the end of jobs is necessary. Also for release jobs (of which we have many), I prefer to run a brand new force sync to retrieve the code to be built. I am using the workspace-cleanup-plugin to delete job workspaces, however this is part of the solution, as it deletes the jenkins workspace, but does not safely cleanup the perforce workspace. It would be nice, if there was a post build action that could be activated to delete or at the least safely cleanup (i.e. revert any checked out files, delete any pending changelists, remove all workspace files) as a post build action that could be run before the workspace-cleanup-plugin deletes the jenkins workspace.

          Rob Petti added a comment -

          Omar, deleting the workspace through Jenkins automatically flushes the client back to #0. It is not necessary to also delete the client itself, and in many cases it is not desirable to do so.

          I am not adding this option, partly because this plugin is no longer maintained, but mostly because it's not a useful option given the plethora of alternatives that have been listed. If someone else wants to write a post-build task, or extend the workspace clean functionality to optionally delete the client spec, they are more than welcome to.

          Rob Petti added a comment - Omar, deleting the workspace through Jenkins automatically flushes the client back to #0. It is not necessary to also delete the client itself, and in many cases it is not desirable to do so. I am not adding this option, partly because this plugin is no longer maintained, but mostly because it's not a useful option given the plethora of alternatives that have been listed. If someone else wants to write a post-build task, or extend the workspace clean functionality to optionally delete the client spec, they are more than welcome to.

          Paul Allen added a comment -

          Hi deepu_s, not sure why you have assigned this from the old perforce-plugin to the new p4-plugin?

          The p4-plugin can delete a client workspace if the global option is enabled and the Job uses the cleanup step or 'cleanup' dsl.

          Paul Allen added a comment - Hi deepu_s , not sure why you have assigned this from the old perforce-plugin to the new p4-plugin? The p4-plugin can delete a client workspace if the global option is enabled and the Job uses the cleanup step or 'cleanup' dsl.

            Unassigned Unassigned
            calebmayeux Caleb Mayeux
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: