Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17813

Jobs should have built in option to never clean workspace

    • Icon: Improvement Improvement
    • Resolution: Not A Defect
    • Icon: Major Major
    • core
    • None
    • Windows 7 x64

      I am relatively new to Jenkins and was recently working on a new build configuration for my company when I discovered that Jenkins has a very odd feature I have never noticed before: it automatically tries to delete the working folder (ie: workspace) before performing the job operations. At best, this is an annoying feature and at worse it is a feature that can have extreme consequences if the content being deleted is irreplaceable.

      For example, the build configuration I was configuring was essentially an automation task that monitors a folder in SVN for changes and updates the local working folder on the build box with one or more custom script files required by our build infrastructure. Further, because these configuration scripts are global we need to store them in one of several different places including c:\Program Data, c:\users\local_build_user, etc. Given that these folders are system folders, often co-existing with other system critical files, you can imagine my surprise when Jenkins just magically started deleting anything and everything in these folders without my approval.

      So, on the one hand, I am very surprised that Jenkins would automatically delete anything without the sys admin explicitly telling the system to do so. In this vein I would suggest that it would be preferable to remove all such "automatic cleanup logic" from the application.

      On the other hand, I suspect this logic was put in place for some reason, and may be critical the behavior of the application. This being the case there should, at the very least, be an option on the configuration page for each job to turn off such logic in situations like mine.

      So, to summarize my build configuration, I have the following job parameters:

      • Use custom workspace: c:\
      • Source code management: Subversion
        • Repo URL: http:
          path_to_my_repo
        • Local module directory: users\local_builder_profile
        • Checkout strategy: Use 'svn update' as much as possible
      • Build triggers: Poll SCM
        • Schedule: H * * * *

      Then, as soon as I try to run this job for the first time Jenkins attempts to delete my entire user profile folder.

      It is also worth noting that I currently have a trivial configuration of Jenkins on one build PC (no distributed builds, agents, etc.) with minimal changes to the host OS to allow Jenkins to work.

      Also, I am using the latest version of Jenkins (1.512 at the moment).

      Finally, I did come across several other defects in your system related to this cleanup task, most notably JENKINS-3841. In the comment thread there in they mention the hudson.model.WorkspaceCleanupThread.disabled custom property as well. I have tried setting this to true but Jenkins' behavior seems to be unaffected.

          [JENKINS-17813] Jobs should have built in option to never clean workspace

          John Elliot added a comment -

          I have a similar request to this. Jenkins will delete the working copy if it detects any errors in it (workspace locked etc.)

          I'm sure this is ok for small workspaces, but we have multiple GB workspaces so checking out again takes a lot of time. On top of that a full rebuild can take many hours which we don't want to happen unexpectedly.

          I would really like a configuration option to not delete the workspace so I can try and fix the repository manually.

          John Elliot added a comment - I have a similar request to this. Jenkins will delete the working copy if it detects any errors in it (workspace locked etc.) I'm sure this is ok for small workspaces, but we have multiple GB workspaces so checking out again takes a lot of time. On top of that a full rebuild can take many hours which we don't want to happen unexpectedly. I would really like a configuration option to not delete the workspace so I can try and fix the repository manually.

          Daniel Beck added a comment -

          Workspaces, where Jenkins stores temporary build data, are supposed to be managed by Jenkins as much as possible (it's not an 'advanced project option' hidden by default by accident!). Deletion of workspace contents is trivial, and cleanup of leftover state is a common occurrence in any build, ingrained in Jenkins and plugins.

          For your specific use case (which seems very unusual), it'd likely be best to not use a custom workspace, but just cd or xcopy out of that directory to accomplish your specific goals.

          Maybe you're just using the wrong tool for the job: Something like Windows Scheduled Tasks might be better suited for what you're trying to accomplish

          If you disagree with this point of view, consider emailing the jenkinsci-users list asking for advice in your specific situation. Issues in the tracker will reach a significant audience (and this is also the wrong place to discuss this).

          Daniel Beck added a comment - Workspaces, where Jenkins stores temporary build data , are supposed to be managed by Jenkins as much as possible (it's not an 'advanced project option' hidden by default by accident!). Deletion of workspace contents is trivial, and cleanup of leftover state is a common occurrence in any build, ingrained in Jenkins and plugins. For your specific use case (which seems very unusual), it'd likely be best to not use a custom workspace, but just cd or xcopy out of that directory to accomplish your specific goals. Maybe you're just using the wrong tool for the job: Something like Windows Scheduled Tasks might be better suited for what you're trying to accomplish If you disagree with this point of view, consider emailing the jenkinsci-users list asking for advice in your specific situation. Issues in the tracker will reach a significant audience (and this is also the wrong place to discuss this).

          Workspaces, where Jenkins stores temporary build data, are supposed to be managed by Jenkins as much as possible

          Agreed

          it's not an 'advanced project option' hidden by default by accident!

          Can you point me to this "advanced project option"? I have been unable to locate any option that effectively disables this behavior, advanced or not. Perhaps it's too well hidden

          Deletion of workspace contents is trivial

          Possibly for small projects, or projects that are relatively well structured. Large projects, with code written decades ago with varying degrees of "cleanliness" are not so easily managed.

          For your specific use case (which seems very unusual), it'd likely be best to not use a custom workspace, but just cd or xcopy out of that directory to accomplish your specific goals.

          Doing so would require me to do all of my own artifact management, thus defeating the purpose of using Jenkins in the first place, but may be worth considering.

          Maybe you're just using the wrong tool for the job: Something like Windows Scheduled Tasks might be better suited for what you're trying to accomplish

          Interesting suggestion. If I could create a scheduled task that could trigger when changes were made to a source-code repository, I would consider that approach, but so far as I can tell this is not possible. The closest thing would be to have a task scheduled to run every minute, which I'm not sure would work so well on Windows.

          ...consider emailing the jenkinsci-users list asking for advice in your specific situation ... this is also the wrong place to discuss this...

          For future reference, can you give me more details of where this 'jenkinsci-users' list is, or where the 'appropriate' venue for this discussion would be?

          Kevin Phillips added a comment - Workspaces, where Jenkins stores temporary build data, are supposed to be managed by Jenkins as much as possible Agreed it's not an 'advanced project option' hidden by default by accident! Can you point me to this "advanced project option"? I have been unable to locate any option that effectively disables this behavior, advanced or not. Perhaps it's too well hidden Deletion of workspace contents is trivial Possibly for small projects, or projects that are relatively well structured. Large projects, with code written decades ago with varying degrees of "cleanliness" are not so easily managed. For your specific use case (which seems very unusual), it'd likely be best to not use a custom workspace, but just cd or xcopy out of that directory to accomplish your specific goals. Doing so would require me to do all of my own artifact management, thus defeating the purpose of using Jenkins in the first place, but may be worth considering. Maybe you're just using the wrong tool for the job: Something like Windows Scheduled Tasks might be better suited for what you're trying to accomplish Interesting suggestion. If I could create a scheduled task that could trigger when changes were made to a source-code repository, I would consider that approach, but so far as I can tell this is not possible. The closest thing would be to have a task scheduled to run every minute, which I'm not sure would work so well on Windows. ...consider emailing the jenkinsci-users list asking for advice in your specific situation ... this is also the wrong place to discuss this... For future reference, can you give me more details of where this 'jenkinsci-users' list is, or where the 'appropriate' venue for this discussion would be?

          For future reference, we ended up working around our original use case using symbolic links, something like this:

          1. Setup Jenkins job to monitor source files in SVN
          2. Run the job once on the agent of choice
          3. Log into remote machine and make symbolic links to the appropriate files / folders in the appropriate locations (e.g.: mklink "%userprofile%\project.settings" "c:\jenkins\workspaces\job_folder\project.settings" )

          Then the files and folders exist within the Jenkins workspace realm, their sources can be monitored directly in the SCM and kept up to date automatically, without requiring further human intervention. Also deleting and restoring the files leaves the symbolic links largely unaffected - so long as you take care not to run tasks in parallel that may require access to those artifacts.

          Kevin Phillips added a comment - For future reference, we ended up working around our original use case using symbolic links, something like this: Setup Jenkins job to monitor source files in SVN Run the job once on the agent of choice Log into remote machine and make symbolic links to the appropriate files / folders in the appropriate locations (e.g.: mklink "%userprofile%\project.settings" "c:\jenkins\workspaces\job_folder\project.settings" ) Then the files and folders exist within the Jenkins workspace realm, their sources can be monitored directly in the SCM and kept up to date automatically, without requiring further human intervention. Also deleting and restoring the files leaves the symbolic links largely unaffected - so long as you take care not to run tasks in parallel that may require access to those artifacts.

          Also for reference sake, you should be aware that I've found it helpful to use Jenkins as a generic build farm management tool as well as a build automation tool. Including such configuration management operations on the dashboard provides a consistent location and toolset which I've found to be much easier than having several different tools which serve similar purposes all monitored and maintained separately.

          Although it may not have been designed as such, it serves the purpose well - most of the time.

          Kevin Phillips added a comment - Also for reference sake, you should be aware that I've found it helpful to use Jenkins as a generic build farm management tool as well as a build automation tool. Including such configuration management operations on the dashboard provides a consistent location and toolset which I've found to be much easier than having several different tools which serve similar purposes all monitored and maintained separately. Although it may not have been designed as such, it serves the purpose well - most of the time.

          Daniel Beck added a comment -

          Can you point me to this "advanced project option

          The section in a freestyle job's configuration where you can set a custom workspace is called 'Advanced Project Options' and hidden by default. Just meant that while it's included in Jenkins core, it's usually not something you should need to use (and brings you slightly nearer to unknown "here be dragons" territory).

          If I could create a scheduled task that could trigger when changes were made to a source-code repository, I would consider that approach, but so far as I can tell this is not possible

          You could use Jenkins just for triggering your batch script at the cost of an additional copy of the source code (unless you can configure it to do a sparse/non-recursive/... checkout, in which case it's cheaper). Do everything else as you would from a standalone script.

          Also for reference sake, you should be aware that I've found it helpful to use Jenkins as a generic build farm management tool as well as a build automation tool. Including such configuration management operations on the dashboard provides a consistent location and toolset which I've found to be much easier than having several different tools which serve similar purposes all monitored and maintained separately.

          Yeah, we've also gone into use-Jenkins-for-everything-happening-on-a-server mode. But the limitations (e.g. limited interactivity) clearly show.

          For future reference, can you give me more details of where this 'jenkinsci-users' list is, or where the 'appropriate' venue for this discussion would be?

          You can find the mailing lists here: http://jenkins-ci.org/content/mailing-lists (via the 'Connect' menu item).


          If you use the "workspace" mostly to provide a remote read-only file browser, there might be an easy solution to provide that for arbitrary directories. Let me know.

          Daniel Beck added a comment - Can you point me to this "advanced project option The section in a freestyle job's configuration where you can set a custom workspace is called 'Advanced Project Options' and hidden by default. Just meant that while it's included in Jenkins core, it's usually not something you should need to use (and brings you slightly nearer to unknown "here be dragons" territory). If I could create a scheduled task that could trigger when changes were made to a source-code repository, I would consider that approach, but so far as I can tell this is not possible You could use Jenkins just for triggering your batch script at the cost of an additional copy of the source code (unless you can configure it to do a sparse/non-recursive/... checkout, in which case it's cheaper). Do everything else as you would from a standalone script. Also for reference sake, you should be aware that I've found it helpful to use Jenkins as a generic build farm management tool as well as a build automation tool. Including such configuration management operations on the dashboard provides a consistent location and toolset which I've found to be much easier than having several different tools which serve similar purposes all monitored and maintained separately. Yeah, we've also gone into use-Jenkins-for-everything-happening-on-a-server mode. But the limitations (e.g. limited interactivity) clearly show. For future reference, can you give me more details of where this 'jenkinsci-users' list is, or where the 'appropriate' venue for this discussion would be? You can find the mailing lists here: http://jenkins-ci.org/content/mailing-lists (via the 'Connect' menu item). If you use the "workspace" mostly to provide a remote read-only file browser, there might be an easy solution to provide that for arbitrary directories. Let me know.

          Thanks for the info. That helps a lot.

          Given our current solution of using symbolic file system links, we get the best of both worlds. For my purposes I consider this a workable solution to this issue. As you said, Jenkins isn't really intended for this kind of purpose and has some obvious limitations which prevent you from stepping outside the "build automation" box.

          Thanks again.

          Kevin Phillips added a comment - Thanks for the info. That helps a lot. Given our current solution of using symbolic file system links, we get the best of both worlds. For my purposes I consider this a workable solution to this issue. As you said, Jenkins isn't really intended for this kind of purpose and has some obvious limitations which prevent you from stepping outside the "build automation" box. Thanks again.

            Unassigned Unassigned
            leedega Kevin Phillips
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: