Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12783

Delete workspace before build starts ignores Exlude patterns (for directories)

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • ws-cleanup-plugin

      I need to exclude some directories from being deleted on workspace cleanup. It always deleted the entire workspace, pattern is always ignored.
      I used several combinations: */DIRNAME/., DIRNAME/., DIRNAME, *DIRNAME with or without checking "Apply pattern also on directories"

          [JENKINS-12783] Delete workspace before build starts ignores Exlude patterns (for directories)

          James Pace added a comment -

          I am also having issues with the exclude pattern. I do not want to delete a directory called "node_modules", or any of its sub directories or files, in the root of the workspace.

          I have added an exclude pattern "*/node_modules//" but it still gets deleted. It doesn't seem to matter if the delete directories checkbox is selected or not.

          James Pace added a comment - I am also having issues with the exclude pattern. I do not want to delete a directory called "node_modules", or any of its sub directories or files, in the root of the workspace. I have added an exclude pattern "* /node_modules/ / " but it still gets deleted. It doesn't seem to matter if the delete directories checkbox is selected or not.

          David Ackerman added a comment - - edited

          I had the same issue, and I think I figured out what was going wrong (after cloning the repo and adding enough logging statements).

          Excluding (not-deleting) everything under a toplevel directory works fine, i.e "toplevel/**" excludes everything under the folder "$WORKSPACE/toplevel". However, excluding something in a subdirectory does not work - i.e. "toplevel/asubdir/**" ends up deleting everything. This happens when you have "apply pattern also on directories" turned on. The reason is that DirectoryScanner will find directories to delete that might be parents of files that you've excluded, but the plugin deletes those folders recursively (i.e. rm -rf). The way to fix this is to add "toplevel" explicitly as an exclude to prevent it from deleting the parent directory as well. The reason I have the directories checkbox turned on is because I don't want the ".git" directory (or any extra directories) lying around and confusing jenkins when it builds the next time.

          My project looks something like this (in a multi-module maven project, where node_modules/ resides in several subdirectories).

          * folder-1
            - node_modules
          * folder-2
            - node_modules
          * folder-3
          * folder-4
            - node_modules
          

          My Configuration looks like this:

          EXCLUDE  **/node_modules/**
          EXCLUDE folder-1
          EXCLUDE folder-2
          EXCLUDE folder-4
          

          And that seems to work as expected. It seems like the plugin needs to be more careful about the directories it deletes recursively - it needs to avoid deleting a parent that would delete excluded files (or maybe just do "rm" instead of "rm -r" so it won't delete the directory if it has otherwise excluded files in it). I would offer to create a patch, but my Intellij doesn't seem to be resolving dependencies in the project properly (although maven commands work fine), and I am not sure what others would consider the ideal behavior. At the very least, I think more documentation in the plugin help and/or plugin page would save people a lot of time.

          David Ackerman added a comment - - edited I had the same issue, and I think I figured out what was going wrong (after cloning the repo and adding enough logging statements). Excluding (not-deleting) everything under a toplevel directory works fine, i.e "toplevel/**" excludes everything under the folder "$WORKSPACE/toplevel". However, excluding something in a subdirectory does not work - i.e. "toplevel/asubdir/**" ends up deleting everything. This happens when you have "apply pattern also on directories" turned on. The reason is that DirectoryScanner will find directories to delete that might be parents of files that you've excluded, but the plugin deletes those folders recursively (i.e. rm -rf). The way to fix this is to add "toplevel" explicitly as an exclude to prevent it from deleting the parent directory as well. The reason I have the directories checkbox turned on is because I don't want the ".git" directory (or any extra directories) lying around and confusing jenkins when it builds the next time. My project looks something like this (in a multi-module maven project, where node_modules/ resides in several subdirectories). * folder-1 - node_modules * folder-2 - node_modules * folder-3 * folder-4 - node_modules My Configuration looks like this: EXCLUDE **/node_modules/** EXCLUDE folder-1 EXCLUDE folder-2 EXCLUDE folder-4 And that seems to work as expected. It seems like the plugin needs to be more careful about the directories it deletes recursively - it needs to avoid deleting a parent that would delete excluded files (or maybe just do "rm" instead of "rm -r" so it won't delete the directory if it has otherwise excluded files in it). I would offer to create a patch, but my Intellij doesn't seem to be resolving dependencies in the project properly (although maven commands work fine), and I am not sure what others would consider the ideal behavior. At the very least, I think more documentation in the plugin help and/or plugin page would save people a lot of time.

          Larry Sanders added a comment - - edited

          I spent quite a bit of time on this very issue (ironically, also trying to avoid the node penalty).

          Of course leaving empty directories laying around is problematic, specifically for .repository when using a private maven repository for your build.

          Larry Sanders added a comment - - edited I spent quite a bit of time on this very issue (ironically, also trying to avoid the node penalty). Of course leaving empty directories laying around is problematic, specifically for .repository when using a private maven repository for your build.

          Nick Romanyshyn added a comment - - edited

          I don't know if this applies to the OP, but I had a very similar issue. I specified patterns to exclude, but it would completely delete everything in the workspace. If I instead specified a pattern to include that would work as expected. In my case the exclude actually was working correctly. I verified this by adding a custom delete command which was just a simple python script that appended the name of the file to be deleted to a text file (so it didn't actually do any deleting). It turned out that I wasn't excluding the .git directory, so that was getting deleted. You can't clone into a non empty directory, so Jenkins had to remove the workspace completely in order to do a new clone. I haven't tried with .svn, but I would expect similar results. One other thing to note is that if that you don't specify an include pattern then the plugin automatically uses the pattern **/* So if you have included anything that doesn't match the .git directory that will also spare it from deletion.

          tl;dr Make sure you also exclude your .git directory

          Nick Romanyshyn added a comment - - edited I don't know if this applies to the OP, but I had a very similar issue. I specified patterns to exclude, but it would completely delete everything in the workspace. If I instead specified a pattern to include that would work as expected. In my case the exclude actually was working correctly. I verified this by adding a custom delete command which was just a simple python script that appended the name of the file to be deleted to a text file (so it didn't actually do any deleting). It turned out that I wasn't excluding the .git directory, so that was getting deleted. You can't clone into a non empty directory, so Jenkins had to remove the workspace completely in order to do a new clone. I haven't tried with .svn, but I would expect similar results. One other thing to note is that if that you don't specify an include pattern then the plugin automatically uses the pattern ** / * So if you have included anything that doesn't match the .git directory that will also spare it from deletion. tl;dr Make sure you also exclude your .git directory

          pierre Salaun added a comment -

          Hi,
          I'm having the same issue, Include work fine but not Exclude.
          I have a maven project with custom workspace and a private maven repository local to workspace.
          I want to keep the ".repository" directory but it is delete by the "Delete workspace before build starts" function.
          I have add a Exclude ".respository/**" but it isn't work.
          Can you help me?

          pierre Salaun added a comment - Hi, I'm having the same issue, Include work fine but not Exclude. I have a maven project with custom workspace and a private maven repository local to workspace. I want to keep the ".repository" directory but it is delete by the "Delete workspace before build starts" function. I have add a Exclude ".respository/**" but it isn't work. Can you help me?

          Grim Reaper added a comment - - edited

          Jenkins Versions: 2.60.3 & 2.73.3 (WAR deployments on WildFly)
          Plugin Version: 0.34
          Platform: Linux
          Plugins:
          Git client plugin 2.6.0
          Git plugin 3.6.4

          Details:
          No issues when the SCM is CVS. However we are migrating from CVS to GIT.
          When I configure a freestyle job to use GIT as the SCM and with "Delete workspace before build starts" checked it ignores any "Advanced" options I use (I'm trying to exclude ' *.build.* ' and a 'reports/**' directory) and just deletes everything in the entire workspace.

          In troubleshooting I also testing the Post Build Action > "Delete workspace when build is done". This half works. I can tell it to "Exclude" specific files, but as soon as the "Apply pattern also on directories" is checked it goes back to the above behavior of deleting everything regardless of excludes.

          Really hoping someone can have a look at this and find the root cause and a fix.

          Thanks in advance!

          Edit:
          I forgot to add that the GIT install is on a windows machine and the repo is located through a UNC share. Not that I think this has any bearing as the repo pull/clone works fine. Just trying to provide as much info as possible.

          Grim Reaper added a comment - - edited Jenkins Versions: 2.60.3 & 2.73.3 (WAR deployments on WildFly) Plugin Version: 0.34 Platform: Linux Plugins: Git client plugin 2.6.0 Git plugin 3.6.4 Details: No issues when the SCM is CVS. However we are migrating from CVS to GIT. When I configure a freestyle job to use GIT as the SCM and with "Delete workspace before build starts" checked it ignores any "Advanced" options I use (I'm trying to exclude ' * .build. * ' and a 'reports/**' directory) and just deletes everything in the entire workspace. In troubleshooting I also testing the Post Build Action > "Delete workspace when build is done". This half works. I can tell it to "Exclude" specific files, but as soon as the "Apply pattern also on directories" is checked it goes back to the above behavior of deleting everything regardless of excludes. Really hoping someone can have a look at this and find the root cause and a fix. Thanks in advance! Edit: I forgot to add that the GIT install is on a windows machine and the repo is located through a UNC share. Not that I think this has any bearing as the repo pull/clone works fine. Just trying to provide as much info as possible.

          Grim Reaper added a comment -

          Assigning to you as the plugins page indicates you are the ws-cleanup maintainer.
          Also for visibility.

          Grim Reaper added a comment - Assigning to you as the plugins page indicates you are the ws-cleanup maintainer. Also for visibility.

          Grim Reaper added a comment -

          For those looking for a work around, I've use the following plugin:
          https://plugins.jenkins.io/preSCMbuildstep

          With the following Shell script to do the same thing as WS-cleanup.

          echo "Deleting..."
          ls ${WORKSPACE}/ | egrep -v '*.build.*|suppressions.xml|reports' 
          ls ${WORKSPACE}/ | egrep -v '*.build.*|suppressions.xml|reports' | xargs rm -rf
          

           

          The above deletes everything except the items in the "egrep" command (excludes).

          Grim Reaper added a comment - For those looking for a work around, I've use the following plugin: https://plugins.jenkins.io/preSCMbuildstep With the following Shell script to do the same thing as WS-cleanup. echo "Deleting..." ls ${WORKSPACE}/ | egrep -v '*.build.*|suppressions.xml|reports' ls ${WORKSPACE}/ | egrep -v '*.build.*|suppressions.xml|reports' | xargs rm -rf   The above deletes everything except the items in the "egrep" command (excludes).

          Tim Bradt added a comment -

          This bug is unfortunate.  We're trying to preserve our IntelliJ project files so we don't have to restore project settings every time we build.  This should really be addressed.

          +1 vote

          Tim Bradt added a comment - This bug is unfortunate.  We're trying to preserve our IntelliJ project files so we don't have to restore project settings every time we build.  This should really be addressed. +1 vote

          Grim Reaper added a comment - - edited

          Updated my test server to Jenkins 2.89.4 and we are now using "Gitea" as our backend SCM.

          Still the same issue.

          Also update on the work around. Found out it wasn't working as intended. So now I shuffle files around to make it work. This is all on Linux.

          Steps to configure the job:

          1. Under "Source Code management" (Git), select "Wipe out repo & force clone" under "Additional behaviours".
          2. Under "Build Environment", note you still need to have the preSCMBuildStep plugin and check "Run buildstep before SCM runs" and add "Execute shell", then use the following to "Exclude files" (You may need to modify for your needs):
            # Move build files to temp location before workspace wipe
            ls -Al "${WORKSPACE}"
            [ ! -d "/tmp/${BUILD_TAG}/" ] && mkdir -p "/tmp/${BUILD_TAG}/"
            mv -v "${WORKSPACE}"/*.build.* "${WORKSPACE}"/suppressions.xml "/tmp/${BUILD_TAG}/"
            ls -Al "/tmp/${BUILD_TAG}/"
            
          1. Under the "Build" section of your job. Add "Execute shell" and move it to the top. In the textbox add the following script commands to move your files back into the workspace:
            # Move build files back to workspace before build runs 
            mv -v "/tmp/${BUILD_TAG}"/* "${WORKSPACE}" 
            ls -Al "${WORKSPACE}" 
            rm -rfv "/tmp/${BUILD_TAG}/"
            

          Off to try the latest and greatest Jenkins (Hopefully it can still build our legacy apps >.<).

           

          Grim Reaper added a comment - - edited Updated my test server to Jenkins 2.89.4 and we are now using "Gitea" as our backend SCM. Still the same issue. Also update on the work around. Found out it wasn't working as intended. So now I shuffle files around to make it work. This is all on Linux. Steps to configure the job: Under "Source Code management" (Git), select "Wipe out repo & force clone" under "Additional behaviours". Under "Build Environment", note you still need to have the preSCMBuildStep plugin and check "Run buildstep before SCM runs" and add "Execute shell", then use the following to "Exclude files" (You may need to modify for your needs): # Move build files to temp location before workspace wipe ls -Al "${WORKSPACE}" [ ! -d "/tmp/${BUILD_TAG}/" ] && mkdir -p "/tmp/${BUILD_TAG}/" mv -v "${WORKSPACE}" /*.build.* "${WORKSPACE}" /suppressions.xml "/tmp/${BUILD_TAG}/" ls -Al "/tmp/${BUILD_TAG}/" Under the "Build" section of your job. Add "Execute shell" and move it to the top. In the textbox add the following script commands to move your files back into the workspace: # Move build files back to workspace before build runs mv -v "/tmp/${BUILD_TAG}" /* "${WORKSPACE}" ls -Al "${WORKSPACE}" rm -rfv "/tmp/${BUILD_TAG}/" Off to try the latest and greatest Jenkins (Hopefully it can still build our legacy apps >.<).  

            olivergondza Oliver Gondža
            bavoiculescu Voiculescu Bogdan Alexandru
            Votes:
            8 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated: