Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12783

Delete workspace before build starts ignores Exlude patterns (for directories)

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Critical
    • Resolution: Unresolved
    • ws-cleanup-plugin

    Description

      I need to exclude some directories from being deleted on workspace cleanup. It always deleted the entire workspace, pattern is always ignored.
      I used several combinations: */DIRNAME/., DIRNAME/., DIRNAME, *DIRNAME with or without checking "Apply pattern also on directories"

      Attachments

        Issue Links

          Activity

            vjuranek vjuranek added a comment -

            Hi,
            I cannot reproduce it (to be more specific, DIRNAME/** keeps DIRNAME and all files in it, DIRNAME keeps only this directory, no matter if "apply to directories" is switched on or off). Do you still have this issue?

            vjuranek vjuranek added a comment - Hi, I cannot reproduce it (to be more specific, DIRNAME/** keeps DIRNAME and all files in it, DIRNAME keeps only this directory, no matter if "apply to directories" is switched on or off). Do you still have this issue?
            rrrrando Rando Õispuu added a comment -

            When "Apply pattern also on directories" everything gets deleted regardless of the Exclude pattern. I use **/DIR/DIR/ it works when "Apply pattern also on directories" is not checked.

            rrrrando Rando Õispuu added a comment - When "Apply pattern also on directories" everything gets deleted regardless of the Exclude pattern. I use **/DIR/DIR/ it works when "Apply pattern also on directories" is not checked.

            I'm having the same issue. I have a workspace with "foo" and "bar" directories. I check the checkbox for "Delete workspace before build starts", and then underneath I add a directory and specify "Exclude" and then "foo". I also check the checkbox for "Apply patterns on directories". Then when I run my build, it cleans out BOTH the "foo" and "bar" directories. I would expect that it would delete "bar", but not "foo". But no matter what I specify for an "exclude", it always gets ignored.

            jmartenstein Justin Martenstein added a comment - I'm having the same issue. I have a workspace with "foo" and "bar" directories. I check the checkbox for "Delete workspace before build starts", and then underneath I add a directory and specify "Exclude" and then "foo". I also check the checkbox for "Apply patterns on directories". Then when I run my build, it cleans out BOTH the "foo" and "bar" directories. I would expect that it would delete "bar", but not "foo". But no matter what I specify for an "exclude", it always gets ignored.
            pacey James Pace added a comment -

            I am also having issues with the exclude pattern. I do not want to delete a directory called "node_modules", or any of its sub directories or files, in the root of the workspace.

            I have added an exclude pattern "*/node_modules//" but it still gets deleted. It doesn't seem to matter if the delete directories checkbox is selected or not.

            pacey James Pace added a comment - I am also having issues with the exclude pattern. I do not want to delete a directory called "node_modules", or any of its sub directories or files, in the root of the workspace. I have added an exclude pattern "* /node_modules/ / " but it still gets deleted. It doesn't seem to matter if the delete directories checkbox is selected or not.
            dackerman David Ackerman added a comment - - edited

            I had the same issue, and I think I figured out what was going wrong (after cloning the repo and adding enough logging statements).

            Excluding (not-deleting) everything under a toplevel directory works fine, i.e "toplevel/**" excludes everything under the folder "$WORKSPACE/toplevel". However, excluding something in a subdirectory does not work - i.e. "toplevel/asubdir/**" ends up deleting everything. This happens when you have "apply pattern also on directories" turned on. The reason is that DirectoryScanner will find directories to delete that might be parents of files that you've excluded, but the plugin deletes those folders recursively (i.e. rm -rf). The way to fix this is to add "toplevel" explicitly as an exclude to prevent it from deleting the parent directory as well. The reason I have the directories checkbox turned on is because I don't want the ".git" directory (or any extra directories) lying around and confusing jenkins when it builds the next time.

            My project looks something like this (in a multi-module maven project, where node_modules/ resides in several subdirectories).

            * folder-1
              - node_modules
            * folder-2
              - node_modules
            * folder-3
            * folder-4
              - node_modules
            

            My Configuration looks like this:

            EXCLUDE  **/node_modules/**
            EXCLUDE folder-1
            EXCLUDE folder-2
            EXCLUDE folder-4
            

            And that seems to work as expected. It seems like the plugin needs to be more careful about the directories it deletes recursively - it needs to avoid deleting a parent that would delete excluded files (or maybe just do "rm" instead of "rm -r" so it won't delete the directory if it has otherwise excluded files in it). I would offer to create a patch, but my Intellij doesn't seem to be resolving dependencies in the project properly (although maven commands work fine), and I am not sure what others would consider the ideal behavior. At the very least, I think more documentation in the plugin help and/or plugin page would save people a lot of time.

            dackerman David Ackerman added a comment - - edited I had the same issue, and I think I figured out what was going wrong (after cloning the repo and adding enough logging statements). Excluding (not-deleting) everything under a toplevel directory works fine, i.e "toplevel/**" excludes everything under the folder "$WORKSPACE/toplevel". However, excluding something in a subdirectory does not work - i.e. "toplevel/asubdir/**" ends up deleting everything. This happens when you have "apply pattern also on directories" turned on. The reason is that DirectoryScanner will find directories to delete that might be parents of files that you've excluded, but the plugin deletes those folders recursively (i.e. rm -rf). The way to fix this is to add "toplevel" explicitly as an exclude to prevent it from deleting the parent directory as well. The reason I have the directories checkbox turned on is because I don't want the ".git" directory (or any extra directories) lying around and confusing jenkins when it builds the next time. My project looks something like this (in a multi-module maven project, where node_modules/ resides in several subdirectories). * folder-1 - node_modules * folder-2 - node_modules * folder-3 * folder-4 - node_modules My Configuration looks like this: EXCLUDE **/node_modules/** EXCLUDE folder-1 EXCLUDE folder-2 EXCLUDE folder-4 And that seems to work as expected. It seems like the plugin needs to be more careful about the directories it deletes recursively - it needs to avoid deleting a parent that would delete excluded files (or maybe just do "rm" instead of "rm -r" so it won't delete the directory if it has otherwise excluded files in it). I would offer to create a patch, but my Intellij doesn't seem to be resolving dependencies in the project properly (although maven commands work fine), and I am not sure what others would consider the ideal behavior. At the very least, I think more documentation in the plugin help and/or plugin page would save people a lot of time.
            lsanders Larry Sanders added a comment - - edited

            I spent quite a bit of time on this very issue (ironically, also trying to avoid the node penalty).

            Of course leaving empty directories laying around is problematic, specifically for .repository when using a private maven repository for your build.

            lsanders Larry Sanders added a comment - - edited I spent quite a bit of time on this very issue (ironically, also trying to avoid the node penalty). Of course leaving empty directories laying around is problematic, specifically for .repository when using a private maven repository for your build.
            spikeynick Nick Romanyshyn added a comment - - edited

            I don't know if this applies to the OP, but I had a very similar issue. I specified patterns to exclude, but it would completely delete everything in the workspace. If I instead specified a pattern to include that would work as expected. In my case the exclude actually was working correctly. I verified this by adding a custom delete command which was just a simple python script that appended the name of the file to be deleted to a text file (so it didn't actually do any deleting). It turned out that I wasn't excluding the .git directory, so that was getting deleted. You can't clone into a non empty directory, so Jenkins had to remove the workspace completely in order to do a new clone. I haven't tried with .svn, but I would expect similar results. One other thing to note is that if that you don't specify an include pattern then the plugin automatically uses the pattern **/* So if you have included anything that doesn't match the .git directory that will also spare it from deletion.

            tl;dr Make sure you also exclude your .git directory

            spikeynick Nick Romanyshyn added a comment - - edited I don't know if this applies to the OP, but I had a very similar issue. I specified patterns to exclude, but it would completely delete everything in the workspace. If I instead specified a pattern to include that would work as expected. In my case the exclude actually was working correctly. I verified this by adding a custom delete command which was just a simple python script that appended the name of the file to be deleted to a text file (so it didn't actually do any deleting). It turned out that I wasn't excluding the .git directory, so that was getting deleted. You can't clone into a non empty directory, so Jenkins had to remove the workspace completely in order to do a new clone. I haven't tried with .svn, but I would expect similar results. One other thing to note is that if that you don't specify an include pattern then the plugin automatically uses the pattern ** / * So if you have included anything that doesn't match the .git directory that will also spare it from deletion. tl;dr Make sure you also exclude your .git directory
            joyeux pierre Salaun added a comment -

            Hi,
            I'm having the same issue, Include work fine but not Exclude.
            I have a maven project with custom workspace and a private maven repository local to workspace.
            I want to keep the ".repository" directory but it is delete by the "Delete workspace before build starts" function.
            I have add a Exclude ".respository/**" but it isn't work.
            Can you help me?

            joyeux pierre Salaun added a comment - Hi, I'm having the same issue, Include work fine but not Exclude. I have a maven project with custom workspace and a private maven repository local to workspace. I want to keep the ".repository" directory but it is delete by the "Delete workspace before build starts" function. I have add a Exclude ".respository/**" but it isn't work. Can you help me?
            grim Grim Reaper added a comment - - edited

            Jenkins Versions: 2.60.3 & 2.73.3 (WAR deployments on WildFly)
            Plugin Version: 0.34
            Platform: Linux
            Plugins:
            Git client plugin 2.6.0
            Git plugin 3.6.4

            Details:
            No issues when the SCM is CVS. However we are migrating from CVS to GIT.
            When I configure a freestyle job to use GIT as the SCM and with "Delete workspace before build starts" checked it ignores any "Advanced" options I use (I'm trying to exclude ' *.build.* ' and a 'reports/**' directory) and just deletes everything in the entire workspace.

            In troubleshooting I also testing the Post Build Action > "Delete workspace when build is done". This half works. I can tell it to "Exclude" specific files, but as soon as the "Apply pattern also on directories" is checked it goes back to the above behavior of deleting everything regardless of excludes.

            Really hoping someone can have a look at this and find the root cause and a fix.

            Thanks in advance!

            Edit:
            I forgot to add that the GIT install is on a windows machine and the repo is located through a UNC share. Not that I think this has any bearing as the repo pull/clone works fine. Just trying to provide as much info as possible.

            grim Grim Reaper added a comment - - edited Jenkins Versions: 2.60.3 & 2.73.3 (WAR deployments on WildFly) Plugin Version: 0.34 Platform: Linux Plugins: Git client plugin 2.6.0 Git plugin 3.6.4 Details: No issues when the SCM is CVS. However we are migrating from CVS to GIT. When I configure a freestyle job to use GIT as the SCM and with "Delete workspace before build starts" checked it ignores any "Advanced" options I use (I'm trying to exclude ' * .build. * ' and a 'reports/**' directory) and just deletes everything in the entire workspace. In troubleshooting I also testing the Post Build Action > "Delete workspace when build is done". This half works. I can tell it to "Exclude" specific files, but as soon as the "Apply pattern also on directories" is checked it goes back to the above behavior of deleting everything regardless of excludes. Really hoping someone can have a look at this and find the root cause and a fix. Thanks in advance! Edit: I forgot to add that the GIT install is on a windows machine and the repo is located through a UNC share. Not that I think this has any bearing as the repo pull/clone works fine. Just trying to provide as much info as possible.
            grim Grim Reaper added a comment -

            Assigning to you as the plugins page indicates you are the ws-cleanup maintainer.
            Also for visibility.

            grim Grim Reaper added a comment - Assigning to you as the plugins page indicates you are the ws-cleanup maintainer. Also for visibility.
            grim Grim Reaper added a comment -

            For those looking for a work around, I've use the following plugin:
            https://plugins.jenkins.io/preSCMbuildstep

            With the following Shell script to do the same thing as WS-cleanup.

            echo "Deleting..."
            ls ${WORKSPACE}/ | egrep -v '*.build.*|suppressions.xml|reports' 
            ls ${WORKSPACE}/ | egrep -v '*.build.*|suppressions.xml|reports' | xargs rm -rf
            

             

            The above deletes everything except the items in the "egrep" command (excludes).

            grim Grim Reaper added a comment - For those looking for a work around, I've use the following plugin: https://plugins.jenkins.io/preSCMbuildstep With the following Shell script to do the same thing as WS-cleanup. echo "Deleting..." ls ${WORKSPACE}/ | egrep -v '*.build.*|suppressions.xml|reports' ls ${WORKSPACE}/ | egrep -v '*.build.*|suppressions.xml|reports' | xargs rm -rf   The above deletes everything except the items in the "egrep" command (excludes).
            tpbradt Tim Bradt added a comment -

            This bug is unfortunate.  We're trying to preserve our IntelliJ project files so we don't have to restore project settings every time we build.  This should really be addressed.

            +1 vote

            tpbradt Tim Bradt added a comment - This bug is unfortunate.  We're trying to preserve our IntelliJ project files so we don't have to restore project settings every time we build.  This should really be addressed. +1 vote
            grim Grim Reaper added a comment - - edited

            Updated my test server to Jenkins 2.89.4 and we are now using "Gitea" as our backend SCM.

            Still the same issue.

            Also update on the work around. Found out it wasn't working as intended. So now I shuffle files around to make it work. This is all on Linux.

            Steps to configure the job:

            1. Under "Source Code management" (Git), select "Wipe out repo & force clone" under "Additional behaviours".
            2. Under "Build Environment", note you still need to have the preSCMBuildStep plugin and check "Run buildstep before SCM runs" and add "Execute shell", then use the following to "Exclude files" (You may need to modify for your needs):
              # Move build files to temp location before workspace wipe
              ls -Al "${WORKSPACE}"
              [ ! -d "/tmp/${BUILD_TAG}/" ] && mkdir -p "/tmp/${BUILD_TAG}/"
              mv -v "${WORKSPACE}"/*.build.* "${WORKSPACE}"/suppressions.xml "/tmp/${BUILD_TAG}/"
              ls -Al "/tmp/${BUILD_TAG}/"
              
            1. Under the "Build" section of your job. Add "Execute shell" and move it to the top. In the textbox add the following script commands to move your files back into the workspace:
              # Move build files back to workspace before build runs 
              mv -v "/tmp/${BUILD_TAG}"/* "${WORKSPACE}" 
              ls -Al "${WORKSPACE}" 
              rm -rfv "/tmp/${BUILD_TAG}/"
              

            Off to try the latest and greatest Jenkins (Hopefully it can still build our legacy apps >.<).

             

            grim Grim Reaper added a comment - - edited Updated my test server to Jenkins 2.89.4 and we are now using "Gitea" as our backend SCM. Still the same issue. Also update on the work around. Found out it wasn't working as intended. So now I shuffle files around to make it work. This is all on Linux. Steps to configure the job: Under "Source Code management" (Git), select "Wipe out repo & force clone" under "Additional behaviours". Under "Build Environment", note you still need to have the preSCMBuildStep plugin and check "Run buildstep before SCM runs" and add "Execute shell", then use the following to "Exclude files" (You may need to modify for your needs): # Move build files to temp location before workspace wipe ls -Al "${WORKSPACE}" [ ! -d "/tmp/${BUILD_TAG}/" ] && mkdir -p "/tmp/${BUILD_TAG}/" mv -v "${WORKSPACE}" /*.build.* "${WORKSPACE}" /suppressions.xml "/tmp/${BUILD_TAG}/" ls -Al "/tmp/${BUILD_TAG}/" Under the "Build" section of your job. Add "Execute shell" and move it to the top. In the textbox add the following script commands to move your files back into the workspace: # Move build files back to workspace before build runs mv -v "/tmp/${BUILD_TAG}" /* "${WORKSPACE}" ls -Al "${WORKSPACE}" rm -rfv "/tmp/${BUILD_TAG}/" Off to try the latest and greatest Jenkins (Hopefully it can still build our legacy apps >.<).  

            People

              olivergondza Oliver Gondža
              bavoiculescu Voiculescu Bogdan Alexandru
              Votes:
              8 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated: