Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-10222

Jenkins cleaning out workspace unnecessarily when polling SCM (SVN)

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • subversion-plugin
    • None
    • Windows

      I am using a SVN repository which is being checked out at the beginning of every build in a Jenkins job. It is set to "poll SCM" for updates. However, regardless of whether there were changes or not it is cleaning out the workspace and downloading the whole repository in it's entirety every time which is unnecessary.

      Builds are taking much longer than they should as the SVN poll should only be checking out newly-updated files. Sample log:

      Checking out a fresh workspace because the workspace is not file://PICKLE/Repositories/project
      Cleaning workspace.....

          [JENKINS-10222] Jenkins cleaning out workspace unnecessarily when polling SCM (SVN)

          Jeff Payne added a comment -

          I found one way to reproduce this: if I check out more than one path and I don't specify a target directory, it appears that the subsequent check-outs corrupt the files in the .svn subdirectory. Specifying a different target for each make it work correctly.

          I also googled and found an example where the user used different capitalization for the host name than was stored in the .svn, and had the same effect.

          Hope this helps!

          Jeff Payne added a comment - I found one way to reproduce this: if I check out more than one path and I don't specify a target directory, it appears that the subsequent check-outs corrupt the files in the .svn subdirectory. Specifying a different target for each make it work correctly. I also googled and found an example where the user used different capitalization for the host name than was stored in the .svn, and had the same effect. Hope this helps!

          Damien Finck added a comment - - edited

          Hello,

          I have exactly the same problem.

          In Jenkins, I have about 30 projects.
          For two of my projects, I have this problem for some time.

          The first one has this configuration:

          • An SVN repository on "https://<company>.fr:433/svn/<project>" in the folder "trunk" of my workspace
          • "Use 'svn update' as much as possible"
            And at each build, Jenkins said "Checking out a fresh workspace Because The workspace is not "https://<company>.fr:443/svn/<project>" "

          For the second project that is the problem, it is configured this way:

          • 3 SVN repositories
          • https://<company>.fr/svn/<project1> in the folder "project1"
          • https://<company>.fr:433/svn/<project2> in the "Project2"
          • https://<company>.fr:433/svn/<project3> in the "project3"
          • "Use 'svn update' as much as possible"
            And at each build, Jenkins said "Checking out a fresh workspace Because The workspace is not "https://<company>.fr:443/svn/<project3>" "
            I have a problem only on the third repositorie.

          I tried to remove the first project and recreate it: Same problem.

          I read in a forum user who said: "I had a similar problem .... but It Was Because The name of the SVN server in my path Was in CAPS. I changed it to lower case and it worked. ... "
          So, I changed my path from https://<company>.fr:433/svn/<folder> to https://<company>.fr/svn/<folder> and it works in my two projects .

          It like if repositorie has once time a bug, Jenkins do each other time a "svn checkout" on this repositorie instead of an "svn update".

          Version :
          Jenkins : 1.474
          Jenkins Subversion Plug-in : 1.42
          Subversion version (in Jenkins configuration) : 1.7

          Sincerely,
          Damien

          Damien Finck added a comment - - edited Hello, I have exactly the same problem. In Jenkins, I have about 30 projects. For two of my projects, I have this problem for some time. The first one has this configuration: An SVN repository on "https://<company>.fr:433/svn/<project>" in the folder "trunk" of my workspace "Use 'svn update' as much as possible" And at each build, Jenkins said "Checking out a fresh workspace Because The workspace is not "https://<company>.fr:443/svn/<project>" " For the second project that is the problem, it is configured this way: 3 SVN repositories https://<company>.fr/svn/<project1> in the folder "project1" https://<company>.fr:433/svn/<project2> in the "Project2" https://<company>.fr:433/svn/<project3> in the "project3" "Use 'svn update' as much as possible" And at each build, Jenkins said "Checking out a fresh workspace Because The workspace is not "https://<company>.fr:443/svn/<project3>" " I have a problem only on the third repositorie. I tried to remove the first project and recreate it: Same problem. I read in a forum user who said: "I had a similar problem .... but It Was Because The name of the SVN server in my path Was in CAPS. I changed it to lower case and it worked. ... " So, I changed my path from https://<company>.fr:433/svn/<folder> to https://<company>.fr/svn/<folder> and it works in my two projects . It like if repositorie has once time a bug, Jenkins do each other time a "svn checkout" on this repositorie instead of an "svn update". Version : Jenkins : 1.474 Jenkins Subversion Plug-in : 1.42 Subversion version (in Jenkins configuration) : 1.7 Sincerely, Damien

          Damien Finck added a comment - - edited

          Hello,

          I tried to reproduce the problem, but I'm not happened.
          In all my jobs I remove the port SVN URL (https://<company>.fr:433/svn/<project> to https://<company>.fr/svn/<project>)) and since I have no problem!

          Sincerely,
          Damien

          Damien Finck added a comment - - edited Hello, I tried to reproduce the problem, but I'm not happened. In all my jobs I remove the port SVN URL (https://<company>.fr:433/svn/<project> to https://<company>.fr/svn/<project>)) and since I have no problem! Sincerely, Damien

          I had the same problem with a project with the Repository URL of the following pattern:
          svn://<servername>/applications/xxx//trunk/yyy

          The polling message was:
          Workspace doesn't contain svn://<servername>/applications/xxx//trunk/yyy. Need a new build.

          However, the polling did work successfully (no changes, at revision...) when I set the Repository URL to
          svn://<servername>/applications/xxx/trunk/yyy
          (without the second slash after xxx).

          It seems that the repository url in the working copy is normalized (e.g. removing double slashes, port number) by the svn client, but the check if the working copy matches the project does not perform this kind of normalization.

          Jenkins ver. 1.471
          Subversion Plugin version 1.4

          Andreas Schoen added a comment - I had the same problem with a project with the Repository URL of the following pattern: svn://<servername>/applications/xxx//trunk/yyy The polling message was: Workspace doesn't contain svn://<servername>/applications/xxx//trunk/yyy. Need a new build. However, the polling did work successfully (no changes, at revision...) when I set the Repository URL to svn://<servername>/applications/xxx/trunk/yyy (without the second slash after xxx). It seems that the repository url in the working copy is normalized (e.g. removing double slashes, port number) by the svn client, but the check if the working copy matches the project does not perform this kind of normalization. Jenkins ver. 1.471 Subversion Plugin version 1.4

          Damien Finck added a comment -

          @Andreas : I agree with you.
          I try to add double '/' in URL and this problem come back.

          A good idea is to add a message when a URL is not normalized.

          Jenkins v1.491
          Subversion Plugin v1.43

          Damien Finck added a comment - @Andreas : I agree with you. I try to add double '/' in URL and this problem come back. A good idea is to add a message when a URL is not normalized. Jenkins v1.491 Subversion Plugin v1.43

          Matt Layman added a comment -

          This error message comes from the hudson.scm.subversion.UpdateUpdater.java at line 96. The reason for the failure is because svnInfo.url != url. So I built a test version of the plugin and logged some extra information so I could find out what 'svnInfo.url' was and what 'url' was.

          My job configuration uses the 'svn update' option and my SVN URL looks something like this (I've had to modify the URL because the path is on an intranet where info is proprietary):

          https://<some IP address>/svn/REPO/directory/branches/some_branch/@HEAD

          In this scenario, I got the following:

          • svnInfo.url = https://<some IP address>/svn/REPO/directory/branches/some_branch
          • url = https://<some IP address>/svn/REPO/directory/branches/some_branch/

          The observable difference is that svnInfo.url has no trailing slash and url has a trailing slash.

          Therefore, I presume that the fix is to modify one of those two variables in some appropriate manner so that they agree with each other. If I have time to investigate the appropriate place to add (or remove) the slash, I'll report back.

          Matt Layman added a comment - This error message comes from the hudson.scm.subversion.UpdateUpdater.java at line 96. The reason for the failure is because svnInfo.url != url. So I built a test version of the plugin and logged some extra information so I could find out what 'svnInfo.url' was and what 'url' was. My job configuration uses the 'svn update' option and my SVN URL looks something like this (I've had to modify the URL because the path is on an intranet where info is proprietary): https://<some IP address>/svn/REPO/directory/branches/some_branch/@HEAD In this scenario, I got the following: svnInfo.url = https://<some IP address>/svn/REPO/directory/branches/some_branch url = https://<some IP address>/svn/REPO/directory/branches/some_branch/ The observable difference is that svnInfo.url has no trailing slash and url has a trailing slash. Therefore, I presume that the fix is to modify one of those two variables in some appropriate manner so that they agree with each other. If I have time to investigate the appropriate place to add (or remove) the slash, I'll report back.

          Matt Layman added a comment -

          I did some more research and have concluded that changing svnInfo.url to introduce a slash when present would be very difficult (even though it would be more correct since including a trailing slash in a URL is perfectly valid syntax).

          The problem is that the SVNInfo object is constructed with a File object. I ran a test and confirmed that none of the string getters from java.io.File will return a path with a trailing slash even if the trailing slash is provided in the File constructor. Thus, the SVNInfo object has no way of knowing if a user provided the trailing slash in a path or not.

          This means that the only feasible way to fix this is either to:

          1. Change ModuleLocation.getURL() so that it won't return a trailing slash (which could have unforseen impacts to other code which might depend on the trailing slash). OR...
          2. Change UpdateUpdater.isUpdatable so that when getURL is called, a trailing slash is removed if it exists, in order to normalize url and svnInfo.url.

          By the way, for those looking for a workaround, remove the trailing slash from the URL in your job. This has the effect of removing it from url so that it will match svnInfo.url. My earlier example would look like:

          https://<some IP address>/svn/REPO/directory/branches/some_branch@HEAD

          Of course, all of this may not be the only reason that url and svnInfo.url wouldn't match, but it is definitely one reason.

          Matt Layman added a comment - I did some more research and have concluded that changing svnInfo.url to introduce a slash when present would be very difficult (even though it would be more correct since including a trailing slash in a URL is perfectly valid syntax). The problem is that the SVNInfo object is constructed with a File object. I ran a test and confirmed that none of the string getters from java.io.File will return a path with a trailing slash even if the trailing slash is provided in the File constructor. Thus, the SVNInfo object has no way of knowing if a user provided the trailing slash in a path or not. This means that the only feasible way to fix this is either to: Change ModuleLocation.getURL() so that it won't return a trailing slash (which could have unforseen impacts to other code which might depend on the trailing slash). OR... Change UpdateUpdater.isUpdatable so that when getURL is called, a trailing slash is removed if it exists, in order to normalize url and svnInfo.url . By the way, for those looking for a workaround , remove the trailing slash from the URL in your job. This has the effect of removing it from url so that it will match svnInfo.url . My earlier example would look like: https://<some IP address>/svn/REPO/directory/branches/some_branch@HEAD Of course, all of this may not be the only reason that url and svnInfo.url wouldn't match, but it is definitely one reason.

          From what I see, the comments so far describe two problems.

          1. multiple checkouts onto the workspace collide
          2. svn URLs do not match internally when ending on a slash

          Personally, I had problem No. 1. Jeff Payne's solution worked for that. Also, that problem yielded the very same error message as given in the description section above, while problem No. 2 seems to give a different error message under circumstances.

          Pjotr Vladrisk added a comment - From what I see, the comments so far describe two problems. multiple checkouts onto the workspace collide svn URLs do not match internally when ending on a slash Personally, I had problem No. 1. Jeff Payne's solution worked for that. Also, that problem yielded the very same error message as given in the description section above, while problem No. 2 seems to give a different error message under circumstances.

          Alan Birtles added a comment -

          just had a similar issue, in our case we had the svn url defined as http://server:80/svn/repo, I think the svn checkout strips out the port number then the polling thinks that the workspace doesn't contain the same working copy.

          Alan Birtles added a comment - just had a similar issue, in our case we had the svn url defined as http://server:80/svn/repo , I think the svn checkout strips out the port number then the polling thinks that the workspace doesn't contain the same working copy.

          Alexandru Gheorghe added a comment - - edited

          Same issue as damienfinck67 described we're encoutering with latest Jenkins version to this date: v1.602:

          1. subversion plugin: 2.5
          2. multiple repositories:
            • some fetched to different folders
            • the rest fetched to one folder
          3. each time there's a trigger (timed at 15 minutes) even if there are no changes polling takes place
          4. the checkout will delete files and add ones on the first poll of the first repository, then the second does the same; what is interesting is that in the end it will balance out and it'll work but each time the build is triggered it will fetch the files in the workspace
          5. at each 15 minutes a new build takes 11MB so that in a couple of days we got over 1GB of disk space occupied; not something wanted

          Tried to append @HEAD at the URL but no luck:

          https://url/svn/branch@HEAD

          , it will indeed go to HEAD and poll there (unlike before where the revision was the date at which it was started) but still with "No changes" in the subversion polling log it still gets the files and checks out as if it goes back 1 revision and sees new changes.

          Alexandru Gheorghe added a comment - - edited Same issue as damienfinck67 described we're encoutering with latest Jenkins version to this date: v1.602 : subversion plugin: 2.5 multiple repositories: some fetched to different folders the rest fetched to one folder each time there's a trigger (timed at 15 minutes) even if there are no changes polling takes place the checkout will delete files and add ones on the first poll of the first repository, then the second does the same; what is interesting is that in the end it will balance out and it'll work but each time the build is triggered it will fetch the files in the workspace at each 15 minutes a new build takes 11MB so that in a couple of days we got over 1GB of disk space occupied; not something wanted Tried to append @HEAD at the URL but no luck: https://url/svn/branch@HEAD , it will indeed go to HEAD and poll there (unlike before where the revision was the date at which it was started) but still with "No changes" in the subversion polling log it still gets the files and checks out as if it goes back 1 revision and sees new changes.

            Unassigned Unassigned
            giles Giles Dermody
            Votes:
            17 Vote for this issue
            Watchers:
            18 Start watching this issue

              Created:
              Updated: