Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29914

Subversion working copy is slow (low resolution timestamps?)

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • subversion-plugin
    • Jenkins 1.609.2, Subversion plugin 2.5, OpenJDK 7u79-2.5.6 (as provided by Ubuntu 14.04)

      Working copies created by the Jenkins Subversion plugin are very slow to work with. I think that the problem is that the plugin and its Subversion libraries don't maintain enough precision in file timestamps.

      Details:

      One of the steps in our build process runs svnversion on a large working copy. We noticed that this operation took over 30 times longer in a working copy created by Jenkins' Subversion plugin, compared to a working copy checked out in the standard svn command-line (CLI) client.

      I ran strace svnversion (with the appropriate arguments) from the Jenkins working copy and the CLI working copy and noticed that, for files in the CLI working copy, svnversion did an lstat64 and then proceeded with the next file. For files in Jenkins' working copy, svnversion appeared to do an lstat64 three times then open both the file and its .svn/pristine original version and compare their contents.

      I inspected the .svn/wc.db and noticed that almost every entry in nodes table for the Jenkins working copy has no sub-second precision in the last_mod_time column. (I.e., last_mod_time % 1000000 is 0 for nearly every entry.) In the CLI working copy, the sub-second timestamps in the last_mod_time column match the files' modified timestamps in the filesystem down to the microsecond. I suspect that svnversion detects that the timestamps match down to the microsecond and skips comparing their contents. Because the Jenkins working copy's wc.db doesn't have this sub-second precision, svnversion has to compare file contents, which can be very, very slow.

      I'm unclear on the relationship between the Subversion plugin and any upstream libraries, so if this should be reported to an upstream Subversion implementation, please let me know, and I'll report it there. Thanks.

          [JENKINS-29914] Subversion working copy is slow (low resolution timestamps?)

          Since the implementation of subversion client is done in SVNKit library, I think you should report the bug there.

          Florent Fievez added a comment - Since the implementation of subversion client is done in SVNKit library, I think you should report the bug there.

          Hi, I'm new here and don't know about protocol. Has anyone reported this bug to the SVNkit component area?

          Andrew Wozniak added a comment - Hi, I'm new here and don't know about protocol. Has anyone reported this bug to the SVNkit component area?

          Shannon Kerr added a comment -

          I noticed this too. I actually use svnkit on some older Ubuntu machines that don't have the standard client available in a deb and I do not see svnkit performing this poorly as a stand-alone. Only within the Jenkins context do I see this terribly slow behavior. Also, I use the svn cleanup of the workspace built-in to Jenkins and it just took me 6 minutes to cleanup a handful of files in a workspace. Running svn-clean from the command line took far less time. I can run some tests and get some performance times to compare if desired.

          Shannon Kerr added a comment - I noticed this too. I actually use svnkit on some older Ubuntu machines that don't have the standard client available in a deb and I do not see svnkit performing this poorly as a stand-alone. Only within the Jenkins context do I see this terribly slow behavior. Also, I use the svn cleanup of the workspace built-in to Jenkins and it just took me 6 minutes to cleanup a handful of files in a workspace. Running svn-clean from the command line took far less time. I can run some tests and get some performance times to compare if desired.

          Shannon Kerr added a comment -

          Any update on this? I just executed a build that took over 20 minutes to do a cleanup and svn switch. It takes less than a minute from a command prompt. Seems that anytime there is more than one build running and svn switch on a shared machine, it takes incredibly long to run the svn commands. it's bad enough when it's just one instance, but more than one brings the svn client to its knees. I checked the resources on the machine and the CPU usage was at 30% and there were gigabytes of memory free. Really, the Jenkins implementation svn needs some attention please.

          Shannon Kerr added a comment - Any update on this? I just executed a build that took over 20 minutes to do a cleanup and svn switch. It takes less than a minute from a command prompt. Seems that anytime there is more than one build running and svn switch on a shared machine, it takes incredibly long to run the svn commands. it's bad enough when it's just one instance, but more than one brings the svn client to its knees. I checked the resources on the machine and the CPU usage was at 30% and there were gigabytes of memory free. Really, the Jenkins implementation svn needs some attention please.

          Shannon Kerr added a comment -

          I watching a build right now and I'm at 41 minutes to clean and svn switch a workspace that on the command line would take less than a minute. The system's CPU is at 48% and there are several gigabytes of memory free. This is just crazy that it's taking this long for something so simple. If this is not going to be fixed, are there any suggestions on what we can do to work around this besides switching from SVN to something else?

          Shannon Kerr added a comment - I watching a build right now and I'm at 41 minutes to clean and svn switch a workspace that on the command line would take less than a minute. The system's CPU is at 48% and there are several gigabytes of memory free. This is just crazy that it's taking this long for something so simple. If this is not going to be fixed, are there any suggestions on what we can do to work around this besides switching from SVN to something else?

          Holger added a comment -

          According to https://stackoverflow.com/a/24806116 the underlying problem depends on the Java build you are using.
          I've tried Java8 on Linux: here the resolution of Files.getLastModifiedTime() is only seconds.
          With Java9 on Linux or Java8 on Windows Files.getLastModifiedTime() returns full microseconds resolution.
          If you switch to Windows orJava9, it could work.
          With Java8 on Linux i think SVNKit can't do anything if Java provides only low resolution timestamps. I'd suggest to only use one client either the native svn client or SVNKit but not mix them on the same working copy.

          If you encounter problems with a setup the other way around: full resolution timestamps in wc.db (i.e. when checked out with the native svn client) but SVNKit is slow when used on that working copy? For that situation i've reported a bug at SVNKit https://issues.tmatesoft.com/issue/SVNKIT-711 to let SVNKit interpret the timestamps from wc.db also only with second resolution when necessary.

          Holger added a comment - According to https://stackoverflow.com/a/24806116 the underlying problem depends on the Java build you are using. I've tried Java8 on Linux: here the resolution of Files.getLastModifiedTime() is only seconds. With Java9 on Linux or Java8 on Windows Files.getLastModifiedTime() returns full microseconds resolution. If you switch to Windows orJava9, it could work. With Java8 on Linux i think SVNKit can't do anything if Java provides only low resolution timestamps. I'd suggest to only use one client either the native svn client or SVNKit but not mix them on the same working copy. If you encounter problems with a setup the other way around: full resolution timestamps in wc.db (i.e. when checked out with the native svn client) but SVNKit is slow when used on that working copy? For that situation i've reported a bug at SVNKit https://issues.tmatesoft.com/issue/SVNKIT-711 to let SVNKit interpret the timestamps from wc.db also only with second resolution when necessary.

            Unassigned Unassigned
            jkelley Josh Kelley
            Votes:
            5 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated: