Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17616

Performance issue with Subversion plugin - trigger build on SVN commit takes too long

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • subversion-plugin
    • None
    • Windows, Visual SVN

      After upgrade of Jenkins/Subversion plugin from version 1.447/1.39 to version 1.501/1.45, we have noticed significant delay in triggering builds on commit (around 1 minute). We are using notify CI method in our post-commit hooks.Users who do not have jobs configured in Jenkins are being badly impacted, while post commit hook notifies Jenkins.
      Downgrading Subversion plugin back to 1.39 removes this issue, but we would like to be on a higher version as Subversion 1.7 is supported those versions.

          [JENKINS-17616] Performance issue with Subversion plugin - trigger build on SVN commit takes too long

          Glen Core created issue -
          Michael Fowler made changes -
          Link New: This issue is duplicated by JENKINS-17808 [ JENKINS-17808 ]

          kutzi added a comment -

          A threadDump of Jenkins while the post commit hook is notifying Jenkins, would help to track down the problem.

          kutzi added a comment - A threadDump of Jenkins while the post commit hook is notifying Jenkins, would help to track down the problem.

          Jose Sa added a comment -

          I have the same problem after upgrading jenkins svn plugin to 1.45 or later. The svn post-commit just times out with following visible message in user side:

          HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
          Read error (Connection timed out) in headers.
          

          No immediate exception is produced in Jenkins console, but after a long time I noticed some exceptions being displayed about this timeout and failure to handle post-commit notification, see exception_long_after_timeout.txt

          Jose Sa added a comment - I have the same problem after upgrading jenkins svn plugin to 1.45 or later. The svn post-commit just times out with following visible message in user side: HTTP request sent, awaiting response... Read error (Connection timed out) in headers. Read error (Connection timed out) in headers. No immediate exception is produced in Jenkins console, but after a long time I noticed some exceptions being displayed about this timeout and failure to handle post-commit notification, see exception_long_after_timeout.txt
          Jose Sa made changes -
          Attachment New: exception_long_after_timeout.txt [ 23772 ]

          Jose Sa added a comment -

          Additional info:
          I've experienced this With jenkins masters running both in Solaris 10 sparc and RHEL6.2 x86_64, with lot of jobs in them.

          Up to subversion plugin 1.44 the post-commit doesn't have any delay so that's the one we are currently using.

          On newly fresh installations this problem doesn't seem to happen, so it may be either factored by the amount of jobs or combination of plugins that could be causing this. If anyone has any idea how to troubleshoot this and pinpoint the problem I'll be glad to experiment it.

          Jose Sa added a comment - Additional info: I've experienced this With jenkins masters running both in Solaris 10 sparc and RHEL6.2 x86_64, with lot of jobs in them. Up to subversion plugin 1.44 the post-commit doesn't have any delay so that's the one we are currently using. On newly fresh installations this problem doesn't seem to happen, so it may be either factored by the amount of jobs or combination of plugins that could be causing this. If anyone has any idea how to troubleshoot this and pinpoint the problem I'll be glad to experiment it.

          Glen Core added a comment -

          This has definitely have to do with the fact that we have large amount of jobs (around 185) configured on a single Master Jenkins.
          On a test installation with significantly smaller number of jobs the delay is unnoticeable.
          When calling a job directly through post-commit hooks it starts building instantaneously, however if Jenkins has to scan through the job configs to find the matching job the delay becomes unacceptable.

          Glen Core added a comment - This has definitely have to do with the fact that we have large amount of jobs (around 185) configured on a single Master Jenkins. On a test installation with significantly smaller number of jobs the delay is unnoticeable. When calling a job directly through post-commit hooks it starts building instantaneously, however if Jenkins has to scan through the job configs to find the matching job the delay becomes unacceptable.

          kutzi added a comment -

          > If anyone has any idea how to troubleshoot this and pinpoint the problem I'll be glad to experiment it.

          As I've said, a threadDump would be helpful

          kutzi added a comment - > If anyone has any idea how to troubleshoot this and pinpoint the problem I'll be glad to experiment it. As I've said, a threadDump would be helpful

          Richard Bergoin added a comment - - edited

          We were having a similar issue after 1.43 -> 1.54 plugin upgrade : our post-commit hook was during from 2 minutes to 5 minutes... and sometimes the job ended before the commit...
          We have 75 jobs with 1 svn location, and each uses two or more externals that are common of multiple repositories (some jobs are also using same location to a trunk).
          I've patched the doNotifyCommit method and our post commit now is around 20 seconds

          The issue is in the UUID determination method that makes an svn call (https in our case) and repeat this call for each locations. I have just add a simple "cache" map that prevents newer calls on already gathered UUIDs.

          Here is the pull request : https://github.com/jenkinsci/subversion-plugin/pull/59

          EDIT to be more precise on results (20 seconds were measured at client side, including upload commit time, at server side, it lower : 11 seconds) :

          • with UUID caching :

          Dec 12, 2013 9:41:35 PM FINE hudson.scm.SubversionRepositoryStatus
          Starting subversion locations checks for all jobs
          Dec 12, 2013 9:41:37 PM FINE hudson.scm.SubversionRepositoryStatus
          Scheduling the immediate polling of hudson.model.FreeStyleProject@25a598f5[test-job]
          Dec 12, 2013 9:41:37 PM FINE hudson.triggers.SCMTrigger
          Scheduling a polling for hudson.model.FreeStyleProject@25a598f5[test-job]
          Dec 12, 2013 9:41:37 PM FINE hudson.triggers.SCMTrigger
          scheduling the trigger to (asynchronously) run
          Dec 12, 2013 9:41:42 PM INFO hudson.triggers.SCMTrigger$Runner run
          SCM changes detected in test-job. Triggering #137
          Dec 12, 2013 9:41:46 PM FINE hudson.scm.SubversionRepositoryStatus
          Ended subversion locations checks for all job

          • without UUID caching :

          Dec 12, 2013 10:15:13 PM FINE hudson.scm.SubversionRepositoryStatus
          Starting subversion locations checks for all jobs
          Dec 12, 2013 10:15:40 PM FINE hudson.scm.SubversionRepositoryStatus
          Scheduling the immediate polling of hudson.model.FreeStyleProject@350d5a52[test-job]
          Dec 12, 2013 10:15:40 PM FINE hudson.triggers.SCMTrigger
          Scheduling a polling for hudson.model.FreeStyleProject@350d5a52[test-job]
          Dec 12, 2013 10:15:40 PM FINE hudson.triggers.SCMTrigger
          scheduling the trigger to (asynchronously) run
          Dec 12, 2013 10:15:51 PM INFO hudson.triggers.SCMTrigger$Runner run
          SCM changes detected in test-job. Triggering #138
          Dec 12, 2013 10:18:20 PM FINE hudson.scm.SubversionRepositoryStatus
          Ended subversion locations checks for all jobs

          Richard Bergoin added a comment - - edited We were having a similar issue after 1.43 -> 1.54 plugin upgrade : our post-commit hook was during from 2 minutes to 5 minutes... and sometimes the job ended before the commit... We have 75 jobs with 1 svn location, and each uses two or more externals that are common of multiple repositories (some jobs are also using same location to a trunk). I've patched the doNotifyCommit method and our post commit now is around 20 seconds The issue is in the UUID determination method that makes an svn call (https in our case) and repeat this call for each locations. I have just add a simple "cache" map that prevents newer calls on already gathered UUIDs. Here is the pull request : https://github.com/jenkinsci/subversion-plugin/pull/59 EDIT to be more precise on results (20 seconds were measured at client side, including upload commit time, at server side, it lower : 11 seconds) : with UUID caching : Dec 12, 2013 9:41:35 PM FINE hudson.scm.SubversionRepositoryStatus Starting subversion locations checks for all jobs Dec 12, 2013 9:41:37 PM FINE hudson.scm.SubversionRepositoryStatus Scheduling the immediate polling of hudson.model.FreeStyleProject@25a598f5 [test-job] Dec 12, 2013 9:41:37 PM FINE hudson.triggers.SCMTrigger Scheduling a polling for hudson.model.FreeStyleProject@25a598f5 [test-job] Dec 12, 2013 9:41:37 PM FINE hudson.triggers.SCMTrigger scheduling the trigger to (asynchronously) run Dec 12, 2013 9:41:42 PM INFO hudson.triggers.SCMTrigger$Runner run SCM changes detected in test-job. Triggering #137 Dec 12, 2013 9:41:46 PM FINE hudson.scm.SubversionRepositoryStatus Ended subversion locations checks for all job without UUID caching : Dec 12, 2013 10:15:13 PM FINE hudson.scm.SubversionRepositoryStatus Starting subversion locations checks for all jobs Dec 12, 2013 10:15:40 PM FINE hudson.scm.SubversionRepositoryStatus Scheduling the immediate polling of hudson.model.FreeStyleProject@350d5a52 [test-job] Dec 12, 2013 10:15:40 PM FINE hudson.triggers.SCMTrigger Scheduling a polling for hudson.model.FreeStyleProject@350d5a52 [test-job] Dec 12, 2013 10:15:40 PM FINE hudson.triggers.SCMTrigger scheduling the trigger to (asynchronously) run Dec 12, 2013 10:15:51 PM INFO hudson.triggers.SCMTrigger$Runner run SCM changes detected in test-job. Triggering #138 Dec 12, 2013 10:18:20 PM FINE hudson.scm.SubversionRepositoryStatus Ended subversion locations checks for all jobs

          Jose Sa added a comment -

          Nice. Looking forward to test this.

          Jose Sa added a comment - Nice. Looking forward to test this.

            recena Manuel Recena Soto
            gcadmin Glen Core
            Votes:
            3 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated: