-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Windows, Visual SVN
-
Powered by SuggestiMate
After upgrade of Jenkins/Subversion plugin from version 1.447/1.39 to version 1.501/1.45, we have noticed significant delay in triggering builds on commit (around 1 minute). We are using notify CI method in our post-commit hooks.Users who do not have jobs configured in Jenkins are being badly impacted, while post commit hook notifies Jenkins.
Downgrading Subversion plugin back to 1.39 removes this issue, but we would like to be on a higher version as Subversion 1.7 is supported those versions.
- is duplicated by
-
JENKINS-17808 notifyCommit response time far slower in Subversion plugin 1.45 vs 1.44 - causes svn checkouts to 'hang'
-
- Reopened
-
[JENKINS-17616] Performance issue with Subversion plugin - trigger build on SVN commit takes too long
I have the same problem after upgrading jenkins svn plugin to 1.45 or later. The svn post-commit just times out with following visible message in user side:
HTTP request sent, awaiting response... Read error (Connection timed out) in headers. Read error (Connection timed out) in headers.
No immediate exception is produced in Jenkins console, but after a long time I noticed some exceptions being displayed about this timeout and failure to handle post-commit notification, see exception_long_after_timeout.txt
Additional info:
I've experienced this With jenkins masters running both in Solaris 10 sparc and RHEL6.2 x86_64, with lot of jobs in them.
Up to subversion plugin 1.44 the post-commit doesn't have any delay so that's the one we are currently using.
On newly fresh installations this problem doesn't seem to happen, so it may be either factored by the amount of jobs or combination of plugins that could be causing this. If anyone has any idea how to troubleshoot this and pinpoint the problem I'll be glad to experiment it.
This has definitely have to do with the fact that we have large amount of jobs (around 185) configured on a single Master Jenkins.
On a test installation with significantly smaller number of jobs the delay is unnoticeable.
When calling a job directly through post-commit hooks it starts building instantaneously, however if Jenkins has to scan through the job configs to find the matching job the delay becomes unacceptable.
> If anyone has any idea how to troubleshoot this and pinpoint the problem I'll be glad to experiment it.
As I've said, a threadDump would be helpful
We were having a similar issue after 1.43 -> 1.54 plugin upgrade : our post-commit hook was during from 2 minutes to 5 minutes... and sometimes the job ended before the commit...
We have 75 jobs with 1 svn location, and each uses two or more externals that are common of multiple repositories (some jobs are also using same location to a trunk).
I've patched the doNotifyCommit method and our post commit now is around 20 seconds
The issue is in the UUID determination method that makes an svn call (https in our case) and repeat this call for each locations. I have just add a simple "cache" map that prevents newer calls on already gathered UUIDs.
Here is the pull request : https://github.com/jenkinsci/subversion-plugin/pull/59
EDIT to be more precise on results (20 seconds were measured at client side, including upload commit time, at server side, it lower : 11 seconds) :
- with UUID caching :
Dec 12, 2013 9:41:35 PM FINE hudson.scm.SubversionRepositoryStatus
Starting subversion locations checks for all jobs
Dec 12, 2013 9:41:37 PM FINE hudson.scm.SubversionRepositoryStatus
Scheduling the immediate polling of hudson.model.FreeStyleProject@25a598f5[test-job]
Dec 12, 2013 9:41:37 PM FINE hudson.triggers.SCMTrigger
Scheduling a polling for hudson.model.FreeStyleProject@25a598f5[test-job]
Dec 12, 2013 9:41:37 PM FINE hudson.triggers.SCMTrigger
scheduling the trigger to (asynchronously) run
Dec 12, 2013 9:41:42 PM INFO hudson.triggers.SCMTrigger$Runner run
SCM changes detected in test-job. Triggering #137
Dec 12, 2013 9:41:46 PM FINE hudson.scm.SubversionRepositoryStatus
Ended subversion locations checks for all job
- without UUID caching :
Dec 12, 2013 10:15:13 PM FINE hudson.scm.SubversionRepositoryStatus
Starting subversion locations checks for all jobs
Dec 12, 2013 10:15:40 PM FINE hudson.scm.SubversionRepositoryStatus
Scheduling the immediate polling of hudson.model.FreeStyleProject@350d5a52[test-job]
Dec 12, 2013 10:15:40 PM FINE hudson.triggers.SCMTrigger
Scheduling a polling for hudson.model.FreeStyleProject@350d5a52[test-job]
Dec 12, 2013 10:15:40 PM FINE hudson.triggers.SCMTrigger
scheduling the trigger to (asynchronously) run
Dec 12, 2013 10:15:51 PM INFO hudson.triggers.SCMTrigger$Runner run
SCM changes detected in test-job. Triggering #138
Dec 12, 2013 10:18:20 PM FINE hudson.scm.SubversionRepositoryStatus
Ended subversion locations checks for all jobs
Code changed in jenkins
User: kenji21
Path:
src/main/java/hudson/scm/SubversionRepositoryStatus.java
http://jenkins-ci.org/commit/subversion-plugin/5a079c2e9f421184229231cbcc76cdd4be991135
Log:
Adding log to get duration of UUID checks in doNotifyCommit. Added a HashMap to no more make connections to svn repository for asking its UUID if previously asked. JENKINS-17616
Code changed in jenkins
User: Stephen Connolly
Path:
src/main/java/hudson/scm/SubversionRepositoryStatus.java
http://jenkins-ci.org/commit/subversion-plugin/3c89f87f5930a2c1cf1bb38f800177e9a542e429
Log:
Merge pull request #59 from kenji21/master
[FIXED JENKINS-17616] Add a HashMap for limiting connection to gathering UUID from repository with same locations
Compare: https://github.com/jenkinsci/subversion-plugin/compare/a7c53082055e...3c89f87f5930
This seems to be working better with caching (tested with svn plugin 2.2), however due to credentials refactoring affecting svn:externals JENKINS-21785 there are still some issues that makes some cache misses.
This bug is reproducible in the Jenkins version 1.633, with Subversion Plug-in 2.5.3 (pinned).
The error in the post-commit is:
Connecting to myjenkins:8080... connected.
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
I used the next timeout values: 2, 8, 16, 32, 64, 128, 256, 1024
In all cases the error is the same.
Does anybody knows a fix or workaround for this?
roberto87 Please try the previous plugin versions (2.5, 2.5.1, …) to see whether this is a regression.
Also, provide the output of http://yourjenkins/threadDump during the time the post-commit hook is connected.
danielbeck I tried with all previous version to 2.5.3 and the problem persists.
Here is my threadDump:
Handling POST /subversion/*********************/notifyCommit from myIP : RequestHandlerThread30
"Handling POST /subversion/********************/notifyCommit from myIP : RequestHandlerThread30" Id=138 Group=main BLOCKED on hudson.scm.SubversionSCM$ModuleLocation@4389df0f owned by "Handling POST /subversion/********************/notifyCommit from myIP : RequestHandlerThread22" Id=124
at hudson.scm.SubversionSCM$ModuleLocation.getUUID(SubversionSCM.java:2726)
- blocked on hudson.scm.SubversionSCM$ModuleLocation@4389df0f
at hudson.scm.SubversionRepositoryStatus$JobTriggerListenerImpl.onNotify(SubversionRepositoryStatus.java:209)
at hudson.scm.SubversionRepositoryStatus.doNotifyCommit(SubversionRepositoryStatus.java:137)
at sun.reflect.GeneratedMethodAccessor291.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
at org.kohsuke.stapler.MetaClass$13.dispatch(MetaClass.java:411)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
at org.kohsuke.stapler.MetaClass$13.dispatch(MetaClass.java:411)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:123)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@6a898dd2
Looks like a possible deadlock. Please provide the entire output, not an excerpt.
danielbeck just to confirm, did you see the output? I mean this http://justpaste.it/oibx, is working?
roberto87, Could you provide a step by step process to reproduce the problem? I'd like to help here.
recena From the stack trace it's many projects with many SCM locations. There should be caching going on that limits the number of necessary connections to determine each SCMLocation's UUID (to determine whether the post-commit notification is relevant), but maybe it's broken again?
recena The request to send a SVN poll event is taking long time, more details in the attached log.
roberto87, this issue is too old. I'd like to confirm what version of Subversion Plugin are you using.
recena The version that I have installed is "Jenkins Subversion Plug-in 2.5.2"
Thank you in advance
recena The version that I have installed is "Jenkins Subversion Plug-in 2.5.2"
Thank you in advance
A threadDump of Jenkins while the post commit hook is notifying Jenkins, would help to track down the problem.