Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-11503

Too many svn connections

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Major Major
    • subversion-plugin
    • None

      Hello,

      I use Jenkins version 1.436 . I use svn+ssh. I found that Jenkins opens too many svn connections.

      sudo lsof -c /java/ -a -i
      java 26534 root 2073u IPv6 2366723255 TCP myserver:35173->server:ssh (ESTABLISHED)
      java 26534 root 2074u IPv6 2366723380 TCP myserver:35174->server:ssh (ESTABLISHED)
      java 26534 root 2076u IPv6 2366724740 TCP myserver:35177->server:ssh (ESTABLISHED)
      java 26534 root 2087u IPv6 2366728878 TCP myserver:35180->server:ssh (ESTABLISHED)
      java 26534 root 2089u IPv6 2366730081 TCP myserver:35182->server:ssh (ESTABLISHED)
      java 26534 root 2090u IPv6 2366734351 TCP myserver:35186->server:ssh (ESTABLISHED)

      sudo lsof -c /java/ -a -i | wc -l
      1443

      Subversion Plugin version 1.28.

      All these projects are from the same repo but from different folders. The number of projects approximately 400.

      I also use svn hook that create POST request to Jenkins for every commit. It looks like in documentation
      https://wiki.jenkins-ci.org/display/JENKINS/Subversion+Plugin#

      The problem can be reproduce the following way.
      1) Restart jenkins
      2) Add 400 jobs
      3) Commit something (with svn hook)

      The result wget in svn hook cannot be done since wget cannot pass and retries 20 times (by default) and number of connections grows up.

      ThreadDump (There were also a POST request from hook in thread, but I cannot find it right now):

      Executor #3 for master : executing buildfarm #193

      "Executor #3 for master : executing buildfarm #193" Id=82 Group=main WAITING on java.lang.UNIXProcess@34ba9d81
      at java.lang.Object.wait(Native Method)

      • waiting on java.lang.UNIXProcess@34ba9d81
        at java.lang.Object.wait(Object.java:485)
        at java.lang.UNIXProcess.waitFor(UNIXProcess.java:165)
        at hudson.Proc$LocalProc.join(Proc.java:319)
        at hudson.Launcher$ProcStarter.join(Launcher.java:345)
        at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:82)
        at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
        at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:694)
        at hudson.model.Build$RunnerImpl.build(Build.java:178)
        at hudson.model.Build$RunnerImpl.doRun(Build.java:139)
        at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:460)
        at hudson.model.Run.run(Run.java:1404)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
        at hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:230)

      Handling GET /threadDump : RequestHandlerThread20

      "Handling GET /threadDump : RequestHandlerThread20" Id=1562 Group=main TIMED_WAITING on java.util.concurrent.FutureTask$Sync@66d2774f
      at sun.misc.Unsafe.park(Native Method)

      • waiting on java.util.concurrent.FutureTask$Sync@66d2774f
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:947)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1239)
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:227)
        at java.util.concurrent.FutureTask.get(FutureTask.java:91)
        at hudson.remoting.LocalChannel$2.get(LocalChannel.java:81)
        at jenkins.model.Jenkins.getAllThreadDumps(Jenkins.java:2651)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258)
        at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
        at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
        at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
        at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
        at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
        at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:72)
        at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61)
        at org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsIterator(ExpressionSupport.java:94)
        at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:89)
        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98)
        at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98)
        at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
        at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:63)
        at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
        at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
        at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:92)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:477)
        at org.kohsuke.stapler.Stapler.service(Stapler.java:159)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
        at winstone.ServletConfiguration.execute(ServletConfiguration.java:249)
        at winstone.RequestDispatcher.forward(RequestDispatcher.java:335)
        at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:378)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
        at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
        at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
        at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
        at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
        at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
        at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
        at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
        at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
        at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
        at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
        at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
        at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
        at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
        at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
        at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:244)
        at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)
        at java.lang.Thread.run(Thread.java:619)

      Jenkins cron thread

      "Jenkins cron thread" Id=22 Group=main TIMED_WAITING on java.util.TaskQueue@5583d693
      at java.lang.Object.wait(Native Method)

      • waiting on java.util.TaskQueue@5583d693
        at java.util.TimerThread.mainLoop(Timer.java:509)
        at java.util.TimerThread.run(Timer.java:462)

      Jenkins UDP 33848 monitoring thread

      "Jenkins UDP 33848 monitoring thread" Id=75 Group=main RUNNABLE (in native)
      at java.net.PlainDatagramSocketImpl.receive0(Native Method)

      • locked java.net.PlainDatagramSocketImpl@3a215406
        at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
      • locked java.net.PlainDatagramSocketImpl@3a215406
        at java.net.DatagramSocket.receive(DatagramSocket.java:725)
      • locked java.net.DatagramPacket@6b12eaa3
      • locked java.net.MulticastSocket@26eaaa40
        at hudson.UDPBroadcastThread.run(UDPBroadcastThread.java:79)

      JmDNS(server.local.).State.Timer

      "JmDNS(server.local.).State.Timer" Id=78 Group=main TIMED_WAITING on java.util.TaskQueue@722d5f24
      at java.lang.Object.wait(Native Method)

      • waiting on java.util.TaskQueue@722d5f24
        at java.util.TimerThread.mainLoop(Timer.java:509)
        at java.util.TimerThread.run(Timer.java:462)

            Unassigned Unassigned
            nikita Nikita Aznauryan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: