Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19507

Jenkins runs out of threads when using SVN post commit hook with svn+ssh URL

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • subversion-plugin
    • Linux Red Hat Enterprise Linux Server release 6.4 (Santiago), Jenkins V 1.509.2 Installed via provided rpm, Subversion Plugin V 1.5, Subversion 1.6.9

      We have a running a Jenkins-Instance with about 900 build jobs. Most of them are configured with SCM Trigger (Subversion). Due to performance issues with the SCM we employ the svn+ssh protocol for the coummincation with the SCM. In order to reduce load on Subversion we try to trigger our CI jenkins jobs with the post commit hook. When we trigger the Jenkins-URL in order to notify jenkins on SCM change we get the following exception in the log file and the jenkins server is unaccessible until jenkins server restart:

      Sep 05, 2013 9:10:30 AM winstone.Logger logInternal
      SEVERE: Error while serving http://jenkins-qa.fiducia.de/jenkins/subversion/9f1f1913-8d78-0410-adcc-cdd7385d28bb/notifyCommit
      java.lang.reflect.InvocationTargetException
      at sun.reflect.GeneratedMethodAccessor690.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:601)
      at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:288)
      at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:151)
      at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:90)
      at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111)
      at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
      at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:384)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
      at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:384)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:488)
      at org.kohsuke.stapler.Stapler.service(Stapler.java:162)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
      at winstone.ServletConfiguration.execute(ServletConfiguration.java:248)
      at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
      at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
      at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
      at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86)
      at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
      at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:66)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
      at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
      at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:54)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
      at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
      at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:722)
      Caused by: java.lang.OutOfMemoryError: unable to create new native thread
      at java.lang.Thread.start0(Native Method)
      at java.lang.Thread.start(Thread.java:691)
      at com.trilead.ssh2.transport.TransportManager.initialize(TransportManager.java:516)
      at com.trilead.ssh2.Connection.connect(Connection.java:733)
      at org.tmatesoft.svn.core.internal.io.svn.ssh.SshHost.openConnection(SshHost.java:217)
      at org.tmatesoft.svn.core.internal.io.svn.ssh.SshHost.openSession(SshHost.java:145)
      at org.tmatesoft.svn.core.internal.io.svn.ssh.SshSessionPool.openSession(SshSessionPool.java:81)
      at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector.open(SVNSSHConnector.java:129)
      at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:77)
      at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:1252)
      at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.testConnection(SVNRepositoryImpl.java:95)
      at hudson.scm.SubversionSCM$ModuleLocation.getUUID(SubversionSCM.java:2363)
      at hudson.scm.SubversionRepositoryStatus.doNotifyCommit(SubversionRepositoryStatus.java:138)
      ... 76 more

      Sep 05, 2013 9:10:30 AM org.kohsuke.stapler.compression.CompressionFilter reportException
      WARNING: Untrapped servlet exception
      javax.servlet.ServletException: java.lang.OutOfMemoryError: unable to create new native thread
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:616)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
      at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:384)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
      at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:384)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:488)
      at org.kohsuke.stapler.Stapler.service(Stapler.java:162)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
      at winstone.ServletConfiguration.execute(ServletConfiguration.java:248)
      at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
      at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
      at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
      at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86)
      at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
      at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:66)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
      at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
      at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:54)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
      at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
      at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:722)
      Caused by: java.lang.OutOfMemoryError: unable to create new native thread
      at java.lang.Thread.start0(Native Method)
      at java.lang.Thread.start(Thread.java:691)
      at com.trilead.ssh2.transport.TransportManager.initialize(TransportManager.java:516)
      at com.trilead.ssh2.Connection.connect(Connection.java:733)
      at org.tmatesoft.svn.core.internal.io.svn.ssh.SshHost.openConnection(SshHost.java:217)
      at org.tmatesoft.svn.core.internal.io.svn.ssh.SshHost.openSession(SshHost.java:145)
      at org.tmatesoft.svn.core.internal.io.svn.ssh.SshSessionPool.openSession(SshSessionPool.java:81)
      at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector.open(SVNSSHConnector.java:129)
      at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:77)
      at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:1252)
      at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.testConnection(SVNRepositoryImpl.java:95)
      at hudson.scm.SubversionSCM$ModuleLocation.getUUID(SubversionSCM.java:2363)
      at hudson.scm.SubversionRepositoryStatus.doNotifyCommit(SubversionRepositoryStatus.java:138)
      at sun.reflect.GeneratedMethodAccessor690.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:601)
      at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:288)
      at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:151)
      at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:90)
      at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111)
      at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574)
      ... 67 more

          [JENKINS-19507] Jenkins runs out of threads when using SVN post commit hook with svn+ssh URL

          SshSessionPool maintains SshHost (so communication thread) for each remote node, and purge them on timeout. When jenkins starts, on first /notifyCommit subversion plugin has to compare incoming UUID with job's ones, that are lazy-loaded. If each job uses a distinct repository host, this can result in hundreds ssh thread started.

          Reducing thread stack size (-Xss) can reduce impact for this issue.

          Nicolas De Loof added a comment - SshSessionPool maintains SshHost (so communication thread) for each remote node, and purge them on timeout. When jenkins starts, on first /notifyCommit subversion plugin has to compare incoming UUID with job's ones, that are lazy-loaded. If each job uses a distinct repository host, this can result in hundreds ssh thread started. Reducing thread stack size (-Xss) can reduce impact for this issue.

          Roland Gruber added a comment -

          -Xss does not work because the problem is the number of threads (unable to create new native thread).
          The operating system limits the number of threads to 1024. Of course, we could increase that limit. But running multiple thousand threads in parallel with Subversion calls seems not a good idea.

          In my opinion, there should be a queue that executes the SVN commands. Just like the SVN configuration for polling option (e.g. max 5 parallel).

          Roland Gruber added a comment - -Xss does not work because the problem is the number of threads (unable to create new native thread). The operating system limits the number of threads to 1024. Of course, we could increase that limit. But running multiple thousand threads in parallel with Subversion calls seems not a good idea. In my opinion, there should be a queue that executes the SVN commands. Just like the SVN configuration for polling option (e.g. max 5 parallel).

            Unassigned Unassigned
            vbuehner Valentin Buehner
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: