I'm looking for guidance on debugging slow page load times. Ever since we upgraded from 1.480 to 1.484 (and other versions beyond that) we've had very slow page load times. Sometimes upwards of 120 seconds to load a page, nearly any view tab, job configuration etc.

      I've debugged many Java apps and I'm confounded. There are few, if any, Full GC's occuring, regular GC's occur somewhat frequently but take .02 or less seconds. CPU utilization for the java executeable is under %6 and and disk utilization is reasonable (we're averaging a %90+ idle time). The initial HTTP ack comes back immediately but then we wait for the page response to come back for up to 2 minutes sometimes. Othertimes it loads reasonably fast (under 4 seconds). When it's hanging, it's hanging for all requests, much like it was doing a full garbage collect. It feels like some kind of resource contention.

      But I cannot find where the contention is, thread dumps look nominal (I've included one here for example). Something causes long page load times, we back revved to 1.484 thinking it had something to do with lazy loading features but clearly it does not. On 1.480 we did not have these issues. I could use some help figuring out what else to look at to identify why Jenkins is slow. There is a lack of information available on common reasons for slow page load times, this results in a terrible user experience with an otherwise fine tool.

        1. x.diff
          2 kB
        2. threaddump.txt
          106 kB

          [JENKINS-16474] Slow/hung web UI in 1.483+ (stuck in parseURI)

          Maxfield Stewart created issue -

          FYI we found a change note in Jenkins 1.482 indicating that the number of worker threads was reduced to 10 from 1000. We traced our issue to this problem. We can resolve it by upping the thread count the winstone container is using. We set the min threads to 50 and the max threads to 500.

          Maxfield Stewart added a comment - FYI we found a change note in Jenkins 1.482 indicating that the number of worker threads was reduced to 10 from 1000. We traced our issue to this problem. We can resolve it by upping the thread count the winstone container is using. We set the min threads to 50 and the max threads to 500.

          Jesse Glick added a comment - - edited

          Jesse Glick added a comment - - edited 1.483 you mean: winstone 0.9.10-jenkins-38. https://github.com/jenkinsci/winstone/commit/dcc2b4c847ba57da137cc4dd83af585bbfbea41f
          Jesse Glick made changes -
          Component/s New: core [ 15593 ]
          Component/s Original: jboss [ 15686 ]
          Labels New: performance winstone

          Code changed in jenkins
          User: Jesse Glick
          Path:
          README.html
          contrib/README_jp.html
          src/java/winstone/LocalStrings.properties
          http://jenkins-ci.org/commit/winstone/ffe79e8789bfe42edf04b2dc88539942804b7e1d
          Log:
          Diagnosis of JENKINS-16474 complicated by the fact that documentation did not match revised default values.


          You received this message because you are subscribed to the Google Groups "Jenkins Commits" group.
          To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com.
          For more options, visit https://groups.google.com/groups/opt_out.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: README.html contrib/README_jp.html src/java/winstone/LocalStrings.properties http://jenkins-ci.org/commit/winstone/ffe79e8789bfe42edf04b2dc88539942804b7e1d Log: Diagnosis of JENKINS-16474 complicated by the fact that documentation did not match revised default values. – You received this message because you are subscribed to the Google Groups "Jenkins Commits" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out .

          Jesse Glick added a comment -

          Characteristic thread dump seems to have ~20 threads (the new default maximum) stuck in parseURI:

          "RequestHandlerThread[#…]" Id=… Group=main RUNNABLE (in native)
          	at java.net.SocketInputStream.socketRead0(Native Method)
          	at java.net.SocketInputStream.read(SocketInputStream.java:129)
          	at java.net.SocketInputStream.read(SocketInputStream.java:182)
          	at winstone.WinstoneInputStream.read(WinstoneInputStream.java:49)
          	at javax.servlet.ServletInputStream.readLine(ServletInputStream.java:27)
          	at winstone.WinstoneInputStream.readLine(WinstoneInputStream.java:108)
          	at winstone.HttpListener.parseURI(HttpListener.java:239)
          	at winstone.RequestHandlerThread.run(RequestHandlerThread.java:75)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
          	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          	at java.lang.Thread.run(Thread.java:662)
          

          Seems that perhaps Winstone is receiving socket connections which do not send any data. Setting a larger number of handler threads might merely cause this thread leak to not be encountered as quickly.

          The fix of SECURITY-44 included a general Winstone update, so 1.480.x LTS users would also be affected.

          Jesse Glick added a comment - Characteristic thread dump seems to have ~20 threads (the new default maximum) stuck in parseURI : "RequestHandlerThread[#…]" Id=… Group=main RUNNABLE (in native) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.net.SocketInputStream.read(SocketInputStream.java:182) at winstone.WinstoneInputStream.read(WinstoneInputStream.java:49) at javax.servlet.ServletInputStream.readLine(ServletInputStream.java:27) at winstone.WinstoneInputStream.readLine(WinstoneInputStream.java:108) at winstone.HttpListener.parseURI(HttpListener.java:239) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:75) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Seems that perhaps Winstone is receiving socket connections which do not send any data. Setting a larger number of handler threads might merely cause this thread leak to not be encountered as quickly. The fix of SECURITY-44 included a general Winstone update, so 1.480.x LTS users would also be affected.
          Jesse Glick made changes -
          Assignee Original: JulB4 [ julb4 ] New: Jesse Glick [ jglick ]
          Environment New: 1.481+, also LTS updates with SECURITY-44
          Priority Original: Major [ 3 ] New: Critical [ 2 ]
          Summary Original: jenkins v1.484 - Universally slow page load times/hangs New: Slow/hung web UI in 1.483+ (stuck in parseURI)
          Jesse Glick made changes -
          Environment Original: 1.481+, also LTS updates with SECURITY-44 New: 1.483+, also LTS updates with SECURITY-44

          Jesse Glick added a comment -

          Symptoms look similar to a SYN flood or similar denial-of-service attack. Attached test simulates a basic attack in which clients try to read from the HTTP socket without sending any headers; it passes but takes ~25 seconds while the “legitimate” request is waiting in queue.

          Users running Jenkins on an intranet have reported this, however, so a malicious attack is probably not the cause.

          Jesse Glick added a comment - Symptoms look similar to a SYN flood or similar denial-of-service attack. Attached test simulates a basic attack in which clients try to read from the HTTP socket without sending any headers; it passes but takes ~25 seconds while the “legitimate” request is waiting in queue. Users running Jenkins on an intranet have reported this, however, so a malicious attack is probably not the cause.
          Jesse Glick made changes -
          Attachment New: x.diff [ 23155 ]

            jglick Jesse Glick
            maxfields2000 Maxfield Stewart
            Votes:
            2 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: