I'm looking for guidance on debugging slow page load times. Ever since we upgraded from 1.480 to 1.484 (and other versions beyond that) we've had very slow page load times. Sometimes upwards of 120 seconds to load a page, nearly any view tab, job configuration etc.

      I've debugged many Java apps and I'm confounded. There are few, if any, Full GC's occuring, regular GC's occur somewhat frequently but take .02 or less seconds. CPU utilization for the java executeable is under %6 and and disk utilization is reasonable (we're averaging a %90+ idle time). The initial HTTP ack comes back immediately but then we wait for the page response to come back for up to 2 minutes sometimes. Othertimes it loads reasonably fast (under 4 seconds). When it's hanging, it's hanging for all requests, much like it was doing a full garbage collect. It feels like some kind of resource contention.

      But I cannot find where the contention is, thread dumps look nominal (I've included one here for example). Something causes long page load times, we back revved to 1.484 thinking it had something to do with lazy loading features but clearly it does not. On 1.480 we did not have these issues. I could use some help figuring out what else to look at to identify why Jenkins is slow. There is a lack of information available on common reasons for slow page load times, this results in a terrible user experience with an otherwise fine tool.

        1. threaddump.txt
          106 kB
        2. x.diff
          2 kB

          [JENKINS-16474] Slow/hung web UI in 1.483+ (stuck in parseURI)

          Code changed in jenkins
          User: Jesse Glick
          Path:
          src/java/winstone/cmdline/Option.java
          http://jenkins-ci.org/commit/winstone/b2c2bff8d1e03c5d5fd46843b22d9126fd5e9c81
          Log:
          JENKINS-16474 Make --http

          {,s}

          KeepAliveTimeout actually work.

          Compare: https://github.com/jenkinsci/winstone/compare/78a00ae807d9...b2c2bff8d1e0


          You received this message because you are subscribed to the Google Groups "Jenkins Commits" group.
          To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com.
          For more options, visit https://groups.google.com/groups/opt_out.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/java/winstone/cmdline/Option.java http://jenkins-ci.org/commit/winstone/b2c2bff8d1e03c5d5fd46843b22d9126fd5e9c81 Log: JENKINS-16474 Make --http {,s} KeepAliveTimeout actually work. Compare: https://github.com/jenkinsci/winstone/compare/78a00ae807d9...b2c2bff8d1e0 – You received this message because you are subscribed to the Google Groups "Jenkins Commits" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out .

          Code changed in jenkins
          User: Jesse Glick
          Path:
          changelog.html
          war/pom.xml
          http://jenkins-ci.org/commit/jenkins/fa6c9ba35de4ec98f5487f296bab7ee1b7fb2da5
          Log:
          [FIXED JENKINS-16474] Winstone and executable WAR upgraded to actually support --httpKeepAliveTimeout.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html war/pom.xml http://jenkins-ci.org/commit/jenkins/fa6c9ba35de4ec98f5487f296bab7ee1b7fb2da5 Log: [FIXED JENKINS-16474] Winstone and executable WAR upgraded to actually support --httpKeepAliveTimeout.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #2350
          [FIXED JENKINS-16474] Winstone and executable WAR upgraded to actually support --httpKeepAliveTimeout. (Revision fa6c9ba35de4ec98f5487f296bab7ee1b7fb2da5)

          Result = SUCCESS
          Jesse Glick : fa6c9ba35de4ec98f5487f296bab7ee1b7fb2da5
          Files :

          • war/pom.xml
          • changelog.html

          dogfood added a comment - Integrated in jenkins_main_trunk #2350 [FIXED JENKINS-16474] Winstone and executable WAR upgraded to actually support --httpKeepAliveTimeout. (Revision fa6c9ba35de4ec98f5487f296bab7ee1b7fb2da5) Result = SUCCESS Jesse Glick : fa6c9ba35de4ec98f5487f296bab7ee1b7fb2da5 Files : war/pom.xml changelog.html

          jfigler added a comment -

          Thanks Jesse, I'm unable to reproduce it as well.

          Can anyone confirm if they were experiencing this issue, and now with this partial (or full) fix, they're not?

          jfigler added a comment - Thanks Jesse, I'm unable to reproduce it as well. Can anyone confirm if they were experiencing this issue, and now with this partial (or full) fix, they're not?

          @jfigler: I just was pointed to this JIRA issue: we're running 1.480.1, experiencing very slow network connections to the Jenkins UI when a non-trivial number of users is using Jenkins, and until now (running for an afternoon) it looks like the Groovy script from this earlier comment from Feb 7 makes the problem go away.

          @jglick: Which LTS version will contain this fix? And is there a way to change the httpKeepAliveTimeout for Winstone more permanently, in 1.480.1?

          Thanks for your work on this!

          Marnix Klooster added a comment - @ jfigler : I just was pointed to this JIRA issue: we're running 1.480.1, experiencing very slow network connections to the Jenkins UI when a non-trivial number of users is using Jenkins, and until now (running for an afternoon) it looks like the Groovy script from this earlier comment from Feb 7 makes the problem go away. @ jglick : Which LTS version will contain this fix? And is there a way to change the httpKeepAliveTimeout for Winstone more permanently, in 1.480.1? Thanks for your work on this!

          Jesse Glick added a comment -

          @marnix_klooster: the latest fix is in 1.506 and therefore also 1.509.1 LTS; --httpKeepAliveTimeout may be passed on the command line (see --help for options).

          Jesse Glick added a comment - @marnix_klooster: the latest fix is in 1.506 and therefore also 1.509.1 LTS; --httpKeepAliveTimeout may be passed on the command line (see --help for options).

          Jesse Glick added a comment -

          If anyone encountering this issue is using an Apache proxy, JENKINS-10524 suggests Proxy-nokeepalive, though that is probably unrelated.

          Jesse Glick added a comment - If anyone encountering this issue is using an Apache proxy, JENKINS-10524 suggests Proxy-nokeepalive , though that is probably unrelated.

          Wael Darwich added a comment -

          Hi,
          I believe I am getting similar behaviour, using Jenkins 1.512
          Loading pages is ok sometimes, but it is really slow in other occasions! sometimes several minutes to load few pages only, or sometimes loading any page!

          Wael Darwich added a comment - Hi, I believe I am getting similar behaviour, using Jenkins 1.512 Loading pages is ok sometimes, but it is really slow in other occasions! sometimes several minutes to load few pages only, or sometimes loading any page!

          Jesse Glick added a comment -

          @wael your issue may not have anything to do with this fix. Please use a separate ticket (using JIRA links as needed) unless you can confirm at the code level that this fix did not do what it claimed to do.

          Also please try the new command-line tuning parameters introduced by the fix (--help for details), and if a thread dump confirms that many threads are still stuck in parseURI, attach that thread dump since it will now contain more information.

          Jesse Glick added a comment - @wael your issue may not have anything to do with this fix. Please use a separate ticket (using JIRA links as needed) unless you can confirm at the code level that this fix did not do what it claimed to do. Also please try the new command-line tuning parameters introduced by the fix ( --help for details), and if a thread dump confirms that many threads are still stuck in parseURI , attach that thread dump since it will now contain more information.

          Dave Taddei added a comment -

          We have found if the jenkins master contains a large number of jobs and the user is only allowed to access a subset then the UI takes a long time to load. If the user is granted read access to everything then it loads very quickly. We suspect (though cannot confirm) that the matrix authorization plugin is not working as efficiently as it could.

          We believe the following is happening:
          1. plugin is checking every job against every LDAP group the user is a member of
          2. plugin is then checking every job against the specific LDAP user
          3. it is possible the plugin is making individual calls to an LDAP server for each job

          Dave Taddei added a comment - We have found if the jenkins master contains a large number of jobs and the user is only allowed to access a subset then the UI takes a long time to load. If the user is granted read access to everything then it loads very quickly. We suspect (though cannot confirm) that the matrix authorization plugin is not working as efficiently as it could. We believe the following is happening: 1. plugin is checking every job against every LDAP group the user is a member of 2. plugin is then checking every job against the specific LDAP user 3. it is possible the plugin is making individual calls to an LDAP server for each job

            jglick Jesse Glick
            maxfields2000 Maxfield Stewart
            Votes:
            2 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: