Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-16023

Lazy loading causes massive delays after a period of inactivity when loading dashboard

      After a period of inactivity or when the JVM decides to run GC loading the dashboard can easily take longer than 1-2 minutes. Please see attached stacktrace.

          [JENKINS-16023] Lazy loading causes massive delays after a period of inactivity when loading dashboard

          I suspect this is actually caused by the massive amount of build history we had for some jobs. After enabling fine logging on hudson.model.RunMap it was obvious which jobs were causing Jenkins to hang. I have now configured discard old builds for the jobs, will report back if it helped or not.

          Johno Crawford added a comment - I suspect this is actually caused by the massive amount of build history we had for some jobs. After enabling fine logging on hudson.model.RunMap it was obvious which jobs were causing Jenkins to hang. I have now configured discard old builds for the jobs, will report back if it helped or not.

          Johno Crawford added a comment - - edited

          Purging the build history has helped a little, however when the JVM runs GC Jenkins is forced to reload all the build history again.

          Johno Crawford added a comment - - edited Purging the build history has helped a little, however when the JVM runs GC Jenkins is forced to reload all the build history again.

          Johno Crawford added a comment - - edited

          Looks like hudson.model.Job.getLastFailedBuild(Job.java:780) is the expensive operation here if the job in question has a long build history with no failures as it will load all history or until it finds a failed job, unstable job etc..

          Johno Crawford added a comment - - edited Looks like hudson.model.Job.getLastFailedBuild(Job.java:780) is the expensive operation here if the job in question has a long build history with no failures as it will load all history or until it finds a failed job, unstable job etc..

          <vlatombe> probably now that builds are fetched lazily, it may be worth to store some information at job level like lastSuccessFul/lastUnstableBuild/lastFailedBuild build numbers or timestamps
          [14:26] <Johnou> yes I agree, doesn't make sense to load all jobs until it finds that information
          [14:26] <Johnou> defeats the point of lazy init

          Vincent Latombe added a comment - <vlatombe> probably now that builds are fetched lazily, it may be worth to store some information at job level like lastSuccessFul/lastUnstableBuild/lastFailedBuild build numbers or timestamps [14:26] <Johnou> yes I agree, doesn't make sense to load all jobs until it finds that information [14:26] <Johnou> defeats the point of lazy init

          We are also having some problem with long load times after upgrading to a version with lazy loading. We are using getPreviousSuccessfulBuild quite frequently and we will disable the code that uses this operation and see if it helps.

          We have are running a Jenkins server with close to 1000 different jobs and noticed that we have a lot of hits on the root, that I think will list all builds. After looking at some statistics from JavaMelody we see that we have quite a lot of hits to the root and to cc.xml in the root. Could this force Jenkins to load unnecessary build data to the memory? And if so, is there a way to prevent this?
          Request, % of cumulative time, Hits, Mean time (ms)
          /cc.xml GET 11 79,903 21
          / GET 6 127 7,746 101,459

          Marcus Jacobsson added a comment - We are also having some problem with long load times after upgrading to a version with lazy loading. We are using getPreviousSuccessfulBuild quite frequently and we will disable the code that uses this operation and see if it helps. We have are running a Jenkins server with close to 1000 different jobs and noticed that we have a lot of hits on the root, that I think will list all builds. After looking at some statistics from JavaMelody we see that we have quite a lot of hits to the root and to cc.xml in the root. Could this force Jenkins to load unnecessary build data to the memory? And if so, is there a way to prevent this? Request, % of cumulative time, Hits, Mean time (ms) /cc.xml GET 11 79,903 21 / GET 6 127 7,746 101,459

          Jesse Glick added a comment -

          The implementation of JENKINS-16089 should fix this for the lastSuccessfulBuild permalink (and corresponding view column). I am perplexed by why other Job.getLast*Build methods were not also converted to use Permalink.

          Jesse Glick added a comment - The implementation of JENKINS-16089 should fix this for the lastSuccessfulBuild permalink (and corresponding view column). I am perplexed by why other Job.getLast*Build methods were not also converted to use Permalink .

          Jesse Glick added a comment -

          If my hypothesis is correct, the fix should be easy, but the tricky part is coming up with a proper functional test. Run.onLoad does not call any listeners. I guess you could attach a RunAction to a build when created then call purgeCache on the run map and check that Job.getLastFailedBuild does not load any builds other than the last failed one.

          Jesse Glick added a comment - If my hypothesis is correct, the fix should be easy, but the tricky part is coming up with a proper functional test. Run.onLoad does not call any listeners. I guess you could attach a RunAction to a build when created then call purgeCache on the run map and check that Job.getLastFailedBuild does not load any builds other than the last failed one.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          changelog.html
          core/src/main/java/hudson/model/Job.java
          test/src/test/java/hudson/model/JobTest.java
          http://jenkins-ci.org/commit/jenkins/be5cd4fd169005f50eea8ccefac4b91a9b4c503c
          Log:
          [FIXED JENKINS-16023] Use permalinks for as many Job.getLast*Build methods as possible.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html core/src/main/java/hudson/model/Job.java test/src/test/java/hudson/model/JobTest.java http://jenkins-ci.org/commit/jenkins/be5cd4fd169005f50eea8ccefac4b91a9b4c503c Log: [FIXED JENKINS-16023] Use permalinks for as many Job.getLast*Build methods as possible.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #2563
          [FIXED JENKINS-16023] Use permalinks for as many Job.getLast*Build methods as possible. (Revision be5cd4fd169005f50eea8ccefac4b91a9b4c503c)

          Result = SUCCESS
          Jesse Glick : be5cd4fd169005f50eea8ccefac4b91a9b4c503c
          Files :

          • test/src/test/java/hudson/model/JobTest.java
          • core/src/main/java/hudson/model/Job.java
          • changelog.html

          dogfood added a comment - Integrated in jenkins_main_trunk #2563 [FIXED JENKINS-16023] Use permalinks for as many Job.getLast*Build methods as possible. (Revision be5cd4fd169005f50eea8ccefac4b91a9b4c503c) Result = SUCCESS Jesse Glick : be5cd4fd169005f50eea8ccefac4b91a9b4c503c Files : test/src/test/java/hudson/model/JobTest.java core/src/main/java/hudson/model/Job.java changelog.html

          pancake added a comment -

          Johno, could you please share information on how to get "Local Variable"s included in a stack trace?

          pancake added a comment - Johno , could you please share information on how to get "Local Variable"s included in a stack trace?

          @pancake VisualVM 1.3+ shows local variables in thread dumps.

          Johno Crawford added a comment - @pancake VisualVM 1.3+ shows local variables in thread dumps.

          pancake added a comment -

          @johno oh, indeed "Local Variable"s are shown in "Threads at the heap dump". Thanks!

          pancake added a comment - @johno oh, indeed "Local Variable"s are shown in "Threads at the heap dump". Thanks!

            kohsuke Kohsuke Kawaguchi
            johno Johno Crawford
            Votes:
            3 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated:
              Resolved: