Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-16023

Lazy loading causes massive delays after a period of inactivity when loading dashboard

      After a period of inactivity or when the JVM decides to run GC loading the dashboard can easily take longer than 1-2 minutes. Please see attached stacktrace.

          [JENKINS-16023] Lazy loading causes massive delays after a period of inactivity when loading dashboard

          Johno Crawford added a comment - - edited

          Looks like hudson.model.Job.getLastFailedBuild(Job.java:780) is the expensive operation here if the job in question has a long build history with no failures as it will load all history or until it finds a failed job, unstable job etc..

          Johno Crawford added a comment - - edited Looks like hudson.model.Job.getLastFailedBuild(Job.java:780) is the expensive operation here if the job in question has a long build history with no failures as it will load all history or until it finds a failed job, unstable job etc..

          <vlatombe> probably now that builds are fetched lazily, it may be worth to store some information at job level like lastSuccessFul/lastUnstableBuild/lastFailedBuild build numbers or timestamps
          [14:26] <Johnou> yes I agree, doesn't make sense to load all jobs until it finds that information
          [14:26] <Johnou> defeats the point of lazy init

          Vincent Latombe added a comment - <vlatombe> probably now that builds are fetched lazily, it may be worth to store some information at job level like lastSuccessFul/lastUnstableBuild/lastFailedBuild build numbers or timestamps [14:26] <Johnou> yes I agree, doesn't make sense to load all jobs until it finds that information [14:26] <Johnou> defeats the point of lazy init

          We are also having some problem with long load times after upgrading to a version with lazy loading. We are using getPreviousSuccessfulBuild quite frequently and we will disable the code that uses this operation and see if it helps.

          We have are running a Jenkins server with close to 1000 different jobs and noticed that we have a lot of hits on the root, that I think will list all builds. After looking at some statistics from JavaMelody we see that we have quite a lot of hits to the root and to cc.xml in the root. Could this force Jenkins to load unnecessary build data to the memory? And if so, is there a way to prevent this?
          Request, % of cumulative time, Hits, Mean time (ms)
          /cc.xml GET 11 79,903 21
          / GET 6 127 7,746 101,459

          Marcus Jacobsson added a comment - We are also having some problem with long load times after upgrading to a version with lazy loading. We are using getPreviousSuccessfulBuild quite frequently and we will disable the code that uses this operation and see if it helps. We have are running a Jenkins server with close to 1000 different jobs and noticed that we have a lot of hits on the root, that I think will list all builds. After looking at some statistics from JavaMelody we see that we have quite a lot of hits to the root and to cc.xml in the root. Could this force Jenkins to load unnecessary build data to the memory? And if so, is there a way to prevent this? Request, % of cumulative time, Hits, Mean time (ms) /cc.xml GET 11 79,903 21 / GET 6 127 7,746 101,459

          Jesse Glick added a comment -

          The implementation of JENKINS-16089 should fix this for the lastSuccessfulBuild permalink (and corresponding view column). I am perplexed by why other Job.getLast*Build methods were not also converted to use Permalink.

          Jesse Glick added a comment - The implementation of JENKINS-16089 should fix this for the lastSuccessfulBuild permalink (and corresponding view column). I am perplexed by why other Job.getLast*Build methods were not also converted to use Permalink .

          Jesse Glick added a comment -

          If my hypothesis is correct, the fix should be easy, but the tricky part is coming up with a proper functional test. Run.onLoad does not call any listeners. I guess you could attach a RunAction to a build when created then call purgeCache on the run map and check that Job.getLastFailedBuild does not load any builds other than the last failed one.

          Jesse Glick added a comment - If my hypothesis is correct, the fix should be easy, but the tricky part is coming up with a proper functional test. Run.onLoad does not call any listeners. I guess you could attach a RunAction to a build when created then call purgeCache on the run map and check that Job.getLastFailedBuild does not load any builds other than the last failed one.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          changelog.html
          core/src/main/java/hudson/model/Job.java
          test/src/test/java/hudson/model/JobTest.java
          http://jenkins-ci.org/commit/jenkins/be5cd4fd169005f50eea8ccefac4b91a9b4c503c
          Log:
          [FIXED JENKINS-16023] Use permalinks for as many Job.getLast*Build methods as possible.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html core/src/main/java/hudson/model/Job.java test/src/test/java/hudson/model/JobTest.java http://jenkins-ci.org/commit/jenkins/be5cd4fd169005f50eea8ccefac4b91a9b4c503c Log: [FIXED JENKINS-16023] Use permalinks for as many Job.getLast*Build methods as possible.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #2563
          [FIXED JENKINS-16023] Use permalinks for as many Job.getLast*Build methods as possible. (Revision be5cd4fd169005f50eea8ccefac4b91a9b4c503c)

          Result = SUCCESS
          Jesse Glick : be5cd4fd169005f50eea8ccefac4b91a9b4c503c
          Files :

          • test/src/test/java/hudson/model/JobTest.java
          • core/src/main/java/hudson/model/Job.java
          • changelog.html

          dogfood added a comment - Integrated in jenkins_main_trunk #2563 [FIXED JENKINS-16023] Use permalinks for as many Job.getLast*Build methods as possible. (Revision be5cd4fd169005f50eea8ccefac4b91a9b4c503c) Result = SUCCESS Jesse Glick : be5cd4fd169005f50eea8ccefac4b91a9b4c503c Files : test/src/test/java/hudson/model/JobTest.java core/src/main/java/hudson/model/Job.java changelog.html

          pancake added a comment -

          Johno, could you please share information on how to get "Local Variable"s included in a stack trace?

          pancake added a comment - Johno , could you please share information on how to get "Local Variable"s included in a stack trace?

          @pancake VisualVM 1.3+ shows local variables in thread dumps.

          Johno Crawford added a comment - @pancake VisualVM 1.3+ shows local variables in thread dumps.

          pancake added a comment -

          @johno oh, indeed "Local Variable"s are shown in "Threads at the heap dump". Thanks!

          pancake added a comment - @johno oh, indeed "Local Variable"s are shown in "Threads at the heap dump". Thanks!

            kohsuke Kohsuke Kawaguchi
            johno Johno Crawford
            Votes:
            3 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated:
              Resolved: