Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-16089

Standard view columns interact force builds to be eagerly loaded

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      LastFailureColumn, shown by default in AllView, is calling Job.getLastFailedBuild which then keeps on loading builds until it finds a failed build, with no cutoff. The fix for that would be to either impose a maximum number of builds to search back from (10? 20?); or do the search asynchronously after the page loads, which might still put some load on the server but would at least not stop you from browsing.

      The same would apply to the last success column if all your recent builds failed, etc.

        Attachments

          Issue Links

            Activity

            jglick Jesse Glick created issue -
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Link This issue is blocking JENKINS-8754 [ JENKINS-8754 ]
            Hide
            jsirex jsirex added a comment -

            In fs jenkins have some symlinks like 'lastStable' or 'lastSuccessful'. May be good point is to create other 'last<Failure|Unstable|whatever' and use these links when load jobs in view.

            Also build.xml is to heavy to parse it only for few columns displayed on page. Possibly creating an light json file with only required info also improves performance.

            Show
            jsirex jsirex added a comment - In fs jenkins have some symlinks like 'lastStable' or 'lastSuccessful'. May be good point is to create other 'last<Failure|Unstable|whatever' and use these links when load jobs in view. Also build.xml is to heavy to parse it only for few columns displayed on page. Possibly creating an light json file with only required info also improves performance.
            Hide
            justinsb Justin Santa Barbara added a comment -

            I hit this issue myself. I whipped up a quick proof-of-concept of using caching to fix:
            https://github.com/justinsb/jenkins/commit/d539c6680ea1fb14f45c63ec21ecf8df046f3352

            I debated caching the jobs themselves, but they're not Serializable , so that would eliminate many of the more advanced caching solutions.

            Instead, I just memoize the function results, recording against each (completed) build the build number of e.g. lastFailure. To retrieve, we walk the build chain as normal, but we can short-circuit as soon as we come across a memoized result. When we find a result, we cache it against the most recent completed build, it isn't already there. This relies on an LRU cache to expire old entries.

            Currently this is only in-memory, but if we agree that caching is a good answer, I think we'd want to implement a persistent cache so that this survives across restarts. I suspect this memoization approach could well prove helpful for performance in many places in Jenkins.

            I know the patch is not ready to merge . If someone wants to take the idea and run with it, feel free; otherwise, comments/criticism welcome. If this does have broader applicability, should I post to the Jenkins-dev mail-list?

            Show
            justinsb Justin Santa Barbara added a comment - I hit this issue myself. I whipped up a quick proof-of-concept of using caching to fix: https://github.com/justinsb/jenkins/commit/d539c6680ea1fb14f45c63ec21ecf8df046f3352 I debated caching the jobs themselves, but they're not Serializable , so that would eliminate many of the more advanced caching solutions. Instead, I just memoize the function results, recording against each (completed) build the build number of e.g. lastFailure. To retrieve, we walk the build chain as normal, but we can short-circuit as soon as we come across a memoized result. When we find a result, we cache it against the most recent completed build, it isn't already there. This relies on an LRU cache to expire old entries. Currently this is only in-memory, but if we agree that caching is a good answer, I think we'd want to implement a persistent cache so that this survives across restarts. I suspect this memoization approach could well prove helpful for performance in many places in Jenkins. I know the patch is not ready to merge . If someone wants to take the idea and run with it, feel free; otherwise, comments/criticism welcome. If this does have broader applicability, should I post to the Jenkins-dev mail-list?
            Hide
            justinsb Justin Santa Barbara added a comment -

            I did a little bit more experimentation here, using it to fix the "slow People view" problem that was mentioned on the mailing list:

            I now persist the cache (not a true LRU persistent cache, but I think it's a good start without external dependencies); I've switched to requiring cache objects to be Serializable:
            https://github.com/justinsb/jenkins/commit/a1c68a1156be0dfa1b6c95578cd0675fa50c8fea

            I use the cache to store People, using the same memoize-per-build and incremental update approach:
            https://github.com/justinsb/jenkins/commit/a0ae671785ae96e1c85274332d54effe80bc875d

            So as to avoid behavioural changes, rather than trying to persist the UserInfo object, instead it stores a reference to the build from which the UserInfo should be loaded. That build still needs to be loaded, but it's only the required builds. If we could persist the UserInfo safely, we could avoid this as well.

            Show
            justinsb Justin Santa Barbara added a comment - I did a little bit more experimentation here, using it to fix the "slow People view" problem that was mentioned on the mailing list: I now persist the cache (not a true LRU persistent cache, but I think it's a good start without external dependencies); I've switched to requiring cache objects to be Serializable: https://github.com/justinsb/jenkins/commit/a1c68a1156be0dfa1b6c95578cd0675fa50c8fea I use the cache to store People, using the same memoize-per-build and incremental update approach: https://github.com/justinsb/jenkins/commit/a0ae671785ae96e1c85274332d54effe80bc875d So as to avoid behavioural changes, rather than trying to persist the UserInfo object, instead it stores a reference to the build from which the UserInfo should be loaded. That build still needs to be loaded, but it's only the required builds. If we could persist the UserInfo safely, we could avoid this as well.
            jglick Jesse Glick made changes -
            Assignee Jesse Glick [ jglick ] Justin Santa Barbara [ justinsb ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            changelog.html
            core/src/main/java/hudson/Util.java
            core/src/main/java/hudson/model/Job.java
            core/src/main/java/hudson/model/PermalinkProjectAction.java
            core/src/main/java/hudson/model/Result.java
            core/src/main/java/jenkins/model/PeepholePermalink.java
            http://jenkins-ci.org/commit/jenkins/88feabb4296d6483bdb5c85fed489e5014161ca5
            Log:
            [FIXED JENKINS-16089]

            Remember the permalink target as symlink (or simple text file) so that
            looking that up doesn't cause the walk of the build history.
            I think this is more in line with our general preference of making
            $JENKINS_HOME useful (than trying to persist cache into a blackbox.)

            Having a general purpose in-memory cache could be useful, so I'll see if
            I can add that, too, in a way that allows someone to plug different
            backend.

            Compare: https://github.com/jenkinsci/jenkins/compare/c596ed517c3f...88feabb4296d


            You received this message because you are subscribed to the Google Groups "Jenkins Commits" group.
            To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com.
            For more options, visit https://groups.google.com/groups/opt_out.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/src/main/java/hudson/Util.java core/src/main/java/hudson/model/Job.java core/src/main/java/hudson/model/PermalinkProjectAction.java core/src/main/java/hudson/model/Result.java core/src/main/java/jenkins/model/PeepholePermalink.java http://jenkins-ci.org/commit/jenkins/88feabb4296d6483bdb5c85fed489e5014161ca5 Log: [FIXED JENKINS-16089] Remember the permalink target as symlink (or simple text file) so that looking that up doesn't cause the walk of the build history. I think this is more in line with our general preference of making $JENKINS_HOME useful (than trying to persist cache into a blackbox.) Having a general purpose in-memory cache could be useful, so I'll see if I can add that, too, in a way that allows someone to plug different backend. Compare: https://github.com/jenkinsci/jenkins/compare/c596ed517c3f...88feabb4296d – You received this message because you are subscribed to the Google Groups "Jenkins Commits" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out .
            scm_issue_link SCM/JIRA link daemon made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #2363
            [FIXED JENKINS-16089] (Revision 88feabb4296d6483bdb5c85fed489e5014161ca5)

            Result = UNSTABLE
            kohsuke : 88feabb4296d6483bdb5c85fed489e5014161ca5
            Files :

            • core/src/main/java/jenkins/model/PeepholePermalink.java
            • core/src/main/java/hudson/Util.java
            • core/src/main/java/hudson/model/Result.java
            • core/src/main/java/hudson/model/PermalinkProjectAction.java
            • core/src/main/java/hudson/model/Job.java
            • changelog.html
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #2363 [FIXED JENKINS-16089] (Revision 88feabb4296d6483bdb5c85fed489e5014161ca5) Result = UNSTABLE kohsuke : 88feabb4296d6483bdb5c85fed489e5014161ca5 Files : core/src/main/java/jenkins/model/PeepholePermalink.java core/src/main/java/hudson/Util.java core/src/main/java/hudson/model/Result.java core/src/main/java/hudson/model/PermalinkProjectAction.java core/src/main/java/hudson/model/Job.java changelog.html
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            core/src/main/java/hudson/model/AbstractBuild.java
            core/src/main/java/hudson/model/Job.java
            core/src/main/java/jenkins/model/PeepholePermalink.java
            core/src/test/java/hudson/model/StubJob.java
            test/src/test/groovy/jenkins/model/PeepholePermalinkTest.groovy
            http://jenkins-ci.org/commit/jenkins/f7c9e810605e89b4f8d841cfc2df2342037982f1
            Log:
            JENKINS-16089 Revising 88feabb4296d6483bdb5c85fed489e5014161ca5

            Based on comments from Jesse, revising the fix.

            I'm now putting permlinks inside the builds/ directory to avoid the computing hassle involved in the split $JENKINS_HOME.

            What we historically had in $JENKINS_HOME/jobs/JOB/lastSuccessfulBuild is also now subsumed by this feature. I initially attempted to create these permalinks in the buidl root directory, but turns out those symlinks aren't the same name as the ID of permalinks, so it doesn't mesh well.

            And finally, a test!

            Compare: https://github.com/jenkinsci/jenkins/compare/88feabb4296d...f7c9e810605e


            You received this message because you are subscribed to the Google Groups "Jenkins Commits" group.
            To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com.
            For more options, visit https://groups.google.com/groups/opt_out.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/hudson/model/AbstractBuild.java core/src/main/java/hudson/model/Job.java core/src/main/java/jenkins/model/PeepholePermalink.java core/src/test/java/hudson/model/StubJob.java test/src/test/groovy/jenkins/model/PeepholePermalinkTest.groovy http://jenkins-ci.org/commit/jenkins/f7c9e810605e89b4f8d841cfc2df2342037982f1 Log: JENKINS-16089 Revising 88feabb4296d6483bdb5c85fed489e5014161ca5 Based on comments from Jesse, revising the fix. I'm now putting permlinks inside the builds/ directory to avoid the computing hassle involved in the split $JENKINS_HOME. What we historically had in $JENKINS_HOME/jobs/JOB/lastSuccessfulBuild is also now subsumed by this feature. I initially attempted to create these permalinks in the buidl root directory, but turns out those symlinks aren't the same name as the ID of permalinks, so it doesn't mesh well. And finally, a test! Compare: https://github.com/jenkinsci/jenkins/compare/88feabb4296d...f7c9e810605e – You received this message because you are subscribed to the Google Groups "Jenkins Commits" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out .
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #2364
            JENKINS-16089 Revising 88feabb4296d6483bdb5c85fed489e5014161ca5 (Revision f7c9e810605e89b4f8d841cfc2df2342037982f1)

            Result = UNSTABLE
            kohsuke : f7c9e810605e89b4f8d841cfc2df2342037982f1
            Files :

            • test/src/test/groovy/jenkins/model/PeepholePermalinkTest.groovy
            • core/src/main/java/hudson/model/AbstractBuild.java
            • core/src/main/java/hudson/model/Job.java
            • core/src/main/java/jenkins/model/PeepholePermalink.java
            • core/src/test/java/hudson/model/StubJob.java
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #2364 JENKINS-16089 Revising 88feabb4296d6483bdb5c85fed489e5014161ca5 (Revision f7c9e810605e89b4f8d841cfc2df2342037982f1) Result = UNSTABLE kohsuke : f7c9e810605e89b4f8d841cfc2df2342037982f1 Files : test/src/test/groovy/jenkins/model/PeepholePermalinkTest.groovy core/src/main/java/hudson/model/AbstractBuild.java core/src/main/java/hudson/model/Job.java core/src/main/java/jenkins/model/PeepholePermalink.java core/src/test/java/hudson/model/StubJob.java
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            core/src/main/java/hudson/model/AbstractBuild.java
            test/src/test/java/hudson/model/AbstractProjectTest.java
            http://jenkins-ci.org/commit/jenkins/4d0def46ccf9e97074b3f4b4e66ebfe9ead9b74b
            Log:
            JENKINS-16089 needs a separator here

            And since it's now a symlink that points to another symlink that then
            points to the real stuff, we need recursive onion peeling.

            Compare: https://github.com/jenkinsci/jenkins/compare/2aa0293858d5...4d0def46ccf9


            You received this message because you are subscribed to the Google Groups "Jenkins Commits" group.
            To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com.
            For more options, visit https://groups.google.com/groups/opt_out.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/hudson/model/AbstractBuild.java test/src/test/java/hudson/model/AbstractProjectTest.java http://jenkins-ci.org/commit/jenkins/4d0def46ccf9e97074b3f4b4e66ebfe9ead9b74b Log: JENKINS-16089 needs a separator here And since it's now a symlink that points to another symlink that then points to the real stuff, we need recursive onion peeling. Compare: https://github.com/jenkinsci/jenkins/compare/2aa0293858d5...4d0def46ccf9 – You received this message because you are subscribed to the Google Groups "Jenkins Commits" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out .
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #2370
            JENKINS-16089 needs a separator here (Revision 4d0def46ccf9e97074b3f4b4e66ebfe9ead9b74b)

            Result = SUCCESS
            kohsuke : 4d0def46ccf9e97074b3f4b4e66ebfe9ead9b74b
            Files :

            • core/src/main/java/hudson/model/AbstractBuild.java
            • test/src/test/java/hudson/model/AbstractProjectTest.java
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #2370 JENKINS-16089 needs a separator here (Revision 4d0def46ccf9e97074b3f4b4e66ebfe9ead9b74b) Result = SUCCESS kohsuke : 4d0def46ccf9e97074b3f4b4e66ebfe9ead9b74b Files : core/src/main/java/hudson/model/AbstractBuild.java test/src/test/java/hudson/model/AbstractProjectTest.java
            kutzi kutzi made changes -
            Link This issue is related to JENKINS-17449 [ JENKINS-17449 ]
            martinkutter Martin Kutter made changes -
            Labels performance lts-candidate performance
            Hide
            martinkutter Martin Kutter added a comment -

            Added lts-candidate label, as this item mainly affects bigger installations (with many jobs in a view)

            Show
            martinkutter Martin Kutter added a comment - Added lts-candidate label, as this item mainly affects bigger installations (with many jobs in a view)
            Hide
            jglick Jesse Glick added a comment -

            @martinkutter the fix was already put in 1.507 so lts-candidate is irrelevant now.

            Show
            jglick Jesse Glick added a comment - @martinkutter the fix was already put in 1.507 so lts-candidate is irrelevant now.
            martinkutter Martin Kutter made changes -
            Labels lts-candidate performance performance
            Hide
            martinkutter Martin Kutter added a comment -

            @JesseGlick: Sorry, you're right - label removed

            Show
            martinkutter Martin Kutter added a comment - @JesseGlick: Sorry, you're right - label removed
            jglick Jesse Glick made changes -
            Link This issue is blocking JENKINS-16023 [ JENKINS-16023 ]
            jglick Jesse Glick made changes -
            Link This issue depends on JENKINS-18846 [ JENKINS-18846 ]
            jglick Jesse Glick made changes -
            Labels performance lazy-loading performance
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 146893 ] JNJira + In-Review [ 192130 ]

              People

              Assignee:
              justinsb Justin Santa Barbara
              Reporter:
              jglick Jesse Glick
              Votes:
              5 Vote for this issue
              Watchers:
              12 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: