• Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Critical Critical
    • core
    • Jenkins 1.638
      Chrome
      Java 7
      Windows Server 2008

      We are seeing sporadic slowdowns on our Jenkins server when opening certain jobs containing a lot of history. I believe it is something to do with the build history widget. There are probably >300+ builds in history.

      The symptom is that clicking on the job page link from the main jenkins page takes forever to load (~5 min). Below is the stacktrace taken from one of the jenkins server request threads while experiencing the issue:

      Name: Handling GET /job/nxn-frontend.feature-branch/ from 10.15.130.111 : RequestHandlerThread[#88] Job/index.jelly AbstractProject/sidepanel.jelly HistoryWidget/index.jelly
      State: RUNNABLE
      Total blocked: 1,259  Total waited: 140
      
      Stack trace: 
      com.thoughtworks.xstream.converters.reflection.FieldDictionary.buildMap(FieldDictionary.java:122)
         - locked com.thoughtworks.xstream.converters.reflection.FieldDictionary@627d3b18
      com.thoughtworks.xstream.converters.reflection.FieldDictionary.fieldOrNull(FieldDictionary.java:113)
      com.thoughtworks.xstream.converters.reflection.FieldDictionary.field(FieldDictionary.java:92)
      com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.getFieldType(PureJavaReflectionProvider.java:160)
      hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:292)
      hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:229)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:352)
      hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:290)
      hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:229)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
      hudson.util.RobustMapConverter.read(RobustMapConverter.java:58)
      hudson.util.RobustMapConverter.putCurrentEntryIntoMap(RobustMapConverter.java:49)
      com.thoughtworks.xstream.converters.collections.MapConverter.populateMap(MapConverter.java:98)
      com.thoughtworks.xstream.converters.collections.MapConverter.populateMap(MapConverter.java:92)
      com.thoughtworks.xstream.converters.collections.MapConverter.unmarshal(MapConverter.java:87)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:352)
      hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:290)
      hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:229)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
      hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:85)
      com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:80)
      hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:76)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:352)
      hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:290)
      hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:229)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
      com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
      com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
      hudson.util.XStream2.unmarshal(XStream2.java:113)
      com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
      hudson.XmlFile.unmarshal(XmlFile.java:163)
      hudson.model.Run.reload(Run.java:326)
      hudson.model.Run.<init>(Run.java:314)
      hudson.model.AbstractBuild.<init>(AbstractBuild.java:175)
      hudson.model.Build.<init>(Build.java:103)
      com.tikal.jenkins.plugins.multijob.MultiJobBuild.<init>(MultiJobBuild.java:56)
      sun.reflect.GeneratedConstructorAccessor160.newInstance(Unknown Source)
      sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
      java.lang.reflect.Constructor.newInstance(Unknown Source)
      jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:156)
      jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:133)
      hudson.model.RunMap.retrieve(RunMap.java:223)
      hudson.model.RunMap.retrieve(RunMap.java:56)
      jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:465)
         - locked hudson.model.RunMap@56d45de5
      jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:448)
      jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:356)
      jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
      jenkins.model.lazy.LazyBuildMixIn$RunMixIn.getPreviousBuild(LazyBuildMixIn.java:355)
      hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:196)
      hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:104)
      hudson.model.RunMap$1.next(RunMap.java:109)
      hudson.model.RunMap$1.next(RunMap.java:98)
      org.apache.commons.collections.IteratorUtils.toList(IteratorUtils.java:849)
      org.apache.commons.collections.IteratorUtils.toList(IteratorUtils.java:825)
      hudson.widgets.BuildHistoryWidget.getHistoryPageFilter(BuildHistoryWidget.java:81)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      java.lang.reflect.Method.invoke(Unknown Source)
      org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
      org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
      org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
      org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
      org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
      org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
      org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
      org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
      hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
      org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)
      org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
      org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
      org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
      org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
      org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
      org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)
      org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
      org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
      org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
      org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
      org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
      org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112)
      org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:127)
      org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:735)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
      org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
      org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
      hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:123)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
      org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
      org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
      org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
      org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
      org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
      org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
      org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
      org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
      org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
      org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
      org.eclipse.jetty.server.Server.handle(Server.java:370)
      org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
      org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
      org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
      org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
      org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
      org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
      org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
      org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
      winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      java.lang.Thread.run(Unknown Source)
      
      

      Basically a big long journey is starting here:
      org.apache.commons.collections.IteratorUtils.toList(IteratorUtils.java:825)
      hudson.widgets.BuildHistoryWidget.getHistoryPageFilter(BuildHistoryWidget.java:81)

      that results in what I'm assuming is every single one of those 300 jobs getting init-ed and their XML configurations parsed. Our job config.xml is also quite large. I can consistently reproduce this by clicking back and forth between two jobs with a lot of history.

          [JENKINS-31888] BuildHistoryWidget causing job page slowdowns

          greg mallett added a comment -

          On 2nd inspection, this could be the MultiJob plugin. This particular job is a multijob with 10 sub jobs. I think it is loading each sub job for every job in the history.

          greg mallett added a comment - On 2nd inspection, this could be the MultiJob plugin. This particular job is a multijob with 10 sub jobs. I think it is loading each sub job for every job in the history.

          Josh Kelley added a comment -

          I'm seeing what appears to be the same or very similar issue: the main page loads fine, but clicking a job page takes a few minutes (if the disk cache is cold). strace shows that Jenkins is reading numerous previous builds' build.xml, and jstack shows the following:

                  at java.lang.String.intern(Native Method)
                  at org.xmlpull.mxp1.MXParserCachingStrings.newStringIntern(MXParserCachingStrings.java:317)
                  at org.xmlpull.mxp1.MXParserCachingStrings.newString(MXParserCachingStrings.java:287)
                  at org.xmlpull.mxp1.MXParser.parseStartTag(MXParser.java:1779)
                  at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1127)
                  at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093)
                  at com.thoughtworks.xstream.io.xml.XppReader.pullNextEvent(XppReader.java:109)
                  at com.thoughtworks.xstream.io.xml.AbstractPullReader.readRealEvent(AbstractPullReader.java:148)
                  at com.thoughtworks.xstream.io.xml.AbstractPullReader.readEvent(AbstractPullReader.java:135)
                  at com.thoughtworks.xstream.io.xml.AbstractPullReader.hasMoreChildren(AbstractPullReader.java:87)
                  at com.thoughtworks.xstream.io.ReaderWrapper.hasMoreChildren(ReaderWrapper.java:32)
                  at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:268)
                  at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:229)
                  at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
                  at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
                  at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
                  at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
                  at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
                  at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
                  at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
                  at hudson.util.XStream2.unmarshal(XStream2.java:113)
                  at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
                  at hudson.XmlFile.unmarshal(XmlFile.java:163)
                  at hudson.model.Run.reload(Run.java:326)
                  at hudson.model.Run.<init>(Run.java:314)
                  at hudson.model.AbstractBuild.<init>(AbstractBuild.java:175)
                  at hudson.model.Build.<init>(Build.java:103)
                  at hudson.model.FreeStyleBuild.<init>(FreeStyleBuild.java:38)
                  at sun.reflect.GeneratedConstructorAccessor57.newInstance(Unknown Source)
                  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                  at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
                  at jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:156)
                  at jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:133)
                  at hudson.model.RunMap.retrieve(RunMap.java:223)
                  at hudson.model.RunMap.retrieve(RunMap.java:56)
                  at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:465)
                  - locked <0x7461a168> (a hudson.model.RunMap)
                  at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:448)
                  at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:356)
                  at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
                  at jenkins.model.lazy.LazyBuildMixIn$RunMixIn.getPreviousBuild(LazyBuildMixIn.java:355)
                  at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:196)
                  at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:104)
                  at hudson.model.RunMap$1.next(RunMap.java:109)
                  at hudson.model.RunMap$1.next(RunMap.java:98)
                  at org.apache.commons.collections.IteratorUtils.toList(IteratorUtils.java:849)
                  at org.apache.commons.collections.IteratorUtils.toList(IteratorUtils.java:825)
                  at hudson.widgets.BuildHistoryWidget.getHistoryPageFilter(BuildHistoryWidget.java:81)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                  at java.lang.reflect.Method.invoke(Method.java:606)
                  at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
                  at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
                  at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
                  at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
                  at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
                  at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
                  at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
                  at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
                  at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
                  at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
                  at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                  at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
                  at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
                  at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)
                  at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                  at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
                  at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
                  at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                  at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                  at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
                  at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
                  at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                  at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
                  at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
                  at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
                  at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                  at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
                  at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
                  at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)
                  at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                  at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                  at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
                  at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
                  at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                  at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                  at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
                  at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                  at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
                  at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                  at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
                  at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                  at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
                  at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                  at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
                  at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
                  at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
                  at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
                  at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
                  at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
                  at org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112)
                  at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:127)
                  at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:735)
                  at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
                  at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
                  at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
                  at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
                  at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
                  at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
                  at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
                  at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
                  at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
                  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
                  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
                  at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:123)
                  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
                  at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:92)
                  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
                  at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                  at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                  at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                  at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                  at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                  at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                  at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
                  at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                  at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
                  at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
                  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
                  at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
                  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
                  at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
                  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
                  at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
                  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
                  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
                  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
                  at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
                  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
                  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
                  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
                  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
                  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
                  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
                  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
                  at org.eclipse.jetty.server.Server.handle(Server.java:370)
                  at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
                  at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
                  at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
                  at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
                  at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
                  at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
                  at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
                  at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
                  at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                  at java.lang.Thread.run(Thread.java:745)
          

          Note that MultiJob is not involved in my case.

          Josh Kelley added a comment - I'm seeing what appears to be the same or very similar issue: the main page loads fine, but clicking a job page takes a few minutes (if the disk cache is cold). strace shows that Jenkins is reading numerous previous builds' build.xml , and jstack shows the following: at java.lang. String .intern(Native Method) at org.xmlpull.mxp1.MXParserCachingStrings.newStringIntern(MXParserCachingStrings.java:317) at org.xmlpull.mxp1.MXParserCachingStrings.newString(MXParserCachingStrings.java:287) at org.xmlpull.mxp1.MXParser.parseStartTag(MXParser.java:1779) at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1127) at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093) at com.thoughtworks.xstream.io.xml.XppReader.pullNextEvent(XppReader.java:109) at com.thoughtworks.xstream.io.xml.AbstractPullReader.readRealEvent(AbstractPullReader.java:148) at com.thoughtworks.xstream.io.xml.AbstractPullReader.readEvent(AbstractPullReader.java:135) at com.thoughtworks.xstream.io.xml.AbstractPullReader.hasMoreChildren(AbstractPullReader.java:87) at com.thoughtworks.xstream.io.ReaderWrapper.hasMoreChildren(ReaderWrapper.java:32) at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:268) at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:229) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50) at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189) at hudson.util.XStream2.unmarshal(XStream2.java:113) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173) at hudson.XmlFile.unmarshal(XmlFile.java:163) at hudson.model.Run.reload(Run.java:326) at hudson.model.Run.<init>(Run.java:314) at hudson.model.AbstractBuild.<init>(AbstractBuild.java:175) at hudson.model.Build.<init>(Build.java:103) at hudson.model.FreeStyleBuild.<init>(FreeStyleBuild.java:38) at sun.reflect.GeneratedConstructorAccessor57.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:156) at jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:133) at hudson.model.RunMap.retrieve(RunMap.java:223) at hudson.model.RunMap.retrieve(RunMap.java:56) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:465) - locked <0x7461a168> (a hudson.model.RunMap) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:448) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:356) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332) at jenkins.model.lazy.LazyBuildMixIn$RunMixIn.getPreviousBuild(LazyBuildMixIn.java:355) at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:196) at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:104) at hudson.model.RunMap$1.next(RunMap.java:109) at hudson.model.RunMap$1.next(RunMap.java:98) at org.apache.commons.collections.IteratorUtils.toList(IteratorUtils.java:849) at org.apache.commons.collections.IteratorUtils.toList(IteratorUtils.java:825) at hudson.widgets.BuildHistoryWidget.getHistoryPageFilter(BuildHistoryWidget.java:81) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125) at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314) at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185) at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75) at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83) at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57) at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51) at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80) at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74) at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53) at org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112) at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:127) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:735) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:123) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:92) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) Note that MultiJob is not involved in my case.

          Andrew Lee added a comment -

          I'm experiencing this issue as well on 1.642.1. My stack trace is almost exactly the same; we are not using the MultiJob plugin. We've had performance issues due to large build history in the past so we automatically delete old jobs once they are more than 1000 builds old, but we'd prefer not to have to trim the history down further than that.

          Once the page finally loads (takes around 2-3 minutes) it remains fast for a while, but eventually it triggers another reload of history from disk and takes 2-3 minutes again.

          Our build.xml files are about 15 KB each and our junitResult.xml files are about 6 MB each (not sure if it's actually reading the junitResult file or not).

          Andrew Lee added a comment - I'm experiencing this issue as well on 1.642.1. My stack trace is almost exactly the same; we are not using the MultiJob plugin. We've had performance issues due to large build history in the past so we automatically delete old jobs once they are more than 1000 builds old, but we'd prefer not to have to trim the history down further than that. Once the page finally loads (takes around 2-3 minutes) it remains fast for a while, but eventually it triggers another reload of history from disk and takes 2-3 minutes again. Our build.xml files are about 15 KB each and our junitResult.xml files are about 6 MB each (not sure if it's actually reading the junitResult file or not).

          Andrew Lee added a comment -

          I traced the process using dtruss and can confirm that it was reading build.xml files for the vast majority of the page load time.

          Andrew Lee added a comment - I traced the process using dtruss and can confirm that it was reading build.xml files for the vast majority of the page load time.

          greg mallett added a comment -

          I changed this to "Core" for hopefully more visibility. Reading the comments, it seems this is not related to the multijob plugin as I originally thought.

          greg mallett added a comment - I changed this to "Core" for hopefully more visibility. Reading the comments, it seems this is not related to the multijob plugin as I originally thought.

          greg mallett added a comment -

          After dealing with this issue over the past week, I discovered that this might just be a symptom of another jenkins bug: https://issues.jenkins-ci.org/browse/JENKINS-19022

          We use git and our repository has a large number of branches. This is causing the git plugin's BuildData xml to explode to over 1.5 MB per build.xml file in the build history, as descibed in 19022. Parsing all of this xml is what is taking so much memory and cpu time.

          Still trying to find a workaround/fix for this problem, but this is quite a frustrating issue.

          greg mallett added a comment - After dealing with this issue over the past week, I discovered that this might just be a symptom of another jenkins bug: https://issues.jenkins-ci.org/browse/JENKINS-19022 We use git and our repository has a large number of branches. This is causing the git plugin's BuildData xml to explode to over 1.5 MB per build.xml file in the build history, as descibed in 19022. Parsing all of this xml is what is taking so much memory and cpu time. Still trying to find a workaround/fix for this problem, but this is quite a frustrating issue.

          Andrew Lee added a comment -

          We're using Mercurial in our setup and we're having this problem even with our relatively small build.xml files (~15 KB). Probably Git's larger build.xml files are amplifying the issue, but they're not entirely responsible.

          Andrew Lee added a comment - We're using Mercurial in our setup and we're having this problem even with our relatively small build.xml files (~15 KB). Probably Git's larger build.xml files are amplifying the issue, but they're not entirely responsible.

          greg mallett added a comment -

          The issue definitely is amplified in situations when the available JVM heap space is low. When the garbage collector collects the soft refs in AbstractLazyLoadRunMap, it will cause a fresh load from disk for the build history widget of all the job history build.xml files the next time the job page is accessed. So with sufficient free heap space, the build history widget loading is basically a once time operation. But without enough heap, we start getting this more and more and eventually the server slows to a crawl.

          greg mallett added a comment - The issue definitely is amplified in situations when the available JVM heap space is low. When the garbage collector collects the soft refs in AbstractLazyLoadRunMap, it will cause a fresh load from disk for the build history widget of all the job history build.xml files the next time the job page is accessed. So with sufficient free heap space, the build history widget loading is basically a once time operation. But without enough heap, we start getting this more and more and eventually the server slows to a crawl.

            Unassigned Unassigned
            gregmallett greg mallett
            Votes:
            7 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: