Loading...

Type: Bug
Resolution: Duplicate
Priority: Critical
Component/s: core
Labels:
- performance
Environment:
Jenkins 1.638
Chrome
Java 7
Windows Server 2008

Similar Issues:

Show

We are seeing sporadic slowdowns on our Jenkins server when opening certain jobs containing a lot of history. I believe it is something to do with the build history widget. There are probably >300+ builds in history.

The symptom is that clicking on the job page link from the main jenkins page takes forever to load (~5 min). Below is the stacktrace taken from one of the jenkins server request threads while experiencing the issue:

Name: Handling GET /job/nxn-frontend.feature-branch/ from 10.15.130.111 : RequestHandlerThread[#88] Job/index.jelly AbstractProject/sidepanel.jelly HistoryWidget/index.jelly
State: RUNNABLE
Total blocked: 1,259  Total waited: 140

Stack trace: 
com.thoughtworks.xstream.converters.reflection.FieldDictionary.buildMap(FieldDictionary.java:122)
   - locked com.thoughtworks.xstream.converters.reflection.FieldDictionary@627d3b18
com.thoughtworks.xstream.converters.reflection.FieldDictionary.fieldOrNull(FieldDictionary.java:113)
com.thoughtworks.xstream.converters.reflection.FieldDictionary.field(FieldDictionary.java:92)
com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.getFieldType(PureJavaReflectionProvider.java:160)
hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:292)
hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:229)
com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:352)
hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:290)
hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:229)
com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
hudson.util.RobustMapConverter.read(RobustMapConverter.java:58)
hudson.util.RobustMapConverter.putCurrentEntryIntoMap(RobustMapConverter.java:49)
com.thoughtworks.xstream.converters.collections.MapConverter.populateMap(MapConverter.java:98)
com.thoughtworks.xstream.converters.collections.MapConverter.populateMap(MapConverter.java:92)
com.thoughtworks.xstream.converters.collections.MapConverter.unmarshal(MapConverter.java:87)
com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:352)
hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:290)
hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:229)
com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:85)
com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:80)
hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:76)
com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:352)
hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:290)
hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:229)
com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
hudson.util.XStream2.unmarshal(XStream2.java:113)
com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
hudson.XmlFile.unmarshal(XmlFile.java:163)
hudson.model.Run.reload(Run.java:326)
hudson.model.Run.<init>(Run.java:314)
hudson.model.AbstractBuild.<init>(AbstractBuild.java:175)
hudson.model.Build.<init>(Build.java:103)
com.tikal.jenkins.plugins.multijob.MultiJobBuild.<init>(MultiJobBuild.java:56)
sun.reflect.GeneratedConstructorAccessor160.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
java.lang.reflect.Constructor.newInstance(Unknown Source)
jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:156)
jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:133)
hudson.model.RunMap.retrieve(RunMap.java:223)
hudson.model.RunMap.retrieve(RunMap.java:56)
jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:465)
   - locked hudson.model.RunMap@56d45de5
jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:448)
jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:356)
jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
jenkins.model.lazy.LazyBuildMixIn$RunMixIn.getPreviousBuild(LazyBuildMixIn.java:355)
hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:196)
hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:104)
hudson.model.RunMap$1.next(RunMap.java:109)
hudson.model.RunMap$1.next(RunMap.java:98)
org.apache.commons.collections.IteratorUtils.toList(IteratorUtils.java:849)
org.apache.commons.collections.IteratorUtils.toList(IteratorUtils.java:825)
hudson.widgets.BuildHistoryWidget.getHistoryPageFilter(BuildHistoryWidget.java:81)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
java.lang.reflect.Method.invoke(Unknown Source)
org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)
org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)
org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112)
org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:127)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:735)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
org.kohsuke.stapler.Stapler.service(Stapler.java:238)
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:123)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
org.eclipse.jetty.server.Server.handle(Server.java:370)
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.lang.Thread.run(Unknown Source)

Basically a big long journey is starting here:
org.apache.commons.collections.IteratorUtils.toList(IteratorUtils.java:825)
hudson.widgets.BuildHistoryWidget.getHistoryPageFilter(BuildHistoryWidget.java:81)

that results in what I'm assuming is every single one of those 300 jobs getting init-ed and their XML configurations parsed. Our job config.xml is also quite large. I can consistently reproduce this by clicking back and forth between two jobs with a lot of history.

duplicates

JENKINS-31791 Cannot load job pages due to HistoryWidget hanging

Resolved

Details

Description

Attachments

Issue Links

Activity

People

Dates