Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-31791

Cannot load job pages due to HistoryWidget hanging

    XMLWordPrintable

Details

    Description

      Job pages will not load. Get request never fulfilled. Web browser just idles eventually timing out. ThreadDump reveals it's an issue with HistoryWidget (see below). Reverting back to Jenkins 1.620 fixes problem.

      THREAD DUMP:

      *Handling GET /job/run_test_whck/ from 10.254.24.129 : RequestHandlerThread[#9] Job/index.jelly AbstractProject/sidepanel.jelly HistoryWidget/index.jelly
      *
      "Handling GET /job/run_test_whck/ from 10.254.24.129 : RequestHandlerThread[#9] Job/index.jelly AbstractProject/sidepanel.jelly HistoryWidget/index.jelly" Id=95 Group=main RUNNABLE
      	at java.io.FileInputStream.available(Native Method)
      	at java.io.BufferedInputStream.available(Unknown Source)
      	-  locked java.io.BufferedInputStream@1af40c8
      	at java.io.FilterInputStream.available(Unknown Source)
      	at java.io.PushbackInputStream.available(Unknown Source)
      	at sun.nio.cs.StreamDecoder.inReady(Unknown Source)
      	at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
      	at sun.nio.cs.StreamDecoder.read(Unknown Source)
      	-  locked java.io.InputStreamReader@1dbaa6a
      	at java.io.InputStreamReader.read(Unknown Source)
      	at com.thoughtworks.xstream.core.util.XmlHeaderAwareReader.read(XmlHeaderAwareReader.java:223)
      	at org.xmlpull.mxp1.MXParser.fillBuf(MXParser.java:2992)
      	at org.xmlpull.mxp1.MXParser.more(MXParser.java:3046)
      	at org.xmlpull.mxp1.MXParser.parseStartTag(MXParser.java:1738)
      	at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1127)
      	at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093)
      	at com.thoughtworks.xstream.io.xml.XppReader.pullNextEvent(XppReader.java:109)
      	at com.thoughtworks.xstream.io.xml.AbstractPullReader.readRealEvent(AbstractPullReader.java:148)
      	at com.thoughtworks.xstream.io.xml.AbstractPullReader.readEvent(AbstractPullReader.java:135)
      	at com.thoughtworks.xstream.io.xml.AbstractPullReader.hasMoreChildren(AbstractPullReader.java:87)
      	at com.thoughtworks.xstream.io.ReaderWrapper.hasMoreChildren(ReaderWrapper.java:32)
      	at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:82)
      	at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:80)
      	at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:76)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      	at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:352)
      	at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:290)
      	at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:229)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
      	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
      	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
      	at hudson.util.XStream2.unmarshal(XStream2.java:113)
      	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
      	at hudson.XmlFile.unmarshal(XmlFile.java:163)
      	at hudson.model.Run.reload(Run.java:326)
      	at hudson.model.Run.<init>(Run.java:314)
      	at hudson.model.AbstractBuild.<init>(AbstractBuild.java:175)
      	at hudson.model.Build.<init>(Build.java:103)
      	at hudson.model.FreeStyleBuild.<init>(FreeStyleBuild.java:38)
      	at sun.reflect.GeneratedConstructorAccessor104.newInstance(Unknown Source)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
      	at java.lang.reflect.Constructor.newInstance(Unknown Source)
      	at jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:156)
      	at jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:133)
      	at hudson.model.RunMap.retrieve(RunMap.java:223)
      	at hudson.model.RunMap.retrieve(RunMap.java:56)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:465)
      	-  locked hudson.model.RunMap@91485d
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:448)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:356)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
      	at jenkins.model.lazy.LazyBuildMixIn$RunMixIn.getPreviousBuild(LazyBuildMixIn.java:355)
      	at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:196)
      	at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:104)
      	at hudson.model.RunMap$1.next(RunMap.java:109)
      	at hudson.model.RunMap$1.next(RunMap.java:98)
      	at org.apache.commons.collections.IteratorUtils.toList(IteratorUtils.java:849)
      	at org.apache.commons.collections.IteratorUtils.toList(IteratorUtils.java:825)
      	at hudson.widgets.BuildHistoryWidget.getHistoryPageFilter(BuildHistoryWidget.java:81)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      	at java.lang.reflect.Method.invoke(Unknown Source)
      	at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
      	at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
      	at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
      	at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
      	at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
      	at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
      	at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
      	at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
      	at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
      	at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      	at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
      	at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)
      	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
      	at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
      	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
      	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
      	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      	at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
      	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      	at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
      	at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)
      	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
      	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
      	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
      	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      	at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
      	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
      	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
      	at org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112)
      	at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:127)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:735)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      	at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:123)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
      	at org.eclipse.jetty.server.Server.handle(Server.java:370)
      	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
      	at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
      	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
      	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
      	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
      	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
      	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
      	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
      	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      	at java.lang.Thread.run(Unknown Source)
      
      	Number of locked synchronizers = 1
      	- java.util.concurrent.ThreadPoolExecutor$Worker@10bbcda
      

      Attachments

        Issue Links

          Activity

            Code changed in jenkins
            User: DJ Lee
            Path:
            core/src/main/java/hudson/widgets/BuildHistoryWidget.java
            core/src/main/java/hudson/widgets/HistoryWidget.java
            core/src/main/java/jenkins/widgets/HistoryPageFilter.java
            core/src/test/java/jenkins/widgets/HistoryPageFilterTest.java
            http://jenkins-ci.org/commit/jenkins/55203ebeed1b7e182878d3e3c1184ac042f20473
            Log:
            [FIXED JENKINS-31791] Optimize BuildHistoryWidget (#2456)

            • [FIXED JENKINS-31791] Optimize BuildHistoryWidget

            Refactor HistoryPageFilter to lazily evaluate an iterable of previous
            runs, instead of instantiating a super long list of builds.
            Instantiating the whole list can be problematic with lots of
            historical builds especially if disk IO is slow.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: DJ Lee Path: core/src/main/java/hudson/widgets/BuildHistoryWidget.java core/src/main/java/hudson/widgets/HistoryWidget.java core/src/main/java/jenkins/widgets/HistoryPageFilter.java core/src/test/java/jenkins/widgets/HistoryPageFilterTest.java http://jenkins-ci.org/commit/jenkins/55203ebeed1b7e182878d3e3c1184ac042f20473 Log: [FIXED JENKINS-31791] Optimize BuildHistoryWidget (#2456) [FIXED JENKINS-31791] Optimize BuildHistoryWidget Refactor HistoryPageFilter to lazily evaluate an iterable of previous runs, instead of instantiating a super long list of builds. Instantiating the whole list can be problematic with lots of historical builds especially if disk IO is slow.
            djlee DJ Lee added a comment -

            leedega in my company's Jenkins instance, we were having problems like this with MultiJob and without, as long as there were a ton of builds in the history. Our Jenkins master is on an AWS instance with EBS disk, so the I/O is a bit slower than a physical machine, which is probably a contributing factor.

            The fix for above will be included in Jenkins 2.17.

            djlee DJ Lee added a comment - leedega in my company's Jenkins instance, we were having problems like this with MultiJob and without, as long as there were a ton of builds in the history. Our Jenkins master is on an AWS instance with EBS disk, so the I/O is a bit slower than a physical machine, which is probably a contributing factor. The fix for above will be included in Jenkins 2.17.
            djlee DJ Lee added a comment -

            olivergondza May I ask why you removed the lts-candidate tag from this issue? My company is still waiting for an LTS build containing this fix. Is there anything I can do to help get this considered or merged to the LTS branch?

            djlee DJ Lee added a comment - olivergondza May I ask why you removed the lts-candidate tag from this issue? My company is still waiting for an LTS build containing this fix. Is there anything I can do to help get this considered or merged to the LTS branch?

            djlee, because the fix was naturally absorbed by the next LTS line based on 2.17. RC to be pushed today.

            olivergondza Oliver Gondža added a comment - djlee , because the fix was naturally absorbed by the next LTS line based on 2.17. RC to be pushed today.
            djlee DJ Lee added a comment -

            olivergondza aha - thanks for explaining!

            djlee DJ Lee added a comment - olivergondza aha - thanks for explaining!

            People

              djlee DJ Lee
              mhulth Martin Hulth
              Votes:
              4 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: