Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48868

blue ocean dashboard taking more than 50s to load.

    XMLWordPrintable

Details

    • Blue Ocean 1.5 - beta 1

    Description

      The dashboard is taking more than 50s to load and sometimes throws back 502 error. The error in log is below : - 

      // code placeholder
      Error while serving https://jenkins.XXXXXXXX.com/blue/rest/search/
      java.lang.NullPointerException
      	at io.jenkins.blueocean.rest.ApiHead.search(ApiHead.java:60)
      	at java.lang.invoke.MethodHandle.invokeWithArguments(Unknown Source)
      	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343)
      Caused: java.lang.reflect.InvocationTargetException
      	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:347)
      	at org.kohsuke.stapler.ForwardingFunction.invoke(ForwardingFunction.java:63)
      	at io.jenkins.blueocean.rest.pageable.PagedResponse$Processor.invoke(PagedResponse.java:42)
      	at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
      	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)
      	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
      	at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:686)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
      	at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:105)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      	at org.eclipse.jetty.server.Server.handle(Server.java:564)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
      	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
      	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
      	at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
      	at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
      	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      	at java.lang.Thread.run(Unknown Source)
      

      Attachments

        Issue Links

          Activity

            I too am seeing very slow BO load times but an not getting HTTP 5xxs.  The slow loads seem to be significantly more likely to happen after the instance has been running for several days.  I was suspecting heap management shenanigans but it appears that jenkins is using multiple cores for fairly long duration when this happens – seemingly too long to be GC.  Switching my jenkins master from an ec2 c4.large -> c4.xlarge seems to have helped by adding additional cores.

            jhoblitt Joshua Hoblitt added a comment - I too am seeing very slow BO load times but an not getting HTTP 5xxs.  The slow loads seem to be significantly more likely to happen after the instance has been running for several days.  I was suspecting heap management shenanigans but it appears that jenkins is using multiple cores for fairly long duration when this happens – seemingly too long to be GC.  Switching my jenkins master from an ec2 c4.large -> c4.xlarge seems to have helped by adding additional cores.
            vivek Vivek Pandey added a comment -

            What version Blueocean you are using? Do you see the slow load time when user is logged in or all the time? To help us debug please attached HAR file. Send it to me directly if it has confidential information.

            vivek Vivek Pandey added a comment - What version Blueocean you are using? Do you see the slow load time when user is logged in or all the time? To help us debug please attached HAR file. Send it to me directly if it has confidential information.

            vivek I'm running BO 1.3.4 / LTS core 2.89.1 (PS. I know about the sec. advisories).

            I just attempted to email you a HAR file... the message was about 14MiB with the attachment. If this gets eaten in transmission, let me know and I'll use a file drop.

            jhoblitt Joshua Hoblitt added a comment - vivek I'm running BO 1.3.4 / LTS core 2.89.1 (PS. I know about the sec. advisories). I just attempted to email you a HAR file... the message was about 14MiB with the attachment. If this gets eaten in transmission, let me know and I'll use a file drop.
            vivek Vivek Pandey added a comment -

            Hi jhoblitt,

            Got your HAR files. When you experience the slow page load of dashboard, can you also send me thread dump?

            vivek Vivek Pandey added a comment - Hi jhoblitt , Got your HAR files. When you experience the slow page load of dashboard, can you also send me thread dump?

            I've attached a cut'n'paste of the output from the /threadDump end point – is there a better way to do this?

            jhoblitt Joshua Hoblitt added a comment - I've attached a cut'n'paste of the output from the /threadDump end point – is there a better way to do this?

            What was the resolution?

            jhoblitt Joshua Hoblitt added a comment - What was the resolution?
            vivek Vivek Pandey added a comment -

            jhoblitt, sorry not enough info in this ticket.

             

            Tl;dr version: 

            Fix was to return run summary in latestRun instead of full Run object graph. See https://github.com/jenkinsci/blueocean-plugin/pull/1635.

             

            More details:

            Dashbaord lists pipelines and favorites. In both cases each pipeline also includes 'latestRun' property that includes complete Run object. Run objects can be very expensive and includes expensive objects such as changeSets and many others. Further, it was validated to be the case going over HAR files submitted by users on this ticket and others. 

            Favorites can be expensive too, specially it was resulting in to expensive computation and fact that we autofavorite and need all favorites so that we can show in pipeline/branch/activities pages - had more involved fixes. This is fixed as well, see PR https://github.com/jenkinsci/blueocean-plugin/pull/1638.

             

            Both PRs are merged to master. If you can, please give it a try. By the end of this week or early next week we are going to release 1.5-beta-1 and that should have these fixes as well.

            vivek Vivek Pandey added a comment - jhoblitt , sorry not enough info in this ticket.   Tl;dr version:  Fix was to return run summary in latestRun instead of full Run object graph. See https://github.com/jenkinsci/blueocean-plugin/pull/1635.   More details: Dashbaord lists pipelines and favorites. In both cases each pipeline also includes 'latestRun' property that includes complete Run object. Run objects can be very expensive and includes expensive objects such as changeSets and many others. Further, it was validated to be the case going over HAR files submitted by users on this ticket and others.  Favorites can be expensive too, specially it was resulting in to expensive computation and fact that we autofavorite and need all favorites so that we can show in pipeline/branch/activities pages - had more involved fixes. This is fixed as well, see PR https://github.com/jenkinsci/blueocean-plugin/pull/1638.   Both PRs are merged to master. If you can, please give it a try. By the end of this week or early next week we are going to release 1.5-beta-1 and that should have these fixes as well.

            That sounds very plausible and being related to favorites would explain why my production env seems slower than test instances.

            No code was attached to the ticket – is jira supposed to be doing that for the blueocean repo?

            jhoblitt Joshua Hoblitt added a comment - That sounds very plausible and being related to favorites would explain why my production env seems slower than test instances. No code was attached to the ticket – is jira supposed to be doing that for the blueocean repo?
            vivek Vivek Pandey added a comment -

            jhoblitt Right, need to check how GitHub commits appear on JIRA. michaelneale do you know how to get commits appear on JIRA tickets? I think if we at least update JIRA with PR being worked on is not bad either.

            vivek Vivek Pandey added a comment - jhoblitt  Right, need to check how GitHub commits appear on JIRA. michaelneale do you know how to get commits appear on JIRA tickets? I think if we at least update JIRA with PR being worked on is not bad either.

            People

              vivek Vivek Pandey
              xs2bharat Bharat Bhatia
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: