Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48868

blue ocean dashboard taking more than 50s to load.

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • blueocean-plugin
    • None
    • Blue Ocean 1.5 - beta 1

      The dashboard is taking more than 50s to load and sometimes throws back 502 error. The error in log is below : - 

      // code placeholder
      Error while serving https://jenkins.XXXXXXXX.com/blue/rest/search/
      java.lang.NullPointerException
      	at io.jenkins.blueocean.rest.ApiHead.search(ApiHead.java:60)
      	at java.lang.invoke.MethodHandle.invokeWithArguments(Unknown Source)
      	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343)
      Caused: java.lang.reflect.InvocationTargetException
      	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:347)
      	at org.kohsuke.stapler.ForwardingFunction.invoke(ForwardingFunction.java:63)
      	at io.jenkins.blueocean.rest.pageable.PagedResponse$Processor.invoke(PagedResponse.java:42)
      	at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
      	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)
      	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
      	at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:686)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
      	at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:105)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      	at org.eclipse.jetty.server.Server.handle(Server.java:564)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
      	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
      	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
      	at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
      	at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
      	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      	at java.lang.Thread.run(Unknown Source)
      

          [JENKINS-48868] blue ocean dashboard taking more than 50s to load.

          I too am seeing very slow BO load times but an not getting HTTP 5xxs.  The slow loads seem to be significantly more likely to happen after the instance has been running for several days.  I was suspecting heap management shenanigans but it appears that jenkins is using multiple cores for fairly long duration when this happens – seemingly too long to be GC.  Switching my jenkins master from an ec2 c4.large -> c4.xlarge seems to have helped by adding additional cores.

          Joshua Hoblitt added a comment - I too am seeing very slow BO load times but an not getting HTTP 5xxs.  The slow loads seem to be significantly more likely to happen after the instance has been running for several days.  I was suspecting heap management shenanigans but it appears that jenkins is using multiple cores for fairly long duration when this happens – seemingly too long to be GC.  Switching my jenkins master from an ec2 c4.large -> c4.xlarge seems to have helped by adding additional cores.

          Vivek Pandey added a comment -

          What version Blueocean you are using? Do you see the slow load time when user is logged in or all the time? To help us debug please attached HAR file. Send it to me directly if it has confidential information.

          Vivek Pandey added a comment - What version Blueocean you are using? Do you see the slow load time when user is logged in or all the time? To help us debug please attached HAR file. Send it to me directly if it has confidential information.

          vivek I'm running BO 1.3.4 / LTS core 2.89.1 (PS. I know about the sec. advisories).

          I just attempted to email you a HAR file... the message was about 14MiB with the attachment. If this gets eaten in transmission, let me know and I'll use a file drop.

          Joshua Hoblitt added a comment - vivek I'm running BO 1.3.4 / LTS core 2.89.1 (PS. I know about the sec. advisories). I just attempted to email you a HAR file... the message was about 14MiB with the attachment. If this gets eaten in transmission, let me know and I'll use a file drop.

          Vivek Pandey added a comment -

          Hi jhoblitt,

          Got your HAR files. When you experience the slow page load of dashboard, can you also send me thread dump?

          Vivek Pandey added a comment - Hi jhoblitt , Got your HAR files. When you experience the slow page load of dashboard, can you also send me thread dump?

          I've attached a cut'n'paste of the output from the /threadDump end point – is there a better way to do this?

          Joshua Hoblitt added a comment - I've attached a cut'n'paste of the output from the /threadDump end point – is there a better way to do this?

          What was the resolution?

          Joshua Hoblitt added a comment - What was the resolution?

          Vivek Pandey added a comment -

          jhoblitt, sorry not enough info in this ticket.

           

          Tl;dr version: 

          Fix was to return run summary in latestRun instead of full Run object graph. See https://github.com/jenkinsci/blueocean-plugin/pull/1635.

           

          More details:

          Dashbaord lists pipelines and favorites. In both cases each pipeline also includes 'latestRun' property that includes complete Run object. Run objects can be very expensive and includes expensive objects such as changeSets and many others. Further, it was validated to be the case going over HAR files submitted by users on this ticket and others. 

          Favorites can be expensive too, specially it was resulting in to expensive computation and fact that we autofavorite and need all favorites so that we can show in pipeline/branch/activities pages - had more involved fixes. This is fixed as well, see PR https://github.com/jenkinsci/blueocean-plugin/pull/1638.

           

          Both PRs are merged to master. If you can, please give it a try. By the end of this week or early next week we are going to release 1.5-beta-1 and that should have these fixes as well.

          Vivek Pandey added a comment - jhoblitt , sorry not enough info in this ticket.   Tl;dr version:  Fix was to return run summary in latestRun instead of full Run object graph. See https://github.com/jenkinsci/blueocean-plugin/pull/1635.   More details: Dashbaord lists pipelines and favorites. In both cases each pipeline also includes 'latestRun' property that includes complete Run object. Run objects can be very expensive and includes expensive objects such as changeSets and many others. Further, it was validated to be the case going over HAR files submitted by users on this ticket and others.  Favorites can be expensive too, specially it was resulting in to expensive computation and fact that we autofavorite and need all favorites so that we can show in pipeline/branch/activities pages - had more involved fixes. This is fixed as well, see PR https://github.com/jenkinsci/blueocean-plugin/pull/1638.   Both PRs are merged to master. If you can, please give it a try. By the end of this week or early next week we are going to release 1.5-beta-1 and that should have these fixes as well.

          That sounds very plausible and being related to favorites would explain why my production env seems slower than test instances.

          No code was attached to the ticket – is jira supposed to be doing that for the blueocean repo?

          Joshua Hoblitt added a comment - That sounds very plausible and being related to favorites would explain why my production env seems slower than test instances. No code was attached to the ticket – is jira supposed to be doing that for the blueocean repo?

          Vivek Pandey added a comment -

          jhoblitt Right, need to check how GitHub commits appear on JIRA. michaelneale do you know how to get commits appear on JIRA tickets? I think if we at least update JIRA with PR being worked on is not bad either.

          Vivek Pandey added a comment - jhoblitt  Right, need to check how GitHub commits appear on JIRA. michaelneale do you know how to get commits appear on JIRA tickets? I think if we at least update JIRA with PR being worked on is not bad either.

            vivek Vivek Pandey
            xs2bharat Bharat Bhatia
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: