Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-71627

static file downloads are truncated after a few minutes

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • winstone-jetty

      We have build jobs that download some large (100MB+) dependencies as build artifacts from our Jenkins CI. Sometimes while downloading, jenkins appears to close the connection a few minutes into the download, despite not having sent all the file data. At the time the connection is closed, the data transfer is still active and in progress (based on client side network monitoring).

      In the past this problem was resolved by adding `–httpKeepAliveTimeout=60000` to the `JENKINS_OPTS` however since updating Jenkins from 2.361.4 to 2.375.3 the problem has come back. It continues to occur in Jenkins 2.401.1

      We also changed our reverse proxy from Apache to HA Proxy and moved jenkins to a more powerful server, but the symptoms remain the same.

      No errors are recorded in the proxy log. The following error and stack trace occurs in the Jenkins log whenever the downloads are truncated (the file name will be different, but the error is `java.lang.IllegalStateException: ABORTED` .)

      2023-07-12 07:12:07.177+0000 [id=12206796]	WARNING	o.e.j.s.h.ContextHandler$Context#log: Error while serving https://resource.jenkins.[domain]/static-files/6CGfW9yV2UfO5Joz_H4QYs5MT85Qq7SmEom9NKp4IeoxNjg5MTQ1OTI2Mjc0OjEwOmJ1aWxkYWdlbnQ6am9iL2JhemVsX2RlcHMvam9iL21hc3Rlci9sYXN0U3VjY2Vzc2Z1bEJ1aWxkL2FydGlmYWN0/boost-1.80.0.tar.gz
      	at java.base/java.lang.Thread.run(Thread.java:829)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
      	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
      	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140)
      	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272)
      	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385)
      	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416)
      	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
      	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
      	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
      	at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
      	at org.eclipse.jetty.server.Server.handle(Server.java:563)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:549)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
      	at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
      	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
      	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
      	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
      	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
      	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
      	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
      	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
      	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:145)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
      	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
      	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
      	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
      	at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:160)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:160)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      	at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:64)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      	at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      	at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:81)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:157)
      	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:248)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:698)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
      	at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:475)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
      	at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:475)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
      	at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:501)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:140)
      	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:207)
      	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:409)
      	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:401)
      Caused: java.lang.reflect.InvocationTargetException
      	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:397)
      	at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
      	at jenkins.security.ResourceDomainRootAction$InternalResourceRequest.doDynamic(ResourceDomainRootAction.java:259)
      	at javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:122)
      	at jenkins.metrics.impl.MetricsFilter$StatusExposingServletResponse.sendError(MetricsFilter.java:153)
      	at javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:122)
      	at org.springframework.security.web.util.OnCommittedResponseWrapper.sendError(OnCommittedResponseWrapper.java:116)
      	at javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:122)
      	at javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:122)
      	at org.eclipse.jetty.server.Response.sendError(Response.java:465)
      	at org.eclipse.jetty.server.Response.sendError(Response.java:498)
      	at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:911)
      java.lang.IllegalStateException: ABORTED
      

          [JENKINS-71627] static file downloads are truncated after a few minutes

          Mark Waite added a comment -

          Truncation like that is almost always a result of a reverse proxy performing the truncation. If you can temporarily configure direct access to Jenkins, you could remove the reverse proxy from the configuration to confirm that the issue is not specific to Jenkins.

          Mark Waite added a comment - Truncation like that is almost always a result of a reverse proxy performing the truncation. If you can temporarily configure direct access to Jenkins, you could remove the reverse proxy from the configuration to confirm that the issue is not specific to Jenkins.

          Liam Baker added a comment -

          Thanks, will look further into proxy logging and configuration.

          Liam Baker added a comment - Thanks, will look further into proxy logging and configuration.

            Unassigned Unassigned
            sm__liam_baker Liam Baker
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: