Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-66221

Large artifact download fails at specific size when downloading through Jenkins UI

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Jenkins is hosted on a Ubuntu 20.04.2 LTS machine and is accessible to our users on our company network through HTTPS. Jenkins is sitting on its own subnet behind an Nginx reverse proxy. Users are connected to the network through a VPN and must access Jenkins through this VPN.

      We have a large zip archive artifact (~10GB). When downloading this artifact from the Jenkins UI, it always fails at 6.1GB no matter how long the download takes. Multiple users with different network speeds report the same issue. The download fails at 6.1GB in different browsers (Microsoft Edge, Chrome, etc) and does not matter how long the download took. Whether the download takes 30 minutes, 15 minutes, etc once it hits 6.1GB the download fails.

      The download can be resumed when it fails and it will then complete successfully but we're not sure why its failing in the first place. I don't think this is a timeout issue because it doesn't fail at a specific download time and I've tried increasing HTTPS timeouts by increasing the winstone -httpsKeepAliveTimeout Jenkins setting and this doesn't appear to have any affect.

      Although, I'm not sure if these winstone timeout settings work at all in Jenkins 2.289.2 because the Jenkins logs still show 5000 m/s even though I increased it to 10000 m/s. I'm actually not sure why this is showing in the logs because it doesn't fail the artifact downloads when these are showing until it hits 6.1GB:

      Jul 26, 2021 4:07:07 PM WARNING hudson.init.impl.InstallUncaughtExceptionHandler handleException
      null
      java.util.concurrent.TimeoutException: Idle timeout expired: 5001/5000 ms
      	at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:171)
      	at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:113)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      Caused: java.io.IOException
      	at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:257)
      	at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:270)
      	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:869)
      	at org.springframework.security.web.util.OnCommittedResponseWrapper$SaveContextServletOutputStream.write(OnCommittedResponseWrapper.java:638)
      	at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:628)
      	at org.kohsuke.stapler.ResponseImpl.serveFile(ResponseImpl.java:223)
      	at hudson.model.DirectoryBrowserSupport.serveFile(DirectoryBrowserSupport.java:394)
      	at hudson.model.DirectoryBrowserSupport.generateResponse(DirectoryBrowserSupport.java:156)
      	at org.kohsuke.stapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124)
      	at org.kohsuke.stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69)
      	at org.kohsuke.stapler.Function.renderResponse(Function.java:164)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:147)
      	at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:536)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
      	at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:220)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
      	at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:281)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
      	at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:281)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:694)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:240)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
      	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:159)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:92)
      	at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
      	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
      	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:92)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:218)
      	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:212)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
      	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:62)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:109)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:51)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      	at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:36)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
      	at org.eclipse.jetty.server.Server.handle(Server.java:516)
      	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
      	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
      	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
      	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
      	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
      	at java.lang.Thread.run(Thread.java:748)

      What's going on with these downloads?

        Attachments

          Activity

          Hide
          markewaite Mark Waite added a comment -

          Have you reviewed your nginx configuration to confirm that the nginx reverse proxy is not the cause of the terminated download at 6.1 GB?
          https://serverfault.com/questions/820597/nginx-does-not-serve-large-files indicates that there are nginx settings that may affect large file downloads.

          We generally prefer that questions be asked through the Jenkins user mailing list, since there are many more people monitoring the mailing list than are monitoring issue reports.

          Show
          markewaite Mark Waite added a comment - Have you reviewed your nginx configuration to confirm that the nginx reverse proxy is not the cause of the terminated download at 6.1 GB? https://serverfault.com/questions/820597/nginx-does-not-serve-large-files indicates that there are nginx settings that may affect large file downloads. We generally prefer that questions be asked through the Jenkins user mailing list, since there are many more people monitoring the mailing list than are monitoring issue reports.
          Hide
          ianboudreaux Ian Boudreaux added a comment - - edited

          Mark Waite Thanks for the quick response and for making me aware of the Mailing Lists (jenkins.io) - I did not know that existed! We are double checking our Nginx config now.

          I'm still a bit baffled at the idle timeout error I am receiving in the Jenkins systems logs from winstone that I posted above. I'm not sure why that timeout error above is showing in the logs at all (since it doesn't appear to be failing the downloads and it says "null") and why it isn't showing 10000 m/s since I increased the value.

          Since we're using Ubuntu, I increased the --httpsKeepAliveTimeout value in the /etc/default/jenkins file under JENKINS_ARGS. This is what my /etc/default/jenkins JENKINS_ARGS looks like:

          JENKINS_ARGS="--webroot=/var/cache/$NAME/war --httpPort=-1 --sessionTimeout=60 --httpKeepAliveTimeout=10000 --httpsKeepAliveTimeout=10000"

          Would this be a separate bug for Jenkins 2.289.2?

          Show
          ianboudreaux Ian Boudreaux added a comment - - edited Mark Waite  Thanks for the quick response and for making me aware of the Mailing Lists (jenkins.io) - I did not know that existed! We are double checking our Nginx config now. I'm still a bit baffled at the idle timeout error I am receiving in the Jenkins systems logs from winstone that I posted above. I'm not sure why that timeout error above is showing in the logs at all (since it doesn't appear to be failing the downloads and it says "null") and why it isn't showing 10000 m/s since I increased the value. Since we're using Ubuntu, I increased the --httpsKeepAliveTimeout value in the /etc/default/jenkins file under JENKINS_ARGS. This is what my /etc/default/jenkins JENKINS_ARGS looks like: JENKINS_ARGS= "--webroot=/ var /cache/$NAME/war --httpPort=-1 --sessionTimeout=60 --httpKeepAliveTimeout=10000 --httpsKeepAliveTimeout=10000" Would this be a separate bug for Jenkins 2.289.2?

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            ianboudreaux Ian Boudreaux
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated: