• Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • core, winstone-jetty
    • None

      Multiple reports (like JENKINS-42058) indicate the default timeouts are too aggressive when dealing with terrible connections.

          [JENKINS-42513] Increase default keep-alive timeouts

          Josh Soref added a comment -

          Note: --httpKeepAliveTimeout=10000 appears to be honored whether winstone means jetty or actually winstone...

          This is what we hit:

          java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 5000/5000 ms
                  at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:235)
                  at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:216)
                  at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:514)
                  at net.bull.javamelody.internal.web.FilterServletOutputStream.write(FilterServletOutputStream.java:88)
                  at net.bull.javamelody.internal.web.CounterResponseStream.write(CounterResponseStream.java:81)
                  at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:604)
                  at org.kohsuke.stapler.ResponseImpl.serveFile(ResponseImpl.java:216)
                  at hudson.model.DirectoryBrowserSupport.serveFile(DirectoryBrowserSupport.java:349)
                  at hudson.model.DirectoryBrowserSupport.generateResponse(DirectoryBrowserSupport.java:141)
                  at org.kohsuke.stapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124)
                  at org.kohsuke.stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69)
                  at org.kohsuke.stapler.Function.renderResponse(Function.java:164)
                  at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:147)
                  at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:537)
                  at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
                  at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
                  at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
                  at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:458)
                  at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
                  at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
                  at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:282)
                  at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
                  at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
                  at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
                  at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
                  at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
                  at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
                  at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
                  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
                  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
                  at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243)
                  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
                  at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
                  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
                  at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
                  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
                  at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
                  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
                  at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
                  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
                  at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
                  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
                  at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
                  at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
                  at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
                  at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114)
                  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
                  at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
                  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
                  at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)
                  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
                  at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
                  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
                  at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:105)
                  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
                  at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                  at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                  at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                  at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                  at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                  at jenkins.security.BasicHeaderProcessor.success(BasicHeaderProcessor.java:140)
                  at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:82)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                  at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
                  at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                  at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
                  at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
                  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
                  at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
                  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
                  at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
                  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
                  at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
                  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
                  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
                  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
                  at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
                  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
                  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
                  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
                  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
                  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
                  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
                  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
                  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
                  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
                  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
                  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
                  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
                  at org.eclipse.jetty.server.Server.handle(Server.java:503)
                  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
                  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
                  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
                  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
                  at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
                  at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
                  at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
                  at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
                  at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
                  at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
                  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
                  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
                  at java.lang.Thread.run(Thread.java:748)
          Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 5000/5000 ms
                  at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)
                  at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                  ... 1 more 

          I'm going to try this for a week and see if it causes our problems to go away.

          Afaict, this wasn't a problem until a recent jenkins (or build node) upgrade. But recently it's become a significant problem.

          Josh Soref added a comment - Note: --httpKeepAliveTimeout=10000 appears to be honored whether winstone means jetty or actually winstone... This is what we hit: java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 5000/5000 ms at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:235) at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:216) at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:514) at net.bull.javamelody.internal.web.FilterServletOutputStream.write(FilterServletOutputStream.java:88) at net.bull.javamelody.internal.web.CounterResponseStream.write(CounterResponseStream.java:81) at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:604) at org.kohsuke.stapler.ResponseImpl.serveFile(ResponseImpl.java:216) at hudson.model.DirectoryBrowserSupport.serveFile(DirectoryBrowserSupport.java:349) at hudson.model.DirectoryBrowserSupport.generateResponse(DirectoryBrowserSupport.java:141) at org.kohsuke.stapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124) at org.kohsuke.stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69) at org.kohsuke.stapler.Function.renderResponse(Function.java:164) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:147) at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:537) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870) at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:458) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:282) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:105) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.success(BasicHeaderProcessor.java:140) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:82) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:503) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.lang. Thread .run( Thread .java:748) Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 5000/5000 ms at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166) at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more I'm going to try this for a week and see if it causes our problems to go away. Afaict, this wasn't a problem until a recent jenkins (or build node) upgrade. But recently it's become a significant problem.

          It's not just "terrible connections"; it could also be a consumer being slow for another reason.
          We just ran into this issue with a shell script doing:

            curl -f $JENKINS/$JOB/artifact/binaries.tar.gz | tar xz

          Sometimes unpacking was too slow (due to high load on the VM) and Jenkins terminated the download.

          Daniel Grunwald added a comment - It's not just "terrible connections"; it could also be a consumer being slow for another reason. We just ran into this issue with a shell script doing:   curl -f $JENKINS/$JOB/artifact/binaries.tar.gz | tar xz Sometimes unpacking was too slow (due to high load on the VM) and Jenkins terminated the download.

          Liam Baker added a comment - - edited

          We were experiencing truncation of downloaded artefacts for remote users (VPN), parallel downloads, and nodes that were disk bound or under heavy load (e.g. downloading CI artefacts during a build step). Access logs on the reverse proxy showed Jenkins dropping the connection.
          Following a suggestion on https://issues.jenkins.io/browse/JENKINS-39909 to set --httpKeepAliveTimeout=60000 truncation has not been seen since.

          Liam Baker added a comment - - edited We were experiencing truncation of downloaded artefacts for remote users (VPN), parallel downloads, and nodes that were disk bound or under heavy load (e.g. downloading CI artefacts during a build step). Access logs on the reverse proxy showed Jenkins dropping the connection. Following a suggestion on https://issues.jenkins.io/browse/JENKINS-39909 to set --httpKeepAliveTimeout=60000 truncation has not been seen since.

          Liam Baker added a comment - - edited

          This issue is back since updating to Jenkins 2.387.3
          I can reproduce for archives over 2GB in size, from the client side, the download simply hangs and then is terminated after a minute, the time before hang appears random.
          No errors are seen in the proxy access logs.
          On jenkins log is seen ( in this case for a .zip archive, but it happens for direct file download as well)

          2023-06-15T13:41:45.953007303+10:00 2023-06-15 03:41:45.950+0000 [id=741063]    WARNING h.i.i.InstallUncaughtExceptionHandler#handleException
          2023-06-15T13:41:45.953030710+10:00 java.util.concurrent.TimeoutException: Idle timeout expired: 60000/60000 ms
          2023-06-15T13:41:45.953034428+10:00     at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:170)
          2023-06-15T13:41:45.953035456+10:00     at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:112)
          2023-06-15T13:41:45.953036331+10:00     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          2023-06-15T13:41:45.953037213+10:00     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          2023-06-15T13:41:45.953038155+10:00     at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
          2023-06-15T13:41:45.953039548+10:00     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          2023-06-15T13:41:45.953040729+10:00     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          2023-06-15T13:41:45.953041981+10:00 Also:   org.eclipse.jetty.io.EofException: Closed
          2023-06-15T13:41:45.953043423+10:00             at org.eclipse.jetty.server.HttpOutput.checkWritable(HttpOutput.java:756)
          2023-06-15T13:41:45.953044722+10:00             at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:780)
          2023-06-15T13:41:45.953046155+10:00             at org.springframework.security.web.util.OnCommittedResponseWrapper$SaveContextServletOutputStream.write(OnCommittedResponseWrapper.java:638)
          2023-06-15T13:41:45.953047598+10:00             at org.apache.tools.zip.ZipOutputStream.writeOut(ZipOutputStream.java:1549)
          2023-06-15T13:41:45.953049052+10:00             at org.apache.tools.zip.ZipOutputStream.writeCounted(ZipOutputStream.java:956)
          2023-06-15T13:41:45.953050855+10:00             at org.apache.tools.zip.ZipOutputStream.deflate(ZipOutputStream.java:1065)
          2023-06-15T13:41:45.953052345+10:00             at org.apache.tools.zip.ZipOutputStream.flushDeflater(ZipOutputStream.java:593)
          2023-06-15T13:41:45.953054331+10:00             at org.apache.tools.zip.ZipOutputStream.closeEntry(ZipOutputStream.java:550)
          2023-06-15T13:41:45.953056074+10:00             at org.apache.tools.zip.ZipOutputStream.finish(ZipOutputStream.java:509)
          2023-06-15T13:41:45.953057651+10:00             at org.apache.tools.zip.ZipOutputStream.close(ZipOutputStream.java:1002)
          2023-06-15T13:41:45.953059199+10:00             at hudson.util.io.ZipArchiver.close(ZipArchiver.java:114)
          2023-06-15T13:41:45.953060577+10:00             at jenkins.util.VirtualFile$FileVF.zip(VirtualFile.java:693)
          2023-06-15T13:41:45.953062001+10:00 Caused: java.io.IOException
          2023-06-15T13:41:45.953063820+10:00     at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:221)
          2023-06-15T13:41:45.953065314+10:00     at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:255)
          2023-06-15T13:41:45.953068001+10:00     at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:858)
          2023-06-15T13:41:45.953074241+10:00     at org.springframework.security.web.util.OnCommittedResponseWrapper$SaveContextServletOutputStream.write(OnCommittedResponseWrapper.java:638)
          2023-06-15T13:41:45.953075432+10:00     at org.apache.tools.zip.ZipOutputStream.writeOut(ZipOutputStream.java:1549)
          2023-06-15T13:41:45.953076588+10:00     at org.apache.tools.zip.ZipOutputStream.writeCounted(ZipOutputStream.java:956)
          2023-06-15T13:41:45.953077813+10:00     at org.apache.tools.zip.ZipOutputStream.deflate(ZipOutputStream.java:1065)
          2023-06-15T13:41:45.953079027+10:00     at org.apache.tools.zip.ZipOutputStream.flushDeflater(ZipOutputStream.java:593)
          2023-06-15T13:41:45.953080471+10:00     at org.apache.tools.zip.ZipOutputStream.closeEntry(ZipOutputStream.java:550)
          

          Liam Baker added a comment - - edited This issue is back since updating to Jenkins 2.387.3 I can reproduce for archives over 2GB in size, from the client side, the download simply hangs and then is terminated after a minute, the time before hang appears random. No errors are seen in the proxy access logs. On jenkins log is seen ( in this case for a .zip archive, but it happens for direct file download as well) 2023-06-15T13:41:45.953007303+10:00 2023-06-15 03:41:45.950+0000 [id=741063] WARNING h.i.i.InstallUncaughtExceptionHandler#handleException 2023-06-15T13:41:45.953030710+10:00 java.util.concurrent.TimeoutException: Idle timeout expired: 60000/60000 ms 2023-06-15T13:41:45.953034428+10:00 at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:170) 2023-06-15T13:41:45.953035456+10:00 at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:112) 2023-06-15T13:41:45.953036331+10:00 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) 2023-06-15T13:41:45.953037213+10:00 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 2023-06-15T13:41:45.953038155+10:00 at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) 2023-06-15T13:41:45.953039548+10:00 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 2023-06-15T13:41:45.953040729+10:00 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 2023-06-15T13:41:45.953041981+10:00 Also: org.eclipse.jetty.io.EofException: Closed 2023-06-15T13:41:45.953043423+10:00 at org.eclipse.jetty.server.HttpOutput.checkWritable(HttpOutput.java:756) 2023-06-15T13:41:45.953044722+10:00 at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:780) 2023-06-15T13:41:45.953046155+10:00 at org.springframework.security.web.util.OnCommittedResponseWrapper$SaveContextServletOutputStream.write(OnCommittedResponseWrapper.java:638) 2023-06-15T13:41:45.953047598+10:00 at org.apache.tools.zip.ZipOutputStream.writeOut(ZipOutputStream.java:1549) 2023-06-15T13:41:45.953049052+10:00 at org.apache.tools.zip.ZipOutputStream.writeCounted(ZipOutputStream.java:956) 2023-06-15T13:41:45.953050855+10:00 at org.apache.tools.zip.ZipOutputStream.deflate(ZipOutputStream.java:1065) 2023-06-15T13:41:45.953052345+10:00 at org.apache.tools.zip.ZipOutputStream.flushDeflater(ZipOutputStream.java:593) 2023-06-15T13:41:45.953054331+10:00 at org.apache.tools.zip.ZipOutputStream.closeEntry(ZipOutputStream.java:550) 2023-06-15T13:41:45.953056074+10:00 at org.apache.tools.zip.ZipOutputStream.finish(ZipOutputStream.java:509) 2023-06-15T13:41:45.953057651+10:00 at org.apache.tools.zip.ZipOutputStream.close(ZipOutputStream.java:1002) 2023-06-15T13:41:45.953059199+10:00 at hudson.util.io.ZipArchiver.close(ZipArchiver.java:114) 2023-06-15T13:41:45.953060577+10:00 at jenkins.util.VirtualFile$FileVF.zip(VirtualFile.java:693) 2023-06-15T13:41:45.953062001+10:00 Caused: java.io.IOException 2023-06-15T13:41:45.953063820+10:00 at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:221) 2023-06-15T13:41:45.953065314+10:00 at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:255) 2023-06-15T13:41:45.953068001+10:00 at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:858) 2023-06-15T13:41:45.953074241+10:00 at org.springframework.security.web.util.OnCommittedResponseWrapper$SaveContextServletOutputStream.write(OnCommittedResponseWrapper.java:638) 2023-06-15T13:41:45.953075432+10:00 at org.apache.tools.zip.ZipOutputStream.writeOut(ZipOutputStream.java:1549) 2023-06-15T13:41:45.953076588+10:00 at org.apache.tools.zip.ZipOutputStream.writeCounted(ZipOutputStream.java:956) 2023-06-15T13:41:45.953077813+10:00 at org.apache.tools.zip.ZipOutputStream.deflate(ZipOutputStream.java:1065) 2023-06-15T13:41:45.953079027+10:00 at org.apache.tools.zip.ZipOutputStream.flushDeflater(ZipOutputStream.java:593) 2023-06-15T13:41:45.953080471+10:00 at org.apache.tools.zip.ZipOutputStream.closeEntry(ZipOutputStream.java:550)

            Unassigned Unassigned
            danielbeck Daniel Beck
            Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated: