-
Bug
-
Resolution: Unresolved
-
Minor
-
Jenkins 2.401.1 org.jenkins-ci:winstone:6.10
JAVA: OpenJDK 11.0.19+7 18GB max heap.
Reverse Proxy: HAProxy 2.6
OS; Ubuntu 20.04 LTS
Server: Hyper-V (8 hyper-threaded cores, 32GB RAM)
JAVA_OPTS
-Djenkins.install.runSetupWizard=false -Djava.awt.headless=true -mx18g -Dcom.trilead.ssh2.channel.Channel.bufferSize=30000000 -Duser.country=AU -Duser.timezone=Australia/Sydney -Dcasc.reload.token=fooTODOchangeME -Dhudson.security.ArtifactsPermission=true -Djenkins.displayurl.provider=org.jenkinsci.plugins.displayurlapi.ClassicDisplayURLProvider -DBLUEOCEAN_FEATURE_AUTOFAVORITE_ENABLED=false
JENKINS_OPTS
--httpPort=8080 --httpKeepAliveTimeout=60000Jenkins 2.401.1 org.jenkins-ci:winstone:6.10 JAVA: OpenJDK 11.0.19+7 18GB max heap. Reverse Proxy: HAProxy 2.6 OS; Ubuntu 20.04 LTS Server: Hyper-V (8 hyper-threaded cores, 32GB RAM) JAVA_OPTS -Djenkins.install.runSetupWizard=false -Djava.awt.headless=true -mx18g -Dcom.trilead.ssh2.channel.Channel.bufferSize=30000000 -Duser.country=AU -Duser.timezone=Australia/Sydney -Dcasc.reload.token=fooTODOchangeME -Dhudson.security.ArtifactsPermission=true -Djenkins.displayurl.provider=org.jenkinsci.plugins.displayurlapi.ClassicDisplayURLProvider -DBLUEOCEAN_FEATURE_AUTOFAVORITE_ENABLED=false JENKINS_OPTS --httpPort=8080 --httpKeepAliveTimeout=60000
We have build jobs that download some large (100MB+) dependencies as build artifacts from our Jenkins CI. Sometimes while downloading, jenkins appears to close the connection a few minutes into the download, despite not having sent all the file data. At the time the connection is closed, the data transfer is still active and in progress (based on client side network monitoring).
In the past this problem was resolved by adding `–httpKeepAliveTimeout=60000` to the `JENKINS_OPTS` however since updating Jenkins from 2.361.4 to 2.375.3 the problem has come back. It continues to occur in Jenkins 2.401.1
We also changed our reverse proxy from Apache to HA Proxy and moved jenkins to a more powerful server, but the symptoms remain the same.
No errors are recorded in the proxy log. The following error and stack trace occurs in the Jenkins log whenever the downloads are truncated (the file name will be different, but the error is `java.lang.IllegalStateException: ABORTED` .)
2023-07-12 07:12:07.177+0000 [id=12206796] WARNING o.e.j.s.h.ContextHandler$Context#log: Error while serving https://resource.jenkins.[domain]/static-files/6CGfW9yV2UfO5Joz_H4QYs5MT85Qq7SmEom9NKp4IeoxNjg5MTQ1OTI2Mjc0OjEwOmJ1aWxkYWdlbnQ6am9iL2JhemVsX2RlcHMvam9iL21hc3Rlci9sYXN0U3VjY2Vzc2Z1bEJ1aWxkL2FydGlmYWN0/boost-1.80.0.tar.gz at java.base/java.lang.Thread.run(Thread.java:829) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416) at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762) at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505) at org.eclipse.jetty.server.Server.handle(Server.java:563) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:549) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:145) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:160) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:160) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:64) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:81) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:157) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764) at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) at org.kohsuke.stapler.Stapler.service(Stapler.java:248) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:698) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770) at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:475) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770) at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:475) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770) at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:501) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:140) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:207) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:409) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:401) Caused: java.lang.reflect.InvocationTargetException at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:397) at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710) at jenkins.security.ResourceDomainRootAction$InternalResourceRequest.doDynamic(ResourceDomainRootAction.java:259) at javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:122) at jenkins.metrics.impl.MetricsFilter$StatusExposingServletResponse.sendError(MetricsFilter.java:153) at javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:122) at org.springframework.security.web.util.OnCommittedResponseWrapper.sendError(OnCommittedResponseWrapper.java:116) at javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:122) at javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:122) at org.eclipse.jetty.server.Response.sendError(Response.java:465) at org.eclipse.jetty.server.Response.sendError(Response.java:498) at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:911) java.lang.IllegalStateException: ABORTED
Truncation like that is almost always a result of a reverse proxy performing the truncation. If you can temporarily configure direct access to Jenkins, you could remove the reverse proxy from the configuration to confirm that the issue is not specific to Jenkins.