Loading...

XML

Word

Printable

Type: Bug
Resolution: Unresolved
Priority: Minor
Component/s: winstone-jetty
Labels:
- jetty
Environment:

Hide
Jenkins 2.401.1 org.jenkins-ci:winstone:6.10
JAVA: OpenJDK 11.0.19+7 18GB max heap.
Reverse Proxy: HAProxy 2.6
OS; Ubuntu 20.04 LTS
Server: Hyper-V (8 hyper-threaded cores, 32GB RAM)

JAVA_OPTS
-Djenkins.install.runSetupWizard=false -Djava.awt.headless=true -mx18g -Dcom.trilead.ssh2.channel.Channel.bufferSize=30000000 -Duser.country=AU -Duser.timezone=Australia/Sydney -Dcasc.reload.token=fooTODOchangeME -Dhudson.security.ArtifactsPermission=true -Djenkins.displayurl.provider=org.jenkinsci.plugins.displayurlapi.ClassicDisplayURLProvider -DBLUEOCEAN_FEATURE_AUTOFAVORITE_ENABLED=false

JENKINS_OPTS
--httpPort=8080 --httpKeepAliveTimeout=60000

Show
Jenkins 2.401.1 org.jenkins-ci:winstone:6.10 JAVA: OpenJDK 11.0.19+7 18GB max heap. Reverse Proxy: HAProxy 2.6 OS; Ubuntu 20.04 LTS Server: Hyper-V (8 hyper-threaded cores, 32GB RAM) JAVA_OPTS -Djenkins.install.runSetupWizard=false -Djava.awt.headless=true -mx18g -Dcom.trilead.ssh2.channel.Channel.bufferSize=30000000 -Duser.country=AU -Duser.timezone=Australia/Sydney -Dcasc.reload.token=fooTODOchangeME -Dhudson.security.ArtifactsPermission=true -Djenkins.displayurl.provider=org.jenkinsci.plugins.displayurlapi.ClassicDisplayURLProvider -DBLUEOCEAN_FEATURE_AUTOFAVORITE_ENABLED=false JENKINS_OPTS --httpPort=8080 --httpKeepAliveTimeout=60000

Similar Issues:

Show

We have build jobs that download some large (100MB+) dependencies as build artifacts from our Jenkins CI. Sometimes while downloading, jenkins appears to close the connection a few minutes into the download, despite not having sent all the file data. At the time the connection is closed, the data transfer is still active and in progress (based on client side network monitoring).

In the past this problem was resolved by adding `–httpKeepAliveTimeout=60000` to the `JENKINS_OPTS` however since updating Jenkins from 2.361.4 to 2.375.3 the problem has come back. It continues to occur in Jenkins 2.401.1

We also changed our reverse proxy from Apache to HA Proxy and moved jenkins to a more powerful server, but the symptoms remain the same.

No errors are recorded in the proxy log. The following error and stack trace occurs in the Jenkins log whenever the downloads are truncated (the file name will be different, but the error is `java.lang.IllegalStateException: ABORTED` .)

2023-07-12 07:12:07.177+0000 [id=12206796]	WARNING	o.e.j.s.h.ContextHandler$Context#log: Error while serving https://resource.jenkins.[domain]/static-files/6CGfW9yV2UfO5Joz_H4QYs5MT85Qq7SmEom9NKp4IeoxNjg5MTQ1OTI2Mjc0OjEwOmJ1aWxkYWdlbnQ6am9iL2JhemVsX2RlcHMvam9iL21hc3Rlci9sYXN0U3VjY2Vzc2Z1bEJ1aWxkL2FydGlmYWN0/boost-1.80.0.tar.gz
	at java.base/java.lang.Thread.run(Thread.java:829)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
	at org.eclipse.jetty.server.Server.handle(Server.java:563)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:549)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:145)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:160)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:160)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:64)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:81)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:157)
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:248)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:698)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
	at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:475)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
	at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:475)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
	at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:501)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:140)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:207)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:409)
	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:401)
Caused: java.lang.reflect.InvocationTargetException
	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:397)
	at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
	at jenkins.security.ResourceDomainRootAction$InternalResourceRequest.doDynamic(ResourceDomainRootAction.java:259)
	at javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:122)
	at jenkins.metrics.impl.MetricsFilter$StatusExposingServletResponse.sendError(MetricsFilter.java:153)
	at javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:122)
	at org.springframework.security.web.util.OnCommittedResponseWrapper.sendError(OnCommittedResponseWrapper.java:116)
	at javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:122)
	at javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:122)
	at org.eclipse.jetty.server.Response.sendError(Response.java:465)
	at org.eclipse.jetty.server.Response.sendError(Response.java:498)
	at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:911)
java.lang.IllegalStateException: ABORTED

Assignee:: Unassigned

Reporter:: Liam Baker

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Created:: 2023-07-12 08:11

Updated:: 2023-07-12 23:28

Details

Description

Attachments

Activity

People

Dates