-
Bug
-
Resolution: Fixed
-
Major
-
-
2.509
Hitting the /logText/progressiveText API of a completed build with very large log file ( XXX MB or X GB or more) exhaust the heap memory.
The Jetty thread handling it looks like this:
"Handling GET /job/testLog/5//logText/progressiveText from 127.0.0.1 : Jetty (winstone)-19" id=19 state=RUNNABLE at java.base@17.0.8/java.util.zip.Deflater.deflateBufferBytes(Native Method) at java.base@17.0.8/java.util.zip.Deflater.deflate(Deflater.java:751) at Jenkins Main ClassLoader//org.eclipse.jetty.server.handler.gzip.GzipResponseAndCallback$GzipBufferCB.compressing(GzipResponseAndCallback.java:410) at Jenkins Main ClassLoader//org.eclipse.jetty.server.handler.gzip.GzipResponseAndCallback$GzipBufferCB.process(GzipResponseAndCallback.java:370) at Jenkins Main ClassLoader//org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:262) at Jenkins Main ClassLoader//org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:243) at Jenkins Main ClassLoader//org.eclipse.jetty.server.handler.gzip.GzipResponseAndCallback.gzip(GzipResponseAndCallback.java:157) at Jenkins Main ClassLoader//org.eclipse.jetty.server.handler.gzip.GzipResponseAndCallback.write(GzipResponseAndCallback.java:135) at Jenkins Main ClassLoader//org.eclipse.jetty.server.Response$Wrapper.write(Response.java:768) at Jenkins Main ClassLoader//org.eclipse.jetty.server.handler.ContextResponse.write(ContextResponse.java:56) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HttpChannel.send(HttpChannel.java:1044) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HttpChannel.sendResponse(HttpChannel.java:1028) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HttpChannel.write(HttpChannel.java:1110) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HttpOutput.channelWrite(HttpOutput.java:277) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HttpOutput.channelWrite(HttpOutput.java:261) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HttpOutput.write(HttpOutput.java:876) at java.base@17.0.8/java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:161) - locked org.eclipse.jetty.util.ByteArrayOutputStream2@5ff96858 at Jenkins Main ClassLoader//org.eclipse.jetty.io.WriteThroughWriter$Utf8Writer.append(WriteThroughWriter.java:335) at Jenkins Main ClassLoader//org.eclipse.jetty.io.WriteThroughWriter.write(WriteThroughWriter.java:138) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.ResponseWriter.write(ResponseWriter.java:209) - locked org.eclipse.jetty.io.WriteThroughWriter$Utf8Writer@4cb1d7cd at org.springframework.security.web.util.OnCommittedResponseWrapper$SaveContextPrintWriter.write(OnCommittedResponseWrapper.java:314) at java.base@17.0.8/java.io.FilterWriter.write(FilterWriter.java:83) at org.kohsuke.stapler.framework.io.LineEndNormalizingWriter.write(LineEndNormalizingWriter.java:78) at org.kohsuke.stapler.framework.io.LineEndNormalizingWriter.write(LineEndNormalizingWriter.java:51) at org.kohsuke.stapler.framework.io.CharSpool.writeTo(CharSpool.java:86) at org.kohsuke.stapler.framework.io.LargeText.doProgressTextImpl(LargeText.java:319) at org.kohsuke.stapler.framework.io.LargeText.doProgressText(LargeText.java:274) at hudson.console.AnnotatedLargeText.doProgressiveText(AnnotatedLargeText.java:128) at java.base@17.0.8/java.lang.invoke.LambdaForm$DMH/0x000000013e120400.invokeVirtual(LambdaForm$DMH) at java.base@17.0.8/java.lang.invoke.LambdaForm$MH/0x000000013ece2800.invoke(LambdaForm$MH) at java.base@17.0.8/java.lang.invoke.LambdaForm$MH/0x000000013ec9dc00.invoke(LambdaForm$MH) at java.base@17.0.8/java.lang.invoke.LambdaForm$MH/0x000000013ec9c000.invokeExact_MT(LambdaForm$MH) at java.base@17.0.8/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:732) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:484) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:497) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:218) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:140) at org.kohsuke.stapler.MetaClass$12.doDispatch(MetaClass.java:686) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:61) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:800) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:938) at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:244) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:61) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:800) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:938) at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:590) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:800) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:938) at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:369) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:61) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:800) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:938) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:721) at org.kohsuke.stapler.Stapler.service(Stapler.java:253) at Jenkins Main ClassLoader//jakarta.servlet.http.HttpServlet.service(HttpServlet.java:587) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.ServletHolder.handle(ServletHolder.java:765)
On the other hand, the console output is not impacted by this and efficiently stream the logs.
The API documentation is not clear about how to stream the data should a REST Client want to do something similar:
Or if it is even possible.
How to Reproduce
- Spin a controller
- Create a pipeline that generates a lot of logs, for example:
node {
sh """
head -c 10073741824 /dev/urandom | base64 --break=1000
"""
}
- Build it
- Once completed, go to $JOB_URL/logText/progressiveText
- Generate a heapdump and observe that the Jetty thread holds everything in memory
cc @jglick who helped confirm this bug.
