I can reproduce this issue - or a similar one- in a very big instance where I had a lot of builds in a NFS system.
I could see that the support bundle is corrupted when the option About Jenkins is enabled. When enabling this option, then I could see the support bundle generation looked like a bit stuck and after 60 seconds it fnished, but in a corrupted status.
I took a threadDump in themoment the generation process looked stuck and I saw:
"Handling POST /support/download from XX.XX.XX.XX : RequestHandlerThread[#126]" Id=123772 Group=main RUNNABLE
at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)
at java.io.File.isDirectory(File.java:849)
at com.cloudbees.jenkins.support.impl.AboutJenkins$ItemsContent.printTo(AboutJenkins.java:641)
at com.cloudbees.jenkins.support.api.PrintedContent.writeTo(PrintedContent.java:47)
at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:359)
at com.cloudbees.jenkins.support.SupportAction.doDownload(SupportAction.java:154)
at java.lang.invoke.LambdaForm$DMH/498240235.invokeVirtual_L3_V(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$BMH/2063152025.reinvoke(LambdaForm$BMH)
The stacktrace you will see on Jenkins logs are something like this:
2018-01-04 04:52:17.633+0000 [id=123769] WARNING c.c.j.support.SupportPlugin#writeBundle: Could not attach 'nodes/slave/Slave1/checksums.md5' to support bundle
org.eclipse.jetty.io.EofException: Closed
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:476)
at net.bull.javamelody.FilterServletOutputStream.write(FilterServletOutputStream.java:88)
at net.bull.javamelody.CounterResponseStream.write(CounterResponseStream.java:82)
at org.apache.tools.zip.ZipOutputStream.writeOut(ZipOutputStream.java:1029)
at org.apache.tools.zip.ZipOutputStream.deflate(ZipOutputStream.java:680)
at org.apache.tools.zip.ZipOutputStream.closeEntry(ZipOutputStream.java:432)
at org.apache.tools.zip.ZipOutputStream.putNextEntry(ZipOutputStream.java:489)
at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:358)
at com.cloudbees.jenkins.support.SupportAction.doDownload(SupportAction.java:154)
at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343)
at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:52)
at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)
at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)
at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
at com.cloudbees.jenkins.ha.HAHealthCheckFilter.doFilter(HAHealthCheckFilter.java:35)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
at com.cloudbees.jenkins.support.impl.cloudbees.UnrestrictedApiCallsMonitor$ApiMonitorFilter.doFilter(UnrestrictedApiCallsMonitor.java:120)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at com.cloudbees.opscenter.client.plugin.OfflineSecurityRealmFilter._doFilter(OfflineSecurityRealmFilter.java:95)
at com.cloudbees.opscenter.client.plugin.OfflineSecurityRealmFilter.doFilter(OfflineSecurityRealmFilter.java:70)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at com.cloudbees.opscenter.security.ClusterSessionFilter._doFilter(ClusterSessionFilter.java:69)
at com.cloudbees.opscenter.security.ClusterSessionFilter.doFilter(ClusterSessionFilter.java:44)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at org.jenkinsci.plugins.suppress_stack_trace.SuppressionFilter.doFilter(SuppressionFilter.java:34)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:95)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:230)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:202)
at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:85)
at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:112)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:138)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:92)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:564)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
The issue was due the load balancer was configured with an idle timeout of 60 seconds. Increasing the timeout fixed the problem. We also checked that accessing directly through the machine IP the support-bundle was always correctly generated.
I think we should expose more information in the logs so at very least this common issue is explained.
I can reproduce this issue - or a similar one- in a very big instance where I had a lot of builds in a NFS system.
I could see that the support bundle is corrupted when the option About Jenkins is enabled. When enabling this option, then I could see the support bundle generation looked like a bit stuck and after 60 seconds it fnished, but in a corrupted status.
I took a threadDump in themoment the generation process looked stuck and I saw:
"Handling POST /support/download from XX.XX.XX.XX : RequestHandlerThread[#126]" Id=123772 Group=main RUNNABLE at java.io.UnixFileSystem.getBooleanAttributes0(Native Method) at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242) at java.io.File.isDirectory(File.java:849) at com.cloudbees.jenkins.support.impl.AboutJenkins$ItemsContent.printTo(AboutJenkins.java:641) at com.cloudbees.jenkins.support.api.PrintedContent.writeTo(PrintedContent.java:47) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:359) at com.cloudbees.jenkins.support.SupportAction.doDownload(SupportAction.java:154) at java.lang.invoke.LambdaForm$DMH/498240235.invokeVirtual_L3_V(LambdaForm$DMH) at java.lang.invoke.LambdaForm$BMH/2063152025.reinvoke(LambdaForm$BMH)
The stacktrace you will see on Jenkins logs are something like this:
The issue was due the load balancer was configured with an idle timeout of 60 seconds. Increasing the timeout fixed the problem. We also checked that accessing directly through the machine IP the support-bundle was always correctly generated.
I think we should expose more information in the logs so at very least this common issue is explained.