[JENKINS-57138] Performance issue due to the bundle anonymization

Type: Bug
Resolution: Fixed
Priority: Major
Component/s: support-core-plugin
Labels:
None

Similar Issues:
Powered by SuggestiMate

Show
Released As:
support-core:1315.v0f514a_a_7e133

This is a follow-up of ~~JENKINS-54999~~

In the latest release of the support-core plugin {{support-core:2.56:not-pinned
}} we have seen performance issues

Without anonymization (I have launch 3 times) 
~35seconds, and I see at max ~100% CPU usage

With anonymization (I have launch 1 time regarding the time) 
~11 minutes, and I see at max ~360% CPU usage

The relevant stacktraces are:

ThreadDump 1

"Handling POST /XXXX/support/generateAllBundles from X.X.X.X : qtp2007328737-1361430" #1361430 prio=5 os_prio=0 tid=0x00007efcedfc0000 nid=0x12c4 runnable [0x00007efcb3de0000]
   java.lang.Thread.State: RUNNABLE
	at java.util.zip.Deflater.deflateBytes(Native Method)
	at java.util.zip.Deflater.deflate(Deflater.java:444)
	- locked <0x0000000719156578> (a java.util.zip.ZStreamRef)
	at java.util.zip.Deflater.deflate(Deflater.java:366)
	at org.apache.commons.compress.archivers.zip.StreamCompressor.deflate(StreamCompressor.java:246)
	at org.apache.commons.compress.archivers.zip.StreamCompressor.deflateUntilInputIsNeeded(StreamCompressor.java:241)
	at org.apache.commons.compress.archivers.zip.StreamCompressor.writeDeflated(StreamCompressor.java:222)
	at org.apache.commons.compress.archivers.zip.StreamCompressor.write(StreamCompressor.java:190)
	at org.apache.commons.compress.archivers.zip.ZipArchiveOutputStream.write(ZipArchiveOutputStream.java:900)
	at org.apache.commons.compress.archivers.ArchiveOutputStream.write(ArchiveOutputStream.java:109)
	at java.io.FilterOutputStream.write(FilterOutputStream.java:77)
	at java.io.FilterOutputStream.write(FilterOutputStream.java:125)
	at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
	at org.apache.commons.io.IOUtils.write(IOUtils.java:1527)
	at org.apache.commons.io.IOUtils.write(IOUtils.java:1552)
	at com.cloudbees.jenkins.support.api.FileContent.writeTo(FileContent.java:122)
	at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:314)
	at com.cloudbees.jenkins.support.SupportAction.doGenerateAllBundles(SupportAction.java:205)
	at java.lang.invoke.LambdaForm$DMH/695376641.invokeVirtual_L3_V(LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$BMH/1360969330.reinvoke(LambdaForm$BMH)
	at java.lang.invoke.LambdaForm$MH/318927722.invoker(LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/1073698081.invokeExact_MT(LambdaForm$MH)
	at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343)
	at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77)
	at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)
	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:130)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
	at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:384)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at com.cloudbees.jenkins.support.impl.cloudbees.UnrestrictedApiCallsMonitor$ApiMonitorFilter.doFilter(UnrestrictedApiCallsMonitor.java:120)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at com.cloudbees.opscenter.client.plugin.OfflineSecurityRealmFilter._doFilter(OfflineSecurityRealmFilter.java:95)
	at com.cloudbees.opscenter.client.plugin.OfflineSecurityRealmFilter.doFilter(OfflineSecurityRealmFilter.java:70)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at com.cloudbees.opscenter.security.ClusterSessionFilter._doFilter(ClusterSessionFilter.java:68)
	at com.cloudbees.opscenter.security.ClusterSessionFilter.doFilter(ClusterSessionFilter.java:43)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:531)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
	at java.lang.Thread.run(Thread.java:748)

ThreadDump 2

"Handling POST /XXXXXXe/support/generateAllBundles from X.X.X.X : qtp2007328737-1361430" #1361430 prio=5 os_prio=0 tid=0x00007efcedfc0000 nid=0x12c4 runnable [0x00007efcb3ddf000]
   java.lang.Thread.State: RUNNABLE
	at com.cloudbees.jenkins.support.util.WordReplacer.replaceWord(WordReplacer.java:133)
	at com.cloudbees.jenkins.support.util.WordReplacer.replaceWords(WordReplacer.java:67)
	at com.cloudbees.jenkins.support.util.WordReplacer.replaceWordsIgnoreCase(WordReplacer.java:51)
	at com.cloudbees.jenkins.support.util.WordReplacer.replaceWordsIgnoreCase(WordReplacer.java:30)
	at com.cloudbees.jenkins.support.filter.SensitiveContentFilter.filter(SensitiveContentFilter.java:66)
	at com.cloudbees.jenkins.support.filter.AllContentFilters.filter(AllContentFilters.java:43)
	at com.cloudbees.jenkins.support.api.FileContent.writeTo(FileContent.java:121)
	at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:314)
	at com.cloudbees.jenkins.support.SupportAction.doGenerateAllBundles(SupportAction.java:205)
	at java.lang.invoke.LambdaForm$DMH/695376641.invokeVirtual_L3_V(LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$BMH/1360969330.reinvoke(LambdaForm$BMH)
	at java.lang.invoke.LambdaForm$MH/318927722.invoker(LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/1073698081.invokeExact_MT(LambdaForm$MH)
	at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343)
	at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77)
	at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)
	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:130)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
	at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:384)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at com.cloudbees.jenkins.support.impl.cloudbees.UnrestrictedApiCallsMonitor$ApiMonitorFilter.doFilter(UnrestrictedApiCallsMonitor.java:120)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at com.cloudbees.opscenter.client.plugin.OfflineSecurityRealmFilter._doFilter(OfflineSecurityRealmFilter.java:95)
	at com.cloudbees.opscenter.client.plugin.OfflineSecurityRealmFilter.doFilter(OfflineSecurityRealmFilter.java:70)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at com.cloudbees.opscenter.security.ClusterSessionFilter._doFilter(ClusterSessionFilter.java:68)
	at com.cloudbees.opscenter.security.ClusterSessionFilter.doFilter(ClusterSessionFilter.java:43)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:531)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
	at java.lang.Thread.run(Thread.java:748)

ThreadDump 3

"Handling POST /XXXX/support/generateAllBundles from X.X.X.X : qtp2007328737-1361430" #1361430 prio=5 os_prio=0 tid=0x00007efcedfc0000 nid=0x12c4 runnable [0x00007efcb3ddf000]
   java.lang.Thread.State: RUNNABLE
	at com.cloudbees.jenkins.support.util.WordReplacer.replaceWord(WordReplacer.java:133)
	at com.cloudbees.jenkins.support.util.WordReplacer.replaceWords(WordReplacer.java:67)
	at com.cloudbees.jenkins.support.util.WordReplacer.replaceWordsIgnoreCase(WordReplacer.java:51)
	at com.cloudbees.jenkins.support.util.WordReplacer.replaceWordsIgnoreCase(WordReplacer.java:30)
	at com.cloudbees.jenkins.support.filter.SensitiveContentFilter.filter(SensitiveContentFilter.java:66)
	at com.cloudbees.jenkins.support.filter.AllContentFilters.filter(AllContentFilters.java:43)
	at com.cloudbees.jenkins.support.api.FileContent.writeTo(FileContent.java:121)
	at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:314)
	at com.cloudbees.jenkins.support.SupportAction.doGenerateAllBundles(SupportAction.java:205)

ThreadDump 4

Handling POST /XXXX/support/generateAllBundles from X.X.X.X : qtp2007328737-1361430" #1361430 prio=5 os_prio=0 tid=0x00007efcedfc0000 nid=0x12c4 runnable [0x00007efcb3ddf000]
   java.lang.Thread.State: RUNNABLE
	at hudson.ExtensionList.get(ExtensionList.java:149)
	at com.cloudbees.jenkins.support.filter.ContentMappings.get(ContentMappings.java:68)
	at com.cloudbees.jenkins.support.filter.SensitiveContentFilter.filter(SensitiveContentFilter.java:56)
	at com.cloudbees.jenkins.support.filter.AllContentFilters.filter(AllContentFilters.java:43)
	at com.cloudbees.jenkins.support.filter.FilteredWriter.filterFlushLines(FilteredWriter.java:137)
	at com.cloudbees.jenkins.support.filter.FilteredWriter.write(FilteredWriter.java:100)
	- locked <0x0000000744dd6d90> (a com.cloudbees.jenkins.support.filter.FilteredWriter)
	at java.io.FilterWriter.write(FilterWriter.java:91)
	at java.io.Writer.write(Writer.java:157)
	at java.io.PrintWriter.newLine(PrintWriter.java:480)
	- locked <0x0000000744dd6d78> (a com.cloudbees.jenkins.support.util.IgnoreCloseWriter)
	at java.io.PrintWriter.println(PrintWriter.java:629)
	at com.cloudbees.jenkins.support.impl.LoadStats$CsvContent.printTo(LoadStats.java:238)
	at com.cloudbees.jenkins.support.api.PrintedContent.writeTo(PrintedContent.java:56)
	at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:316)
	at com.cloudbees.jenkins.support.SupportAction.doGenerateAllBundles(SupportAction.java:205)
	at java.lang.invoke.LambdaForm$DMH/695376641.invokeVirtual_L3_V(LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$BMH/1360969330.reinvoke(LambdaForm$BMH)

ThreadDump 5

"Handling POST /XXX/support/generateAllBundles from X.X.X.X : qtp2007328737-1361430" #1361430 prio=5 os_prio=0 tid=0x00007efcedfc0000 nid=0x12c4 runnable [0x00007efcb3ddf000]
   java.lang.Thread.State: RUNNABLE
	at com.cloudbees.jenkins.support.util.WordReplacer.replaceWord(WordReplacer.java:133)
	at com.cloudbees.jenkins.support.util.WordReplacer.replaceWords(WordReplacer.java:67)
	at com.cloudbees.jenkins.support.util.WordReplacer.replaceWordsIgnoreCase(WordReplacer.java:51)
	at com.cloudbees.jenkins.support.util.WordReplacer.replaceWordsIgnoreCase(WordReplacer.java:30)
	at com.cloudbees.jenkins.support.filter.SensitiveContentFilter.filter(SensitiveContentFilter.java:66)
	at com.cloudbees.jenkins.support.filter.AllContentFilters.filter(AllContentFilters.java:43)
	at com.cloudbees.jenkins.support.api.FileContent.writeTo(FileContent.java:130)
	at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:314)
	at com.cloudbees.jenkins.support.SupportAction.doGenerateAllBundles(SupportAction.java:205)
	at java.lang.invoke.LambdaForm$DMH/695376641.invokeVirtual_L3_V(LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$BMH/1360969330.reinvoke(LambdaForm$BMH)
	at java.lang.invoke.LambdaForm$MH/318927722.invoker(LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/1073698081.invokeExact_MT(LambdaForm$MH)
	at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)

relates to

JENKINS-64401 Performance issues in support-core plugin when task logs contains big text files

Resolved

JENKINS-59030 Include only the latest lines of log files (configurable)

Open

Baptiste Mathus added a comment - 2019-09-30 12:34

Hola fbelzunc! Can you please provide the version of the support-core plugin you were using for these tests? And can you retry on the latest version? Many improvements were done in last versions around performance, and hence I think taking a new test is important so we focus on what is still remaining, if anything.
Gracias!

Baptiste Mathus added a comment - 2019-09-30 12:34 Hola fbelzunc ! Can you please provide the version of the support-core plugin you were using for these tests? And can you retry on the latest version? Many improvements were done in last versions around performance, and hence I think taking a new test is important so we focus on what is still remaining, if anything. Gracias!

Arnaud Héritier added a comment - 2019-10-03 12:46

FYI allan_burdajewicz

Arnaud Héritier added a comment - 2019-10-03 12:46 FYI allan_burdajewicz

Ramon Leon added a comment - 2019-10-08 10:55

Sorry fbelzunc there is no enough information here to investigate if there is some issue with the anonymization. It's a heavy operation where all the content written to the zip is parsed to replace sensitive information, so it's usual to see such numbers of CPU usage.

Without any further data, I recommend:

Remove all unneeded log files from the instance to avoid them to be parsed and included in the bundle
Update to the latest version of support-core because we have done some improvements regarding the performance

Please update the ticket with more information to be able to reproduce this behavior (configuration, number of users, jobs, labels, folders, ... in the instance) or any interesting information we can further analyze.

I will keep it open for a while to allow you to provide the requested information or just close it if you agree.

Thanks.

Ramon Leon added a comment - 2019-10-08 10:55 Sorry fbelzunc there is no enough information here to investigate if there is some issue with the anonymization. It's a heavy operation where all the content written to the zip is parsed to replace sensitive information, so it's usual to see such numbers of CPU usage. Without any further data, I recommend: Remove all unneeded log files from the instance to avoid them to be parsed and included in the bundle Update to the latest version of support-core because we have done some improvements regarding the performance Please update the ticket with more information to be able to reproduce this behavior (configuration, number of users, jobs, labels, folders, ... in the instance) or any interesting information we can further analyze. I will keep it open for a while to allow you to provide the requested information or just close it if you agree. Thanks.

Allan BURDAJEWICZ added a comment - 2023-05-10 02:50 - edited

I think some of it has been addressed by https://issues.jenkins.io/browse/JENKINS-64401. Another problem I found is the SensitiveContentFilter that is recreating the same lists over and over at every call to ContentFilter#filter: https://github.com/jenkinsci/support-core-plugin/blob/master/src/main/java/com/cloudbees/jenkins/support/filter/SensitiveContentFilter.java#L61-L64. We should either "cache" those lists or provide a way to pass the ContentMappings to the WordReplacer. Not sure how impactful that is in terms of resource usage and exec time.

Allan BURDAJEWICZ added a comment - 2023-05-10 02:50 - edited I think some of it has been addressed by https://issues.jenkins.io/browse/JENKINS-64401 . Another problem I found is the SensitiveContentFilter that is recreating the same lists over and over at every call to ContentFilter#filter : https://github.com/jenkinsci/support-core-plugin/blob/master/src/main/java/com/cloudbees/jenkins/support/filter/SensitiveContentFilter.java#L61-L64 . We should either "cache" those lists or provide a way to pass the ContentMappings to the WordReplacer. Not sure how impactful that is in terms of resource usage and exec time.

Allan BURDAJEWICZ added a comment - 2023-06-19 03:54

Considerable improvement made in https://github.com/jenkinsci/support-core-plugin/releases/tag/1315.v0f514a_a_7e133 using java Pattern with trie-based regexp approach. See details in https://github.com/jenkinsci/support-core-plugin/pull/473#issue-1751950151

Allan BURDAJEWICZ added a comment - 2023-06-19 03:54 Considerable improvement made in https://github.com/jenkinsci/support-core-plugin/releases/tag/1315.v0f514a_a_7e133 using java Pattern with trie-based regexp approach. See details in https://github.com/jenkinsci/support-core-plugin/pull/473#issue-1751950151

Assignee:: Allan BURDAJEWICZ

Reporter:: Félix Belzunce Arcos

Votes:: 1 Vote for this issue

Watchers:: 7 Start watching this issue

Created:: 2019-04-22 12:22

Updated:: 2023-06-19 03:54

Resolved:: 2023-06-19 03:53

Jenkins

Details

Description

ThreadDump 1

ThreadDump 2

ThreadDump 3

ThreadDump 4

ThreadDump 5

Attachments

Issue Links

Activity

Collapse comment: Baptiste Mathus added a comment - 2019-09-30 12:34

Expand comment: Baptiste Mathus added a comment - 2019-09-30 12:34

Collapse comment: Arnaud Héritier added a comment - 2019-10-03 12:46

Expand comment: Arnaud Héritier added a comment - 2019-10-03 12:46

Collapse comment: Ramon Leon added a comment - 2019-10-08 10:55

Expand comment: Ramon Leon added a comment - 2019-10-08 10:55

Collapse comment: Allan BURDAJEWICZ added a comment - 2023-05-10 02:50, Edited by Allan BURDAJEWICZ - 2023-05-10 03:12

Expand comment: Allan BURDAJEWICZ added a comment - 2023-05-10 02:50, Edited by Allan BURDAJEWICZ - 2023-05-10 03:12

Collapse comment: Allan BURDAJEWICZ added a comment - 2023-06-19 03:54

Expand comment: Allan BURDAJEWICZ added a comment - 2023-06-19 03:54

People

Dates