Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61078

Groovy based parser thread stuck, causing high CPU load on master

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • warnings-ng-plugin

      We have defined custom Groovy based warning parsers, for example to parse custom Matlab Simulink warnings output.

      When adding the following text snippet to the "Example Log Message" of our custom Matlab Warnings parser, we observe excessively high CPU load on the Jenkins master. This load comes from a thread which seems to be stuck and is displayed by JavaMelody as:

      • Handling POST /descriptorByName/io.jenkins.plugins.analysis.warnings.groovy.GroovyParser/checkExample from [ip address removed] : qtp2052001577-666216

      This thread (and possible sub-threads) is constantly consuming CPU time, easily causing something between 50-90% CPU load on our Jenkins master. Rendering the system almost unable to process something else properly.

      Workaround: Killing the thread via JavaMelody reduces the CPU load to normal (around 5-10%).

      The thread reappears once the Jenkins configuration page is opened again, e.g. https://jenkins/configure

      This bug probably first occured after upgrading Jenkins LTS and the Warnings NG plugin sometime in October/November 2019. Unfortunately, I cannot tell anymore which versions were involved, as it took until now to realize the root cause of the sporadic high CPU load of our master...

      And yes, this Groovy script and/or the regex does not work properly. But still, Jenkins should not run into such issues in this case.

      Regular Expression

      ^\[.(Warning|Info|Caused by):(.*\n.*)*?\].\n\[.>(.*\n.*)*?\]

      Mapping Script

       

      import edu.hm.hafner.analysis.Severity
      String type = matcher.group(1)
      String message = matcher.group(2) + "\n" + matcher.group(3)
      // Severity has: ERROR, WARNING_HIGH, WARNING_NORMAL, WARNING_LOW
      switch (type) {
       case 'Warning':
       severity = Severity.ERROR
       break
       case 'Info':
       severity = Severity.WARNING_HIGH
       break
       case 'Caused by':
       severity = Severity.WARNING_LOW
       break
       default:
       severity = Severity.WARNING_NORMAL // unknown, should not happen
       break
      }
      String filename = ''
      if (message =~ m/file ["'](.+)['"]/i) {
       filename = $1
      }
      return builder.setSeverity(severity)
       .setType(type)
       .setFileName(filename)
       .setMessage(message)
       .buildOptional()
      

       

      Example Log Message

      [ Warning: Minimum '0' and maximum '255' from 'Path/ModelElement' are mapped to a portion of the wide
      signal coming from output port 1 of 'Path/SubPath/Switch' and have been ignored during range checking. To
      enable the range checking for the minimum and maximum values, insert a Signal Conversion block with its
      'Output' parameter set to 'Signal copy' in front of 'Path/ModelElement'.] 
      [ > In slbus_get_struct
       In sl (line 15)
       In Simulink.Bus.createObject>createObjectFromBlks (line 169)
       In Simulink.Bus.createObject (line 149)
       In CqsModelCreateBusObjectsMissing/createBusObjects (line 301)
       In CqsModelCreateBusObjectsMissing/processWholeModel (line 86)
       In run_05_prepareImpModelStart (line 142)] 

      Java Thread Dump

      "Handling POST /descriptorByName/io.jenkins.plugins.analysis.warnings.groovy.GroovyParser/checkExample from [ip address removed] : qtp2052001577-803368" prio=5 RUNNABLE
      	java.util.regex.Pattern$GroupTail.match(Unknown Source)
      	java.util.regex.Pattern$Curly.match0(Unknown Source)
      	java.util.regex.Pattern$Curly.match(Unknown Source)
      	java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)
      	java.util.regex.Pattern$Curly.match0(Unknown Source)
      	java.util.regex.Pattern$Curly.match(Unknown Source)
      	java.util.regex.Pattern$GroupHead.match(Unknown Source)
      	java.util.regex.Pattern$LazyLoop.match(Unknown Source)
      	java.util.regex.Pattern$GroupTail.match(Unknown Source)
      	java.util.regex.Pattern$Curly.match0(Unknown Source)
      	java.util.regex.Pattern$Curly.match(Unknown Source)
      	java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)
      	java.util.regex.Pattern$Curly.match0(Unknown Source)
      	java.util.regex.Pattern$Curly.match(Unknown Source)
      	java.util.regex.Pattern$GroupHead.match(Unknown Source)
      	java.util.regex.Pattern$LazyLoop.match(Unknown Source)
      	java.util.regex.Pattern$GroupTail.match(Unknown Source)
      	java.util.regex.Pattern$Curly.match0(Unknown Source)
      	java.util.regex.Pattern$Curly.match(Unknown Source)
      	java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)
      	java.util.regex.Pattern$Curly.match0(Unknown Source)
      	java.util.regex.Pattern$Curly.match(Unknown Source)
      	java.util.regex.Pattern$GroupHead.match(Unknown Source)
      	java.util.regex.Pattern$LazyLoop.match(Unknown Source)
      	java.util.regex.Pattern$GroupTail.match(Unknown Source)
      	java.util.regex.Pattern$Curly.match0(Unknown Source)
      	java.util.regex.Pattern$Curly.match(Unknown Source)
      	java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)
      	java.util.regex.Pattern$Curly.match0(Unknown Source)
      	java.util.regex.Pattern$Curly.match(Unknown Source)
      	java.util.regex.Pattern$GroupHead.match(Unknown Source)
      	java.util.regex.Pattern$LazyLoop.match(Unknown Source)
      	java.util.regex.Pattern$GroupTail.match(Unknown Source)
      	java.util.regex.Pattern$Curly.match0(Unknown Source)
      	java.util.regex.Pattern$Curly.match(Unknown Source)
      	java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)
      	java.util.regex.Pattern$Curly.match0(Unknown Source)
      	java.util.regex.Pattern$Curly.match(Unknown Source)
      	java.util.regex.Pattern$GroupHead.match(Unknown Source)
      	java.util.regex.Pattern$LazyLoop.match(Unknown Source)
      	java.util.regex.Pattern$GroupTail.match(Unknown Source)
      	java.util.regex.Pattern$Curly.match0(Unknown Source)
      	java.util.regex.Pattern$Curly.match(Unknown Source)
      	java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)
      	java.util.regex.Pattern$Curly.match0(Unknown Source)
      	java.util.regex.Pattern$Curly.match(Unknown Source)
      	java.util.regex.Pattern$GroupHead.match(Unknown Source)
      	java.util.regex.Pattern$LazyLoop.match(Unknown Source)
      	java.util.regex.Pattern$GroupTail.match(Unknown Source)
      	java.util.regex.Pattern$Curly.match0(Unknown Source)
      	java.util.regex.Pattern$Curly.match(Unknown Source)
      	java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)
      	java.util.regex.Pattern$Curly.match0(Unknown Source)
      	java.util.regex.Pattern$Curly.match(Unknown Source)
      	java.util.regex.Pattern$GroupHead.match(Unknown Source)
      	java.util.regex.Pattern$LazyLoop.match(Unknown Source)
      	java.util.regex.Pattern$GroupTail.match(Unknown Source)
      	java.util.regex.Pattern$Curly.match0(Unknown Source)
      	java.util.regex.Pattern$Curly.match(Unknown Source)
      	java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)
      	java.util.regex.Pattern$Curly.match0(Unknown Source)
      	java.util.regex.Pattern$Curly.match(Unknown Source)
      	java.util.regex.Pattern$GroupHead.match(Unknown Source)
      	java.util.regex.Pattern$LazyLoop.match(Unknown Source)
      	java.util.regex.Pattern$GroupTail.match(Unknown Source)
      	java.util.regex.Pattern$Curly.match0(Unknown Source)
      	java.util.regex.Pattern$Curly.match(Unknown Source)
      	java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)
      	java.util.regex.Pattern$Curly.match0(Unknown Source)
      	java.util.regex.Pattern$Curly.match(Unknown Source)
      	java.util.regex.Pattern$GroupHead.match(Unknown Source)
      	java.util.regex.Pattern$LazyLoop.match(Unknown Source)
      	java.util.regex.Pattern$GroupTail.match(Unknown Source)
      	java.util.regex.Pattern$Curly.match0(Unknown Source)
      	java.util.regex.Pattern$Curly.match(Unknown Source)
      	java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)
      	java.util.regex.Pattern$Curly.match0(Unknown Source)
      	java.util.regex.Pattern$Curly.match(Unknown Source)
      	java.util.regex.Pattern$GroupHead.match(Unknown Source)
      	java.util.regex.Pattern$LazyLoop.matchInit(Unknown Source)
      	java.util.regex.Pattern$Prolog.match(Unknown Source)
      	java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)
      	java.util.regex.Pattern$GroupTail.match(Unknown Source)
      	java.util.regex.Pattern$BranchConn.match(Unknown Source)
      	java.util.regex.Pattern$Slice.match(Unknown Source)
      	java.util.regex.Pattern$Branch.match(Unknown Source)
      	java.util.regex.Pattern$GroupHead.match(Unknown Source)
      	java.util.regex.Pattern$CharProperty.match(Unknown Source)
      	java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)
      	java.util.regex.Pattern$Caret.match(Unknown Source)
      	java.util.regex.Pattern$Start.match(Unknown Source)
      	java.util.regex.Matcher.search(Unknown Source)
      	java.util.regex.Matcher.find(Unknown Source)
      	io.jenkins.plugins.analysis.warnings.groovy.GroovyParser$DescriptorImpl.parseExample(GroovyParser.java:381)
      	io.jenkins.plugins.analysis.warnings.groovy.GroovyParser$DescriptorImpl.doCheckExample(GroovyParser.java:343)
      	java.lang.invoke.LambdaForm$DMH/1622225974.invokeVirtual_L4_L(LambdaForm$DMH)
      	java.lang.invoke.LambdaForm$BMH/822603816.reinvoke(LambdaForm$BMH)
      	java.lang.invoke.LambdaForm$MH/2096875522.invoker(LambdaForm$MH)
      	java.lang.invoke.LambdaForm$MH/2072658356.invokeExact_MT(LambdaForm$MH)
      	java.lang.invoke.MethodHandle.invokeWithArguments(Unknown Source)
      	org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
      	org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
      	org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77)
      	org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
      	org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
      	org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
      	org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535)
      	org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)
      	org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)
      	org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:280)
      	org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)
      	org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)
      	org.kohsuke.stapler.Stapler.invoke(Stapler.java:676)
      	org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      	javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      	org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
      	org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
      	hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      	jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76)
      	hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
      	hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
      	net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
      	net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
      	org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114)
      	hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:42)
      	hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
      	hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)
      	hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
      	org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
      	hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99)
      	org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
      	hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118)
      	hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      	hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      	hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
      	hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      	hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      	hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
      	hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      	org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
      	org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      	org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
      	hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      	org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
      	org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      	org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
      	org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
      	org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
      	org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
      	org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      	org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
      	org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700)
      	org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
      	org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
      	org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
      	org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
      	org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667)
      	org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
      	org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
      	org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
      	org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      	org.eclipse.jetty.server.Server.handle(Server.java:505)
      	org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
      	org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
      	org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
      	org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
      	org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
      	org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
      	org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
      	org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
      	org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
      	org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
      	org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
      	org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804)
      	java.lang.Thread.run(Unknown Source)

            Unassigned Unassigned
            dbachran Daniel Bachran
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: