Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65811

Script Security causes bottleneck on caffeine cache

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Hi!

      After unsuccessful try to upgrade jenkins core and plugins to 2.277 and revert back to 2.263.4 controller cannot serve  workload: 100+ starting pipelines are in blocked state waiting to UberClassLoader monitor.

      We see thousands of messages like:

      took 2,623ms to load/not load groovy/lang/GroovyObject$java$lang$org$apache$commons$csv$env$NeedsTestScenarioCheck.groovy from classLoader hudson.PluginManager$UberClassLoader 

      builds freezes on pipeline load for hours, hangs on flyweight executors, dependent pipelines are freezed on agent executors, and finally everythin goes to the full degradation with web interface inaccessibility.

      Virtual machine resource consumption are low, pipeline threads are in futex waiting state for 90% of time:

      "Running CpsFlowExecution[Owner[Proj/multibranch/repo/release%2F2021.4.2/1:Proj/multibranch/repo/release%2F2021.4.2 #1]] loading groovy.lang.GroovyObject$groovy$lang$org$apache$common
      s$csv$pipelineConfig$SONAR_PROJECT_KEY" #1288 daemon prio=5 os_prio=0 tid=0x00007f3f08008000 nid=0x45c2 waiting for monitor entry [0x00007f3ee33f3000]
         java.lang.Thread.State: BLOCKED (on object monitor)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:398)
              - waiting to lock <0x00000002ae412670> (a hudson.PluginManager$UberClassLoader)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$loadClass$0(SandboxResolvingClassLoader.java:51)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$489/1150668795.get(Unknown Source)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$load$2(SandboxResolvingClassLoader.java:87)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$495/2071728173.apply(Unknown Source)
              at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139)
              at com.github.benmanes.caffeine.cache.LocalCache$$Lambda$492/1806809958.apply(Unknown Source)
              at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2341)
              at com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$493/1672544621.apply(Unknown Source)
              at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
              - locked <0x000000045ca825d8> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)
              at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2339)
              at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2322)
              at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
              at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.load(SandboxResolvingClassLoader.java:81)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.loadClass(SandboxResolvingClassLoader.java:49)
              - locked <0x00000002e82c5a18> (a org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
              - locked <0x00000002e82c5a60> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)
              at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
              - locked <0x00000002e82c5aa8> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)
              at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
              at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
              - locked <0x00000002e82c5b18> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)
              at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
              - locked <0x00000002e822e248> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)
              at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
              at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545)
              at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185)
              at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170)
              at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126)
              at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676)
              at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313)
              at org.codehaus.groovy.control.ResolveVisitor.resolveNestedClass(ResolveVisitor.java:341)
              at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:308)
              at org.codehaus.groovy.control.ResolveVisitor.resolveFromDefaultImports(ResolveVisitor.java:454)
              at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:311)
              at org.codehaus.groovy.control.ResolveVisitor.resolveFromStaticInnerClasses(ResolveVisitor.java:420)
              at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:312)
              at org.codehaus.groovy.control.ResolveVisitor.resolveFromModule(ResolveVisitor.java:638)
              at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:309)
              at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:276)
              at org.codehaus.groovy.control.ResolveVisitor.transformPropertyExpression(ResolveVisitor.java:833)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:696)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.GStringExpression.transformExpression(GStringExpression.java:59)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformBinaryExpression(ResolveVisitor.java:1033)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:700)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:104)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:115)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitConstructorOrMethod(ClassCodeExpressionTransformer.java:53)
              at org.codehaus.groovy.control.ResolveVisitor.visitConstructorOrMethod(ResolveVisitor.java:201)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitMethod(ClassCodeVisitorSupport.java:126)
              at org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1081)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:53)
              at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1279)
              at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176)
              at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663)
              at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
              at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
              at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554)
              at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
              at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
              - locked <0x00000002e82c6300> (a java.util.HashMap)
              at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
              at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761)
              - locked <0x00000002e82c6300> (a java.util.HashMap)
              at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718)
              at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
              at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:775)
              at org.jenkinsci.plugins.workflow.cps.global.UserDefinedGlobalVariable.getValue(UserDefinedGlobalVariable.java:57)
              at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:113)
              at sun.reflect.GeneratedMethodAccessor561.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
              at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
              at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
              at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
              at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
              at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
              at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
              at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142)
              at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
              at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
              at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
              at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
              at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
              at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
              at sun.reflect.GeneratedMethodAccessor358.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
              at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
              at com.cloudbees.groovy.cps.Next.step(Next.java:83)
              at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
              at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
              at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
              at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
              at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
              at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
              at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:136)
              at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
              at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748) 

        Attachments

          Issue Links

            Activity

            Hide
            brainsam Alexander Moiseenko added a comment -

            we clear Jenkins queue to prevent growth of blocked thread count as workaround and move projects to another jenkins

            Show
            brainsam Alexander Moiseenko added a comment - we clear Jenkins queue to prevent growth of blocked thread count as workaround and move projects to another jenkins
            Hide
            brainsam Alexander Moiseenko added a comment -

            We get caffeine cache stats from groovy script (thx Devin Nusbaum):

            import com.github.benmanes.caffeine.cache.stats.CacheStats;
            import org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader;
            ClassLoader loader = Jenkins.getInstance().getPluginManager().uberClassLoader;
            CacheStats stats = SandboxResolvingClassLoader.parentClassCache.get(loader).stats();
            println("penalty: " + stats.averageLoadPenalty());
            println("hitrate: " + stats.hitRate());
            println("missrate: " + stats.missRate());
            println("load success count: " + stats.loadSuccessCount());
            println("load failure count: " + stats.loadFailureCount());
            println("eviction count: " + stats.evictionCount());
            
            hitrate: 0.8598333830236945
            missrate: 0.14016661697630556
            load success count: 5849871
            load failure count: 0
            eviction count: 5775403
            

            I'm not sure but it seems our loadSuccessCount and evictionCount values says that we need to increase cache size from default 10_000 to 100_000.
            Also hitRate falls down below 0.7 when blocked trhread count grows.

             

            Maybe it would be useful for some cases to have ability to configure caffeine cache size and eviction time on Jenkins configuration/CasC

            Show
            brainsam Alexander Moiseenko added a comment - We get caffeine cache stats from groovy script (thx  Devin Nusbaum ): import com.github.benmanes.caffeine.cache.stats.CacheStats; import org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader; ClassLoader loader = Jenkins.getInstance().getPluginManager().uberClassLoader; CacheStats stats = SandboxResolvingClassLoader.parentClassCache.get(loader).stats(); println( "penalty: " + stats.averageLoadPenalty()); println( "hitrate: " + stats.hitRate()); println( "missrate: " + stats.missRate()); println( "load success count: " + stats.loadSuccessCount()); println( "load failure count: " + stats.loadFailureCount()); println( "eviction count: " + stats.evictionCount()); hitrate: 0.8598333830236945 missrate: 0.14016661697630556 load success count: 5849871 load failure count: 0 eviction count: 5775403 I'm not sure but it seems our loadSuccessCount and evictionCount values says that we need to increase cache size from default 10_000 to 100_000. Also hitRate falls down below 0.7 when blocked trhread count grows.   Maybe it would be useful for some cases to have ability to configure caffeine cache size and eviction time on Jenkins configuration/CasC

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              brainsam Alexander Moiseenko
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated: