Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65811

Script Security causes bottleneck on caffeine cache

      Hi!

      After unsuccessful try to upgrade jenkins core and plugins to 2.277 and revert back to 2.263.4 controller cannot serve  workload: 100+ starting pipelines are in blocked state waiting to UberClassLoader monitor.

      We see thousands of messages like:

      took 2,623ms to load/not load groovy/lang/GroovyObject$java$lang$org$apache$commons$csv$env$NeedsTestScenarioCheck.groovy from classLoader hudson.PluginManager$UberClassLoader 

      builds freezes on pipeline load for hours, hangs on flyweight executors, dependent pipelines are freezed on agent executors, and finally everythin goes to the full degradation with web interface inaccessibility.

      Virtual machine resource consumption are low, pipeline threads are in futex waiting state for 90% of time:

      "Running CpsFlowExecution[Owner[Proj/multibranch/repo/release%2F2021.4.2/1:Proj/multibranch/repo/release%2F2021.4.2 #1]] loading groovy.lang.GroovyObject$groovy$lang$org$apache$common
      s$csv$pipelineConfig$SONAR_PROJECT_KEY" #1288 daemon prio=5 os_prio=0 tid=0x00007f3f08008000 nid=0x45c2 waiting for monitor entry [0x00007f3ee33f3000]
         java.lang.Thread.State: BLOCKED (on object monitor)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:398)
              - waiting to lock <0x00000002ae412670> (a hudson.PluginManager$UberClassLoader)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$loadClass$0(SandboxResolvingClassLoader.java:51)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$489/1150668795.get(Unknown Source)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$load$2(SandboxResolvingClassLoader.java:87)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$495/2071728173.apply(Unknown Source)
              at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139)
              at com.github.benmanes.caffeine.cache.LocalCache$$Lambda$492/1806809958.apply(Unknown Source)
              at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2341)
              at com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$493/1672544621.apply(Unknown Source)
              at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
              - locked <0x000000045ca825d8> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)
              at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2339)
              at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2322)
              at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
              at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.load(SandboxResolvingClassLoader.java:81)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.loadClass(SandboxResolvingClassLoader.java:49)
              - locked <0x00000002e82c5a18> (a org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
              - locked <0x00000002e82c5a60> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)
              at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
              - locked <0x00000002e82c5aa8> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)
              at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
              at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
              - locked <0x00000002e82c5b18> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)
              at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
              - locked <0x00000002e822e248> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)
              at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
              at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545)
              at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185)
              at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170)
              at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126)
              at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676)
              at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313)
              at org.codehaus.groovy.control.ResolveVisitor.resolveNestedClass(ResolveVisitor.java:341)
              at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:308)
              at org.codehaus.groovy.control.ResolveVisitor.resolveFromDefaultImports(ResolveVisitor.java:454)
              at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:311)
              at org.codehaus.groovy.control.ResolveVisitor.resolveFromStaticInnerClasses(ResolveVisitor.java:420)
              at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:312)
              at org.codehaus.groovy.control.ResolveVisitor.resolveFromModule(ResolveVisitor.java:638)
              at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:309)
              at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:276)
              at org.codehaus.groovy.control.ResolveVisitor.transformPropertyExpression(ResolveVisitor.java:833)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:696)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.GStringExpression.transformExpression(GStringExpression.java:59)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformBinaryExpression(ResolveVisitor.java:1033)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:700)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:104)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:115)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitConstructorOrMethod(ClassCodeExpressionTransformer.java:53)
              at org.codehaus.groovy.control.ResolveVisitor.visitConstructorOrMethod(ResolveVisitor.java:201)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitMethod(ClassCodeVisitorSupport.java:126)
              at org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1081)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:53)
              at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1279)
              at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176)
              at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663)
              at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
              at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
              at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554)
              at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
              at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
              - locked <0x00000002e82c6300> (a java.util.HashMap)
              at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
              at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761)
              - locked <0x00000002e82c6300> (a java.util.HashMap)
              at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718)
              at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
              at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:775)
              at org.jenkinsci.plugins.workflow.cps.global.UserDefinedGlobalVariable.getValue(UserDefinedGlobalVariable.java:57)
              at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:113)
              at sun.reflect.GeneratedMethodAccessor561.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
              at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
              at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
              at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
              at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
              at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
              at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
              at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142)
              at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
              at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
              at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
              at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
              at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
              at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
              at sun.reflect.GeneratedMethodAccessor358.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
              at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
              at com.cloudbees.groovy.cps.Next.step(Next.java:83)
              at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
              at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
              at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
              at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
              at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
              at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
              at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:136)
              at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
              at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748) 

          [JENKINS-65811] Script Security causes bottleneck on caffeine cache

          We get caffeine cache stats from groovy script (thx dnusbaum):

          import com.github.benmanes.caffeine.cache.stats.CacheStats;
          import org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader;
          ClassLoader loader = Jenkins.getInstance().getPluginManager().uberClassLoader;
          CacheStats stats = SandboxResolvingClassLoader.parentClassCache.get(loader).stats();
          println("penalty: " + stats.averageLoadPenalty());
          println("hitrate: " + stats.hitRate());
          println("missrate: " + stats.missRate());
          println("load success count: " + stats.loadSuccessCount());
          println("load failure count: " + stats.loadFailureCount());
          println("eviction count: " + stats.evictionCount());
          
          hitrate: 0.8598333830236945
          missrate: 0.14016661697630556
          load success count: 5849871
          load failure count: 0
          eviction count: 5775403
          

          I'm not sure but it seems our loadSuccessCount and evictionCount values says that we need to increase cache size from default 10_000 to 100_000.
          Also hitRate falls down below 0.7 when blocked trhread count grows.

           

          Maybe it would be useful for some cases to have ability to configure caffeine cache size and eviction time on Jenkins configuration/CasC

          Alexander Moiseenko added a comment - We get caffeine cache stats from groovy script (thx  dnusbaum ): import com.github.benmanes.caffeine.cache.stats.CacheStats; import org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader; ClassLoader loader = Jenkins.getInstance().getPluginManager().uberClassLoader; CacheStats stats = SandboxResolvingClassLoader.parentClassCache.get(loader).stats(); println( "penalty: " + stats.averageLoadPenalty()); println( "hitrate: " + stats.hitRate()); println( "missrate: " + stats.missRate()); println( "load success count: " + stats.loadSuccessCount()); println( "load failure count: " + stats.loadFailureCount()); println( "eviction count: " + stats.evictionCount()); hitrate: 0.8598333830236945 missrate: 0.14016661697630556 load success count: 5849871 load failure count: 0 eviction count: 5775403 I'm not sure but it seems our loadSuccessCount and evictionCount values says that we need to increase cache size from default 10_000 to 100_000. Also hitRate falls down below 0.7 when blocked trhread count grows.   Maybe it would be useful for some cases to have ability to configure caffeine cache size and eviction time on Jenkins configuration/CasC

          we clear Jenkins queue to prevent growth of blocked thread count as workaround and move projects to another jenkins

          Alexander Moiseenko added a comment - we clear Jenkins queue to prevent growth of blocked thread count as workaround and move projects to another jenkins

          Ken black added a comment -

          brainsam where can i set cache size from default 10_000 to 100_000. i hit this bug.

          Ken black added a comment - brainsam  where can i set cache size from default 10_000 to 100_000. i hit this bug.

          we exprerienced a similar issue during a Sonar migration
          when Sonar was configured with the old url (not reachable)
          threads on jenkins master got errors trying to contact it to gather sonar version and presumably acquiring some locks without releasing them (not inspected further though)

          Davide Cavestro added a comment - we exprerienced a similar issue during a Sonar migration when Sonar was configured with the old url (not reachable) threads on jenkins master got errors trying to contact it to gather sonar version and presumably acquiring some locks without releasing them (not inspected further though)

          Heidi Rechek added a comment -

          Also experience this issue today. 

          Heidi Rechek added a comment - Also experience this issue today. 

            Unassigned Unassigned
            brainsam Alexander Moiseenko
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: