[JENKINS-65811] Script Security causes bottleneck on caffeine cache

Type: Bug
Resolution: Unresolved
Priority: Minor
Component/s: caffeine-api-plugin, script-security-plugin
Labels:
- performance
Environment:
jenkins 2.263.4
script-security-plugin 1.77
caffeine-api 2.9.1-23.v51c4e2c879c8

Similar Issues:
Powered by SuggestiMate

Show

Hi!

After unsuccessful try to upgrade jenkins core and plugins to 2.277 and revert back to 2.263.4 controller cannot serve workload: 100+ starting pipelines are in blocked state waiting to UberClassLoader monitor.

We see thousands of messages like:

took 2,623ms to load/not load groovy/lang/GroovyObject$java$lang$org$apache$commons$csv$env$NeedsTestScenarioCheck.groovy from classLoader hudson.PluginManager$UberClassLoader

builds freezes on pipeline load for hours, hangs on flyweight executors, dependent pipelines are freezed on agent executors, and finally everythin goes to the full degradation with web interface inaccessibility.

Virtual machine resource consumption are low, pipeline threads are in futex waiting state for 90% of time:

"Running CpsFlowExecution[Owner[Proj/multibranch/repo/release%2F2021.4.2/1:Proj/multibranch/repo/release%2F2021.4.2 #1]] loading groovy.lang.GroovyObject$groovy$lang$org$apache$common
s$csv$pipelineConfig$SONAR_PROJECT_KEY" #1288 daemon prio=5 os_prio=0 tid=0x00007f3f08008000 nid=0x45c2 waiting for monitor entry [0x00007f3ee33f3000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:398)
        - waiting to lock <0x00000002ae412670> (a hudson.PluginManager$UberClassLoader)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$loadClass$0(SandboxResolvingClassLoader.java:51)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$489/1150668795.get(Unknown Source)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$load$2(SandboxResolvingClassLoader.java:87)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$495/2071728173.apply(Unknown Source)
        at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139)
        at com.github.benmanes.caffeine.cache.LocalCache$$Lambda$492/1806809958.apply(Unknown Source)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2341)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$493/1672544621.apply(Unknown Source)
        at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
        - locked <0x000000045ca825d8> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2339)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2322)
        at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
        at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.load(SandboxResolvingClassLoader.java:81)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.loadClass(SandboxResolvingClassLoader.java:49)
        - locked <0x00000002e82c5a18> (a org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
        - locked <0x00000002e82c5a60> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)
        at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
        - locked <0x00000002e82c5aa8> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
        - locked <0x00000002e82c5b18> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)
        at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
        - locked <0x00000002e822e248> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545)
        at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185)
        at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170)
        at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126)
        at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313)
        at org.codehaus.groovy.control.ResolveVisitor.resolveNestedClass(ResolveVisitor.java:341)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:308)
        at org.codehaus.groovy.control.ResolveVisitor.resolveFromDefaultImports(ResolveVisitor.java:454)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:311)
        at org.codehaus.groovy.control.ResolveVisitor.resolveFromStaticInnerClasses(ResolveVisitor.java:420)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:312)
        at org.codehaus.groovy.control.ResolveVisitor.resolveFromModule(ResolveVisitor.java:638)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:309)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:276)
        at org.codehaus.groovy.control.ResolveVisitor.transformPropertyExpression(ResolveVisitor.java:833)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:696)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.GStringExpression.transformExpression(GStringExpression.java:59)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformBinaryExpression(ResolveVisitor.java:1033)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:700)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:104)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:115)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitConstructorOrMethod(ClassCodeExpressionTransformer.java:53)
        at org.codehaus.groovy.control.ResolveVisitor.visitConstructorOrMethod(ResolveVisitor.java:201)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitMethod(ClassCodeVisitorSupport.java:126)
        at org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1081)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:53)
        at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1279)
        at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176)
        at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663)
        at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
        at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
        at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554)
        at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
        - locked <0x00000002e82c6300> (a java.util.HashMap)
        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
        at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761)
        - locked <0x00000002e82c6300> (a java.util.HashMap)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:775)
        at org.jenkinsci.plugins.workflow.cps.global.UserDefinedGlobalVariable.getValue(UserDefinedGlobalVariable.java:57)
        at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:113)
        at sun.reflect.GeneratedMethodAccessor561.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
        at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
        at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142)
        at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
        at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
        at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
        at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
        at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
        at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
        at sun.reflect.GeneratedMethodAccessor358.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
        at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
        at com.cloudbees.groovy.cps.Next.step(Next.java:83)
        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
        at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
        at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
        at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
        at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
        at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:136)
        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

Screenshot 2021-06-06 at 17.12.49.png
376 kB
2021-06-06 14:29
photo_2021-06-06_16-59-44.jpg
149 kB
2021-06-06 14:29

is related to

JENKINS-23784 Allow parallel class loading

In Progress

Alexander Moiseenko created issue - 2021-06-06 14:27

Alexander Moiseenko made changes - 2021-06-06 14:29

Attachment		New: Screenshot 2021-06-06 at 17.12.49.png [ 54939 ]
Attachment		New: photo_2021-06-06_16-59-44.jpg [ 54940 ]

Alexander Moiseenko made changes - 2021-06-06 14:31

Description

Original: Hi!

After unsuccessful try to upgrade jenkins core and plugins to 2.277 and revert back to 2.263.4 controller cannot serve workload: 100+ starting pipelines are in blocked state waiting to UberClasLoader monitor.

We see thousands of messages like:
{noformat}
took 2,623ms to load/not load groovy/lang/GroovyObject$java$lang$org$apache$commons$csv$env$NeedsTestScenarioCheck.groovy from classLoader hudson.PluginManager$UberClassLoader {noformat}
builds freezes on pipeline load for hours, hangs on flyweight executors, dependent pipelines are freezed on agent executors, and finally everythin goes to the full degradation with web interface inaccessibility.

Virtual machine resource consumption are low, pipeline threads are in futex waiting state for 90% of time:
{noformat}
"Running CpsFlowExecution[Owner[SomeProject/multibranch/SomeRepo/release%2F2021.4.2/1:SomeProject/multibranch/SomeRepo/release%2F2021.4.2 #1]] loading groovy.lang.GroovyObject$groovy$lang$org$apache$commons$csv$pipelineConfig$SONAR_PROJECT_KEY" #1288 daemon prio=5 os_prio=0 tid=0x00007f3f08008000 nid=0x45c2 waiting for monitor entry [0x00007f3ee33f3000] java.lang.Thread.State: BLOCKED (on object monitor) at java.lang.ClassLoader.loadClass(ClassLoader.java:398) - waiting to lock <0x00000002ae412670> (a hudson.PluginManager$UberClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$loadClass$0(SandboxResolvingClassLoader.java:51) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$489/1150668795.get(Unknown Source) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$load$2(SandboxResolvingClassLoader.java:87) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$495/2071728173.apply(Unknown Source) at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139) at com.github.benmanes.caffeine.cache.LocalCache$$Lambda$492/1806809958.apply(Unknown Source) at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2341) at com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$493/1672544621.apply(Unknown Source) at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853) - locked <0x000000045ca825d8> (a java.util.concurrent.ConcurrentHashMap$ReservationNode) at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2339) at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2322) at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108) at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.load(SandboxResolvingClassLoader.java:81) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.loadClass(SandboxResolvingClassLoader.java:49) - locked <0x00000002e82c5a18> (a org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:405) - locked <0x00000002e82c5a60> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170) at java.lang.ClassLoader.loadClass(ClassLoader.java:405) - locked <0x00000002e82c5aa8> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787) at java.lang.ClassLoader.loadClass(ClassLoader.java:405) - locked <0x00000002e82c5b18> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170) at java.lang.ClassLoader.loadClass(ClassLoader.java:405) - locked <0x00000002e822e248> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545) at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185) at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170) at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126) at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676) at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313) at org.codehaus.groovy.control.ResolveVisitor.resolveNestedClass(ResolveVisitor.java:341) at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:308) at org.codehaus.groovy.control.ResolveVisitor.resolveFromDefaultImports(ResolveVisitor.java:454) at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:311) at org.codehaus.groovy.control.ResolveVisitor.resolveFromStaticInnerClasses(ResolveVisitor.java:420) at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:312) at org.codehaus.groovy.control.ResolveVisitor.resolveFromModule(ResolveVisitor.java:638) at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:309) at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:276) at org.codehaus.groovy.control.ResolveVisitor.transformPropertyExpression(ResolveVisitor.java:833) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:696) at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51) at org.codehaus.groovy.ast.expr.GStringExpression.transformExpression(GStringExpression.java:59) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711) at org.codehaus.groovy.control.ResolveVisitor.transformBinaryExpression(ResolveVisitor.java:1033) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:700) at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142) at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42) at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166) at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336) at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71) at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704) at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51) at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711) at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702) at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142) at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42) at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166) at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336) at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71) at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704) at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51) at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711) at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702) at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142) at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42) at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166) at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336) at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71) at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704) at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51) at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711) at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702) at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142) at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42) at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166) at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336) at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71) at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704) at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51) at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711) at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702) at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142) at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42) at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166) at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336) at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71) at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704) at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51) at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711) at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702) at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142) at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42) at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166) at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336) at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71) at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704) at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51) at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711) at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702) at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142) at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42) at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166) at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336) at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71) at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704) at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51) at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711) at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702) at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142) at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42) at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166) at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336) at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:104) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:115) at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitConstructorOrMethod(ClassCodeExpressionTransformer.java:53) at org.codehaus.groovy.control.ResolveVisitor.visitConstructorOrMethod(ResolveVisitor.java:201) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitMethod(ClassCodeVisitorSupport.java:126) at org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1081) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:53) at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1279) at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176) at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) - locked <0x00000002e82c6300> (a java.util.HashMap) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254) at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761) - locked <0x00000002e82c6300> (a java.util.HashMap) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:775) at org.jenkinsci.plugins.workflow.cps.global.UserDefinedGlobalVariable.getValue(UserDefinedGlobalVariable.java:57) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:113) at sun.reflect.GeneratedMethodAccessor561.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163) at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165) at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83) at sun.reflect.GeneratedMethodAccessor358.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46) at com.cloudbees.groovy.cps.Next.step(Next.java:83) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:136) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {noformat}

New: Hi!

After unsuccessful try to upgrade jenkins core and plugins to 2.277 and revert back to 2.263.4 controller cannot serve workload: 100+ starting pipelines are in blocked state waiting to UberClasLoader monitor.

We see thousands of messages like:
{noformat}
took 2,623ms to load/not load groovy/lang/GroovyObject$java$lang$org$apache$commons$csv$env$NeedsTestScenarioCheck.groovy from classLoader hudson.PluginManager$UberClassLoader {noformat}
builds freezes on pipeline load for hours, hangs on flyweight executors, dependent pipelines are freezed on agent executors, and finally everythin goes to the full degradation with web interface inaccessibility.

Virtual machine resource consumption are low, pipeline threads are in futex waiting state for 90% of time:
{noformat}
"Running CpsFlowExecution[Owner[Proj/multibranch/repo/release%2F2021.4.2/1:Proj/multibranch/repo/release%2F2021.4.2 #1]] loading groovy.lang.GroovyObject$groovy$lang$org$apache$common
s$csv$pipelineConfig$SONAR_PROJECT_KEY" #1288 daemon prio=5 os_prio=0 tid=0x00007f3f08008000 nid=0x45c2 waiting for monitor entry [0x00007f3ee33f3000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:398)
        - waiting to lock <0x00000002ae412670> (a hudson.PluginManager$UberClassLoader)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$loadClass$0(SandboxResolvingClassLoader.java:51)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$489/1150668795.get(Unknown Source)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$load$2(SandboxResolvingClassLoader.java:87)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$495/2071728173.apply(Unknown Source)
        at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139)
        at com.github.benmanes.caffeine.cache.LocalCache$$Lambda$492/1806809958.apply(Unknown Source)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2341)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$493/1672544621.apply(Unknown Source)
        at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
        - locked <0x000000045ca825d8> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2339)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2322)
        at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
        at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.load(SandboxResolvingClassLoader.java:81)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.loadClass(SandboxResolvingClassLoader.java:49)
        - locked <0x00000002e82c5a18> (a org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
        - locked <0x00000002e82c5a60> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)
        at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
        - locked <0x00000002e82c5aa8> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
        - locked <0x00000002e82c5b18> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)
        at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
        - locked <0x00000002e822e248> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545)
        at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185)
        at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170)
        at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126)
        at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313)
        at org.codehaus.groovy.control.ResolveVisitor.resolveNestedClass(ResolveVisitor.java:341)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:308)
        at org.codehaus.groovy.control.ResolveVisitor.resolveFromDefaultImports(ResolveVisitor.java:454)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:311)
        at org.codehaus.groovy.control.ResolveVisitor.resolveFromStaticInnerClasses(ResolveVisitor.java:420)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:312)
        at org.codehaus.groovy.control.ResolveVisitor.resolveFromModule(ResolveVisitor.java:638)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:309)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:276)
        at org.codehaus.groovy.control.ResolveVisitor.transformPropertyExpression(ResolveVisitor.java:833)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:696)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.GStringExpression.transformExpression(GStringExpression.java:59)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformBinaryExpression(ResolveVisitor.java:1033)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:700)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:104)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:115)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitConstructorOrMethod(ClassCodeExpressionTransformer.java:53)
        at org.codehaus.groovy.control.ResolveVisitor.visitConstructorOrMethod(ResolveVisitor.java:201)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitMethod(ClassCodeVisitorSupport.java:126)
        at org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1081)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:53)
        at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1279)
        at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176)
        at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663)
        at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
        at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
        at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554)
        at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
        - locked <0x00000002e82c6300> (a java.util.HashMap)
        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
        at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761)
        - locked <0x00000002e82c6300> (a java.util.HashMap)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:775)
        at org.jenkinsci.plugins.workflow.cps.global.UserDefinedGlobalVariable.getValue(UserDefinedGlobalVariable.java:57)
        at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:113)
        at sun.reflect.GeneratedMethodAccessor561.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
        at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
        at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142)
        at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
        at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
        at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
        at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
        at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
        at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
        at sun.reflect.GeneratedMethodAccessor358.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
        at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
        at com.cloudbees.groovy.cps.Next.step(Next.java:83)
        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
        at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
        at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
        at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
        at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
        at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:136)
        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748) {noformat}

Alexander Moiseenko added a comment - 2021-06-07 17:21

We get caffeine cache stats from groovy script (thx dnusbaum):

import com.github.benmanes.caffeine.cache.stats.CacheStats;
import org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader;
ClassLoader loader = Jenkins.getInstance().getPluginManager().uberClassLoader;
CacheStats stats = SandboxResolvingClassLoader.parentClassCache.get(loader).stats();
println("penalty: " + stats.averageLoadPenalty());
println("hitrate: " + stats.hitRate());
println("missrate: " + stats.missRate());
println("load success count: " + stats.loadSuccessCount());
println("load failure count: " + stats.loadFailureCount());
println("eviction count: " + stats.evictionCount());

hitrate: 0.8598333830236945
missrate: 0.14016661697630556
load success count: 5849871
load failure count: 0
eviction count: 5775403

I'm not sure but it seems our loadSuccessCount and evictionCount values says that we need to increase cache size from default 10_000 to 100_000.
Also hitRate falls down below 0.7 when blocked trhread count grows.

Maybe it would be useful for some cases to have ability to configure caffeine cache size and eviction time on Jenkins configuration/CasC

Alexander Moiseenko added a comment - 2021-06-07 17:21 We get caffeine cache stats from groovy script (thx dnusbaum ): import com.github.benmanes.caffeine.cache.stats.CacheStats; import org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader; ClassLoader loader = Jenkins.getInstance().getPluginManager().uberClassLoader; CacheStats stats = SandboxResolvingClassLoader.parentClassCache.get(loader).stats(); println( "penalty: " + stats.averageLoadPenalty()); println( "hitrate: " + stats.hitRate()); println( "missrate: " + stats.missRate()); println( "load success count: " + stats.loadSuccessCount()); println( "load failure count: " + stats.loadFailureCount()); println( "eviction count: " + stats.evictionCount()); hitrate: 0.8598333830236945 missrate: 0.14016661697630556 load success count: 5849871 load failure count: 0 eviction count: 5775403 I'm not sure but it seems our loadSuccessCount and evictionCount values says that we need to increase cache size from default 10_000 to 100_000. Also hitRate falls down below 0.7 when blocked trhread count grows. Maybe it would be useful for some cases to have ability to configure caffeine cache size and eviction time on Jenkins configuration/CasC

Alexander Moiseenko made changes - 2021-06-07 20:23

Description

Original: Hi!

After unsuccessful try to upgrade jenkins core and plugins to 2.277 and revert back to 2.263.4 controller cannot serve workload: 100+ starting pipelines are in blocked state waiting to UberClasLoader monitor.

We see thousands of messages like:
{noformat}
took 2,623ms to load/not load groovy/lang/GroovyObject$java$lang$org$apache$commons$csv$env$NeedsTestScenarioCheck.groovy from classLoader hudson.PluginManager$UberClassLoader {noformat}
builds freezes on pipeline load for hours, hangs on flyweight executors, dependent pipelines are freezed on agent executors, and finally everythin goes to the full degradation with web interface inaccessibility.

Virtual machine resource consumption are low, pipeline threads are in futex waiting state for 90% of time:
{noformat}
"Running CpsFlowExecution[Owner[Proj/multibranch/repo/release%2F2021.4.2/1:Proj/multibranch/repo/release%2F2021.4.2 #1]] loading groovy.lang.GroovyObject$groovy$lang$org$apache$common
s$csv$pipelineConfig$SONAR_PROJECT_KEY" #1288 daemon prio=5 os_prio=0 tid=0x00007f3f08008000 nid=0x45c2 waiting for monitor entry [0x00007f3ee33f3000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:398)
        - waiting to lock <0x00000002ae412670> (a hudson.PluginManager$UberClassLoader)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$loadClass$0(SandboxResolvingClassLoader.java:51)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$489/1150668795.get(Unknown Source)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$load$2(SandboxResolvingClassLoader.java:87)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$495/2071728173.apply(Unknown Source)
        at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139)
        at com.github.benmanes.caffeine.cache.LocalCache$$Lambda$492/1806809958.apply(Unknown Source)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2341)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$493/1672544621.apply(Unknown Source)
        at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
        - locked <0x000000045ca825d8> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2339)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2322)
        at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
        at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.load(SandboxResolvingClassLoader.java:81)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.loadClass(SandboxResolvingClassLoader.java:49)
        - locked <0x00000002e82c5a18> (a org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
        - locked <0x00000002e82c5a60> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)
        at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
        - locked <0x00000002e82c5aa8> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
        - locked <0x00000002e82c5b18> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)
        at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
        - locked <0x00000002e822e248> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545)
        at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185)
        at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170)
        at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126)
        at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313)
        at org.codehaus.groovy.control.ResolveVisitor.resolveNestedClass(ResolveVisitor.java:341)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:308)
        at org.codehaus.groovy.control.ResolveVisitor.resolveFromDefaultImports(ResolveVisitor.java:454)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:311)
        at org.codehaus.groovy.control.ResolveVisitor.resolveFromStaticInnerClasses(ResolveVisitor.java:420)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:312)
        at org.codehaus.groovy.control.ResolveVisitor.resolveFromModule(ResolveVisitor.java:638)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:309)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:276)
        at org.codehaus.groovy.control.ResolveVisitor.transformPropertyExpression(ResolveVisitor.java:833)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:696)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.GStringExpression.transformExpression(GStringExpression.java:59)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformBinaryExpression(ResolveVisitor.java:1033)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:700)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:104)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:115)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitConstructorOrMethod(ClassCodeExpressionTransformer.java:53)
        at org.codehaus.groovy.control.ResolveVisitor.visitConstructorOrMethod(ResolveVisitor.java:201)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitMethod(ClassCodeVisitorSupport.java:126)
        at org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1081)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:53)
        at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1279)
        at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176)
        at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663)
        at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
        at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
        at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554)
        at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
        - locked <0x00000002e82c6300> (a java.util.HashMap)
        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
        at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761)
        - locked <0x00000002e82c6300> (a java.util.HashMap)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:775)
        at org.jenkinsci.plugins.workflow.cps.global.UserDefinedGlobalVariable.getValue(UserDefinedGlobalVariable.java:57)
        at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:113)
        at sun.reflect.GeneratedMethodAccessor561.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
        at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
        at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142)
        at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
        at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
        at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
        at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
        at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
        at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
        at sun.reflect.GeneratedMethodAccessor358.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
        at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
        at com.cloudbees.groovy.cps.Next.step(Next.java:83)
        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
        at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
        at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
        at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
        at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
        at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:136)
        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748) {noformat}

New: Hi!

After unsuccessful try to upgrade jenkins core and plugins to 2.277 and revert back to 2.263.4 controller cannot serve workload: 100+ starting pipelines are in blocked state waiting to UberClassLoader monitor.

We see thousands of messages like:
{noformat}
took 2,623ms to load/not load groovy/lang/GroovyObject$java$lang$org$apache$commons$csv$env$NeedsTestScenarioCheck.groovy from classLoader hudson.PluginManager$UberClassLoader {noformat}
builds freezes on pipeline load for hours, hangs on flyweight executors, dependent pipelines are freezed on agent executors, and finally everythin goes to the full degradation with web interface inaccessibility.

Virtual machine resource consumption are low, pipeline threads are in futex waiting state for 90% of time:
{noformat}
"Running CpsFlowExecution[Owner[Proj/multibranch/repo/release%2F2021.4.2/1:Proj/multibranch/repo/release%2F2021.4.2 #1]] loading groovy.lang.GroovyObject$groovy$lang$org$apache$common
s$csv$pipelineConfig$SONAR_PROJECT_KEY" #1288 daemon prio=5 os_prio=0 tid=0x00007f3f08008000 nid=0x45c2 waiting for monitor entry [0x00007f3ee33f3000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:398)
        - waiting to lock <0x00000002ae412670> (a hudson.PluginManager$UberClassLoader)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$loadClass$0(SandboxResolvingClassLoader.java:51)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$489/1150668795.get(Unknown Source)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$load$2(SandboxResolvingClassLoader.java:87)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$495/2071728173.apply(Unknown Source)
        at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139)
        at com.github.benmanes.caffeine.cache.LocalCache$$Lambda$492/1806809958.apply(Unknown Source)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2341)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$493/1672544621.apply(Unknown Source)
        at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
        - locked <0x000000045ca825d8> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2339)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2322)
        at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
        at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.load(SandboxResolvingClassLoader.java:81)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.loadClass(SandboxResolvingClassLoader.java:49)
        - locked <0x00000002e82c5a18> (a org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
        - locked <0x00000002e82c5a60> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)
        at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
        - locked <0x00000002e82c5aa8> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
        - locked <0x00000002e82c5b18> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)
        at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
        - locked <0x00000002e822e248> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545)
        at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185)
        at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170)
        at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126)
        at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313)
        at org.codehaus.groovy.control.ResolveVisitor.resolveNestedClass(ResolveVisitor.java:341)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:308)
        at org.codehaus.groovy.control.ResolveVisitor.resolveFromDefaultImports(ResolveVisitor.java:454)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:311)
        at org.codehaus.groovy.control.ResolveVisitor.resolveFromStaticInnerClasses(ResolveVisitor.java:420)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:312)
        at org.codehaus.groovy.control.ResolveVisitor.resolveFromModule(ResolveVisitor.java:638)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:309)
        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:276)
        at org.codehaus.groovy.control.ResolveVisitor.transformPropertyExpression(ResolveVisitor.java:833)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:696)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.GStringExpression.transformExpression(GStringExpression.java:59)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformBinaryExpression(ResolveVisitor.java:1033)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:700)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:104)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:115)
        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitConstructorOrMethod(ClassCodeExpressionTransformer.java:53)
        at org.codehaus.groovy.control.ResolveVisitor.visitConstructorOrMethod(ResolveVisitor.java:201)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitMethod(ClassCodeVisitorSupport.java:126)
        at org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1081)
        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:53)
        at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1279)
        at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176)
        at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663)
        at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
        at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
        at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554)
        at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
        - locked <0x00000002e82c6300> (a java.util.HashMap)
        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
        at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761)
        - locked <0x00000002e82c6300> (a java.util.HashMap)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:775)
        at org.jenkinsci.plugins.workflow.cps.global.UserDefinedGlobalVariable.getValue(UserDefinedGlobalVariable.java:57)
        at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:113)
        at sun.reflect.GeneratedMethodAccessor561.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
        at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
        at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142)
        at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
        at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
        at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
        at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
        at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
        at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
        at sun.reflect.GeneratedMethodAccessor358.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
        at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
        at com.cloudbees.groovy.cps.Next.step(Next.java:83)
        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
        at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
        at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
        at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
        at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
        at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:136)
        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748) {noformat}

Alexander Moiseenko added a comment - 2021-06-07 20:26

we clear Jenkins queue to prevent growth of blocked thread count as workaround and move projects to another jenkins

Alexander Moiseenko added a comment - 2021-06-07 20:26 we clear Jenkins queue to prevent growth of blocked thread count as workaround and move projects to another jenkins

James Nord made changes - 2021-06-07 20:35

Assignee

Original: James Nord [ teilo ]

Alexander Moiseenko made changes - 2021-09-20 19:47

Link

New: This issue is related to JENKINS-23784 [ JENKINS-23784 ]

Ken black added a comment - 2022-03-20 05:59

brainsam where can i set cache size from default 10_000 to 100_000. i hit this bug.

Ken black added a comment - 2022-03-20 05:59 brainsam where can i set cache size from default 10_000 to 100_000. i hit this bug.

Davide Cavestro added a comment - 2023-01-18 15:51

we exprerienced a similar issue during a Sonar migration
when Sonar was configured with the old url (not reachable)
threads on jenkins master got errors trying to contact it to gather sonar version and presumably acquiring some locks without releasing them (not inspected further though)

Davide Cavestro added a comment - 2023-01-18 15:51 we exprerienced a similar issue during a Sonar migration when Sonar was configured with the old url (not reachable) threads on jenkins master got errors trying to contact it to gather sonar version and presumably acquiring some locks without releasing them (not inspected further though)

Assignee:: Unassigned

Reporter:: Alexander Moiseenko

Votes:: 2 Vote for this issue

Watchers:: 5 Start watching this issue

Created:: 2021-06-06 14:27

Updated:: 2023-06-29 19:25

Jenkins

Details

Description

Attachments

Attachments

Issue Links

Activity

Collapse comment: Alexander Moiseenko added a comment - 2021-06-07 17:21

Expand comment: Alexander Moiseenko added a comment - 2021-06-07 17:21

Collapse comment: Alexander Moiseenko added a comment - 2021-06-07 20:26

Expand comment: Alexander Moiseenko added a comment - 2021-06-07 20:26

Collapse comment: Ken black added a comment - 2022-03-20 05:59

Expand comment: Ken black added a comment - 2022-03-20 05:59

Collapse comment: Davide Cavestro added a comment - 2023-01-18 15:51

Expand comment: Davide Cavestro added a comment - 2023-01-18 15:51

People

Dates