Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65811

Script Security causes bottleneck on caffeine cache

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Hi!

      After unsuccessful try to upgrade jenkins core and plugins to 2.277 and revert back to 2.263.4 controller cannot serveĀ  workload: 100+ starting pipelines are in blocked state waiting to UberClassLoader monitor.

      We see thousands of messages like:

      took 2,623ms to load/not load groovy/lang/GroovyObject$java$lang$org$apache$commons$csv$env$NeedsTestScenarioCheck.groovy from classLoader hudson.PluginManager$UberClassLoader 

      builds freezes on pipeline load for hours, hangs on flyweight executors, dependent pipelines are freezed on agent executors, and finally everythin goes to the full degradation with web interface inaccessibility.

      Virtual machine resource consumption are low, pipeline threads are in futex waiting state for 90% of time:

      "Running CpsFlowExecution[Owner[Proj/multibranch/repo/release%2F2021.4.2/1:Proj/multibranch/repo/release%2F2021.4.2 #1]] loading groovy.lang.GroovyObject$groovy$lang$org$apache$common
      s$csv$pipelineConfig$SONAR_PROJECT_KEY" #1288 daemon prio=5 os_prio=0 tid=0x00007f3f08008000 nid=0x45c2 waiting for monitor entry [0x00007f3ee33f3000]
         java.lang.Thread.State: BLOCKED (on object monitor)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:398)
              - waiting to lock <0x00000002ae412670> (a hudson.PluginManager$UberClassLoader)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$loadClass$0(SandboxResolvingClassLoader.java:51)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$489/1150668795.get(Unknown Source)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$load$2(SandboxResolvingClassLoader.java:87)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$495/2071728173.apply(Unknown Source)
              at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139)
              at com.github.benmanes.caffeine.cache.LocalCache$$Lambda$492/1806809958.apply(Unknown Source)
              at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2341)
              at com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$493/1672544621.apply(Unknown Source)
              at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
              - locked <0x000000045ca825d8> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)
              at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2339)
              at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2322)
              at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
              at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.load(SandboxResolvingClassLoader.java:81)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.loadClass(SandboxResolvingClassLoader.java:49)
              - locked <0x00000002e82c5a18> (a org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
              - locked <0x00000002e82c5a60> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)
              at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
              - locked <0x00000002e82c5aa8> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)
              at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
              at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
              - locked <0x00000002e82c5b18> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)
              at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
              - locked <0x00000002e822e248> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)
              at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
              at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545)
              at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185)
              at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170)
              at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126)
              at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676)
              at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313)
              at org.codehaus.groovy.control.ResolveVisitor.resolveNestedClass(ResolveVisitor.java:341)
              at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:308)
              at org.codehaus.groovy.control.ResolveVisitor.resolveFromDefaultImports(ResolveVisitor.java:454)
              at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:311)
              at org.codehaus.groovy.control.ResolveVisitor.resolveFromStaticInnerClasses(ResolveVisitor.java:420)
              at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:312)
              at org.codehaus.groovy.control.ResolveVisitor.resolveFromModule(ResolveVisitor.java:638)
              at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:309)
              at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:276)
              at org.codehaus.groovy.control.ResolveVisitor.transformPropertyExpression(ResolveVisitor.java:833)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:696)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.GStringExpression.transformExpression(GStringExpression.java:59)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformBinaryExpression(ResolveVisitor.java:1033)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:700)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
              at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
              at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
              at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
              at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
              at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
              at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
              at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
              at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:104)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:115)
              at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitConstructorOrMethod(ClassCodeExpressionTransformer.java:53)
              at org.codehaus.groovy.control.ResolveVisitor.visitConstructorOrMethod(ResolveVisitor.java:201)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitMethod(ClassCodeVisitorSupport.java:126)
              at org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1081)
              at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:53)
              at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1279)
              at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176)
              at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663)
              at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
              at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
              at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554)
              at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
              at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
              - locked <0x00000002e82c6300> (a java.util.HashMap)
              at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
              at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761)
              - locked <0x00000002e82c6300> (a java.util.HashMap)
              at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718)
              at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
              at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:775)
              at org.jenkinsci.plugins.workflow.cps.global.UserDefinedGlobalVariable.getValue(UserDefinedGlobalVariable.java:57)
              at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:113)
              at sun.reflect.GeneratedMethodAccessor561.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
              at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
              at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
              at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
              at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
              at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
              at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
              at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142)
              at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
              at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
              at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
              at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
              at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
              at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
              at sun.reflect.GeneratedMethodAccessor358.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
              at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
              at com.cloudbees.groovy.cps.Next.step(Next.java:83)
              at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
              at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
              at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
              at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
              at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
              at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
              at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:136)
              at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
              at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748) 

        Attachments

          Issue Links

            Activity

            brainsam Alexander Moiseenko created issue -
            brainsam Alexander Moiseenko made changes -
            Field Original Value New Value
            Attachment Screenshot 2021-06-06 at 17.12.49.png [ 54939 ]
            Attachment photo_2021-06-06_16-59-44.jpg [ 54940 ]
            brainsam Alexander Moiseenko made changes -
            Description Hi!

            After unsuccessful try to upgrade jenkins core and plugins to 2.277 and revert back to 2.263.4 controller cannot serve  workload: 100+ starting pipelines are in blocked state waiting to UberClasLoader monitor.

            We see thousands of messages like:
            {noformat}
            took 2,623ms to load/not load groovy/lang/GroovyObject$java$lang$org$apache$commons$csv$env$NeedsTestScenarioCheck.groovy from classLoader hudson.PluginManager$UberClassLoader {noformat}
            builds freezes on pipeline load for hours, hangs on flyweight executors, dependent pipelines are freezed on agent executors, and finally everythin goes to the full degradation with web interface inaccessibility.

            Virtual machine resource consumption are low, pipeline threads are in futex waiting state for 90% of time:
            {noformat}
            "Running CpsFlowExecution[Owner[SomeProject/multibranch/SomeRepo/release%2F2021.4.2/1:SomeProject/multibranch/SomeRepo/release%2F2021.4.2 #1]] loading groovy.lang.GroovyObject$groovy$lang$org$apache$commons$csv$pipelineConfig$SONAR_PROJECT_KEY" #1288 daemon prio=5 os_prio=0 tid=0x00007f3f08008000 nid=0x45c2 waiting for monitor entry [0x00007f3ee33f3000]   java.lang.Thread.State: BLOCKED (on object monitor)        at java.lang.ClassLoader.loadClass(ClassLoader.java:398)        - waiting to lock <0x00000002ae412670> (a hudson.PluginManager$UberClassLoader)        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$loadClass$0(SandboxResolvingClassLoader.java:51)        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$489/1150668795.get(Unknown Source)        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$load$2(SandboxResolvingClassLoader.java:87)        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$495/2071728173.apply(Unknown Source)        at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139)        at com.github.benmanes.caffeine.cache.LocalCache$$Lambda$492/1806809958.apply(Unknown Source)        at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2341)        at com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$493/1672544621.apply(Unknown Source)        at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)        - locked <0x000000045ca825d8> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)        at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2339)        at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2322)        at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)        at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.load(SandboxResolvingClassLoader.java:81)        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.loadClass(SandboxResolvingClassLoader.java:49)        - locked <0x00000002e82c5a18> (a org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader)        at java.lang.ClassLoader.loadClass(ClassLoader.java:405)        - locked <0x00000002e82c5a60> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)        at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)        at java.lang.ClassLoader.loadClass(ClassLoader.java:405)        - locked <0x00000002e82c5aa8> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)        at java.lang.ClassLoader.loadClass(ClassLoader.java:405)        - locked <0x00000002e82c5b18> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)        at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)        at java.lang.ClassLoader.loadClass(ClassLoader.java:405)        - locked <0x00000002e822e248> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545)        at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185)        at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170)        at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126)        at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676)        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313)        at org.codehaus.groovy.control.ResolveVisitor.resolveNestedClass(ResolveVisitor.java:341)        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:308)        at org.codehaus.groovy.control.ResolveVisitor.resolveFromDefaultImports(ResolveVisitor.java:454)        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:311)        at org.codehaus.groovy.control.ResolveVisitor.resolveFromStaticInnerClasses(ResolveVisitor.java:420)        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:312)        at org.codehaus.groovy.control.ResolveVisitor.resolveFromModule(ResolveVisitor.java:638)        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:309)        at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:276)        at org.codehaus.groovy.control.ResolveVisitor.transformPropertyExpression(ResolveVisitor.java:833)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:696)        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)        at org.codehaus.groovy.ast.expr.GStringExpression.transformExpression(GStringExpression.java:59)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)        at org.codehaus.groovy.control.ResolveVisitor.transformBinaryExpression(ResolveVisitor.java:1033)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:700)        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)        at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)        at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)        at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)        at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)        at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)        at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)        at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)        at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)        at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:104)        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:115)        at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitConstructorOrMethod(ClassCodeExpressionTransformer.java:53)        at org.codehaus.groovy.control.ResolveVisitor.visitConstructorOrMethod(ResolveVisitor.java:201)        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitMethod(ClassCodeVisitorSupport.java:126)        at org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1081)        at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:53)        at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1279)        at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176)        at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663)        at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)        at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)        at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554)        at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)        - locked <0x00000002e82c6300> (a java.util.HashMap)        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)        at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761)        - locked <0x00000002e82c6300> (a java.util.HashMap)        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718)        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)        at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:775)        at org.jenkinsci.plugins.workflow.cps.global.UserDefinedGlobalVariable.getValue(UserDefinedGlobalVariable.java:57)        at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:113)        at sun.reflect.GeneratedMethodAccessor561.invoke(Unknown Source)        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)        at java.lang.reflect.Method.invoke(Method.java:498)        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)        at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)        at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157)        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142)        at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)        at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)        at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)        at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)        at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)        at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)        at sun.reflect.GeneratedMethodAccessor358.invoke(Unknown Source)        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)        at java.lang.reflect.Method.invoke(Method.java:498)        at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)        at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)        at com.cloudbees.groovy.cps.Next.step(Next.java:83)        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)        at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)        at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)        at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)        at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)        at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)        at java.util.concurrent.FutureTask.run(FutureTask.java:266)        at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:136)        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)        at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)        at java.util.concurrent.FutureTask.run(FutureTask.java:266)        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)        at java.lang.Thread.run(Thread.java:748) {noformat}
            Hi!

            After unsuccessful try to upgrade jenkins core and plugins to 2.277 and revert back to 2.263.4 controller cannot serve  workload: 100+ starting pipelines are in blocked state waiting to UberClasLoader monitor.

            We see thousands of messages like:
            {noformat}
            took 2,623ms to load/not load groovy/lang/GroovyObject$java$lang$org$apache$commons$csv$env$NeedsTestScenarioCheck.groovy from classLoader hudson.PluginManager$UberClassLoader {noformat}
            builds freezes on pipeline load for hours, hangs on flyweight executors, dependent pipelines are freezed on agent executors, and finally everythin goes to the full degradation with web interface inaccessibility.

            Virtual machine resource consumption are low, pipeline threads are in futex waiting state for 90% of time:
            {noformat}
            "Running CpsFlowExecution[Owner[Proj/multibranch/repo/release%2F2021.4.2/1:Proj/multibranch/repo/release%2F2021.4.2 #1]] loading groovy.lang.GroovyObject$groovy$lang$org$apache$common
            s$csv$pipelineConfig$SONAR_PROJECT_KEY" #1288 daemon prio=5 os_prio=0 tid=0x00007f3f08008000 nid=0x45c2 waiting for monitor entry [0x00007f3ee33f3000]
               java.lang.Thread.State: BLOCKED (on object monitor)
                    at java.lang.ClassLoader.loadClass(ClassLoader.java:398)
                    - waiting to lock <0x00000002ae412670> (a hudson.PluginManager$UberClassLoader)
                    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$loadClass$0(SandboxResolvingClassLoader.java:51)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$489/1150668795.get(Unknown Source)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$load$2(SandboxResolvingClassLoader.java:87)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$495/2071728173.apply(Unknown Source)
                    at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139)
                    at com.github.benmanes.caffeine.cache.LocalCache$$Lambda$492/1806809958.apply(Unknown Source)
                    at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2341)
                    at com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$493/1672544621.apply(Unknown Source)
                    at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
                    - locked <0x000000045ca825d8> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)
                    at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2339)
                    at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2322)
                    at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
                    at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.load(SandboxResolvingClassLoader.java:81)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.loadClass(SandboxResolvingClassLoader.java:49)
                    - locked <0x00000002e82c5a18> (a org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader)
                    at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
                    - locked <0x00000002e82c5a60> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)
                    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)
                    at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
                    - locked <0x00000002e82c5aa8> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)
                    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
                    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
                    at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
                    - locked <0x00000002e82c5b18> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)
                    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)
                    at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
                    - locked <0x00000002e822e248> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)
                    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
                    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545)
                    at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185)
                    at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170)
                    at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126)
                    at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676)
                    at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313)
                    at org.codehaus.groovy.control.ResolveVisitor.resolveNestedClass(ResolveVisitor.java:341)
                    at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:308)
                    at org.codehaus.groovy.control.ResolveVisitor.resolveFromDefaultImports(ResolveVisitor.java:454)
                    at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:311)
                    at org.codehaus.groovy.control.ResolveVisitor.resolveFromStaticInnerClasses(ResolveVisitor.java:420)
                    at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:312)
                    at org.codehaus.groovy.control.ResolveVisitor.resolveFromModule(ResolveVisitor.java:638)
                    at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:309)
                    at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:276)
                    at org.codehaus.groovy.control.ResolveVisitor.transformPropertyExpression(ResolveVisitor.java:833)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:696)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.GStringExpression.transformExpression(GStringExpression.java:59)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformBinaryExpression(ResolveVisitor.java:1033)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:700)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:104)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:115)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitConstructorOrMethod(ClassCodeExpressionTransformer.java:53)
                    at org.codehaus.groovy.control.ResolveVisitor.visitConstructorOrMethod(ResolveVisitor.java:201)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitMethod(ClassCodeVisitorSupport.java:126)
                    at org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1081)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:53)
                    at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1279)
                    at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176)
                    at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663)
                    at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
                    at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
                    at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554)
                    at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
                    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
                    - locked <0x00000002e82c6300> (a java.util.HashMap)
                    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
                    at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761)
                    - locked <0x00000002e82c6300> (a java.util.HashMap)
                    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718)
                    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
                    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:775)
                    at org.jenkinsci.plugins.workflow.cps.global.UserDefinedGlobalVariable.getValue(UserDefinedGlobalVariable.java:57)
                    at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:113)
                    at sun.reflect.GeneratedMethodAccessor561.invoke(Unknown Source)
                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                    at java.lang.reflect.Method.invoke(Method.java:498)
                    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
                    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
                    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
                    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
                    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
                    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
                    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
                    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
                    at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142)
                    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
                    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
                    at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
                    at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
                    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
                    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
                    at sun.reflect.GeneratedMethodAccessor358.invoke(Unknown Source)
                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                    at java.lang.reflect.Method.invoke(Method.java:498)
                    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
                    at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
                    at com.cloudbees.groovy.cps.Next.step(Next.java:83)
                    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
                    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
                    at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
                    at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
                    at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
                    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
                    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
                    at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
                    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
                    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
                    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
                    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
                    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:136)
                    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
                    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
                    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                    at java.lang.Thread.run(Thread.java:748) {noformat}
            brainsam Alexander Moiseenko made changes -
            Description Hi!

            After unsuccessful try to upgrade jenkins core and plugins to 2.277 and revert back to 2.263.4 controller cannot serve  workload: 100+ starting pipelines are in blocked state waiting to UberClasLoader monitor.

            We see thousands of messages like:
            {noformat}
            took 2,623ms to load/not load groovy/lang/GroovyObject$java$lang$org$apache$commons$csv$env$NeedsTestScenarioCheck.groovy from classLoader hudson.PluginManager$UberClassLoader {noformat}
            builds freezes on pipeline load for hours, hangs on flyweight executors, dependent pipelines are freezed on agent executors, and finally everythin goes to the full degradation with web interface inaccessibility.

            Virtual machine resource consumption are low, pipeline threads are in futex waiting state for 90% of time:
            {noformat}
            "Running CpsFlowExecution[Owner[Proj/multibranch/repo/release%2F2021.4.2/1:Proj/multibranch/repo/release%2F2021.4.2 #1]] loading groovy.lang.GroovyObject$groovy$lang$org$apache$common
            s$csv$pipelineConfig$SONAR_PROJECT_KEY" #1288 daemon prio=5 os_prio=0 tid=0x00007f3f08008000 nid=0x45c2 waiting for monitor entry [0x00007f3ee33f3000]
               java.lang.Thread.State: BLOCKED (on object monitor)
                    at java.lang.ClassLoader.loadClass(ClassLoader.java:398)
                    - waiting to lock <0x00000002ae412670> (a hudson.PluginManager$UberClassLoader)
                    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$loadClass$0(SandboxResolvingClassLoader.java:51)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$489/1150668795.get(Unknown Source)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$load$2(SandboxResolvingClassLoader.java:87)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$495/2071728173.apply(Unknown Source)
                    at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139)
                    at com.github.benmanes.caffeine.cache.LocalCache$$Lambda$492/1806809958.apply(Unknown Source)
                    at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2341)
                    at com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$493/1672544621.apply(Unknown Source)
                    at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
                    - locked <0x000000045ca825d8> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)
                    at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2339)
                    at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2322)
                    at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
                    at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.load(SandboxResolvingClassLoader.java:81)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.loadClass(SandboxResolvingClassLoader.java:49)
                    - locked <0x00000002e82c5a18> (a org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader)
                    at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
                    - locked <0x00000002e82c5a60> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)
                    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)
                    at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
                    - locked <0x00000002e82c5aa8> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)
                    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
                    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
                    at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
                    - locked <0x00000002e82c5b18> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)
                    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)
                    at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
                    - locked <0x00000002e822e248> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)
                    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
                    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545)
                    at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185)
                    at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170)
                    at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126)
                    at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676)
                    at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313)
                    at org.codehaus.groovy.control.ResolveVisitor.resolveNestedClass(ResolveVisitor.java:341)
                    at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:308)
                    at org.codehaus.groovy.control.ResolveVisitor.resolveFromDefaultImports(ResolveVisitor.java:454)
                    at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:311)
                    at org.codehaus.groovy.control.ResolveVisitor.resolveFromStaticInnerClasses(ResolveVisitor.java:420)
                    at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:312)
                    at org.codehaus.groovy.control.ResolveVisitor.resolveFromModule(ResolveVisitor.java:638)
                    at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:309)
                    at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:276)
                    at org.codehaus.groovy.control.ResolveVisitor.transformPropertyExpression(ResolveVisitor.java:833)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:696)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.GStringExpression.transformExpression(GStringExpression.java:59)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformBinaryExpression(ResolveVisitor.java:1033)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:700)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:104)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:115)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitConstructorOrMethod(ClassCodeExpressionTransformer.java:53)
                    at org.codehaus.groovy.control.ResolveVisitor.visitConstructorOrMethod(ResolveVisitor.java:201)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitMethod(ClassCodeVisitorSupport.java:126)
                    at org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1081)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:53)
                    at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1279)
                    at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176)
                    at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663)
                    at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
                    at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
                    at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554)
                    at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
                    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
                    - locked <0x00000002e82c6300> (a java.util.HashMap)
                    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
                    at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761)
                    - locked <0x00000002e82c6300> (a java.util.HashMap)
                    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718)
                    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
                    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:775)
                    at org.jenkinsci.plugins.workflow.cps.global.UserDefinedGlobalVariable.getValue(UserDefinedGlobalVariable.java:57)
                    at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:113)
                    at sun.reflect.GeneratedMethodAccessor561.invoke(Unknown Source)
                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                    at java.lang.reflect.Method.invoke(Method.java:498)
                    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
                    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
                    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
                    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
                    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
                    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
                    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
                    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
                    at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142)
                    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
                    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
                    at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
                    at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
                    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
                    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
                    at sun.reflect.GeneratedMethodAccessor358.invoke(Unknown Source)
                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                    at java.lang.reflect.Method.invoke(Method.java:498)
                    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
                    at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
                    at com.cloudbees.groovy.cps.Next.step(Next.java:83)
                    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
                    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
                    at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
                    at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
                    at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
                    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
                    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
                    at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
                    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
                    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
                    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
                    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
                    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:136)
                    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
                    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
                    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                    at java.lang.Thread.run(Thread.java:748) {noformat}
            Hi!

            After unsuccessful try to upgrade jenkins core and plugins to 2.277 and revert back to 2.263.4 controller cannot serve  workload: 100+ starting pipelines are in blocked state waiting to UberClassLoader monitor.

            We see thousands of messages like:
            {noformat}
            took 2,623ms to load/not load groovy/lang/GroovyObject$java$lang$org$apache$commons$csv$env$NeedsTestScenarioCheck.groovy from classLoader hudson.PluginManager$UberClassLoader {noformat}
            builds freezes on pipeline load for hours, hangs on flyweight executors, dependent pipelines are freezed on agent executors, and finally everythin goes to the full degradation with web interface inaccessibility.

            Virtual machine resource consumption are low, pipeline threads are in futex waiting state for 90% of time:
            {noformat}
            "Running CpsFlowExecution[Owner[Proj/multibranch/repo/release%2F2021.4.2/1:Proj/multibranch/repo/release%2F2021.4.2 #1]] loading groovy.lang.GroovyObject$groovy$lang$org$apache$common
            s$csv$pipelineConfig$SONAR_PROJECT_KEY" #1288 daemon prio=5 os_prio=0 tid=0x00007f3f08008000 nid=0x45c2 waiting for monitor entry [0x00007f3ee33f3000]
               java.lang.Thread.State: BLOCKED (on object monitor)
                    at java.lang.ClassLoader.loadClass(ClassLoader.java:398)
                    - waiting to lock <0x00000002ae412670> (a hudson.PluginManager$UberClassLoader)
                    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$loadClass$0(SandboxResolvingClassLoader.java:51)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$489/1150668795.get(Unknown Source)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.lambda$load$2(SandboxResolvingClassLoader.java:87)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$$Lambda$495/2071728173.apply(Unknown Source)
                    at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139)
                    at com.github.benmanes.caffeine.cache.LocalCache$$Lambda$492/1806809958.apply(Unknown Source)
                    at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2341)
                    at com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$493/1672544621.apply(Unknown Source)
                    at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
                    - locked <0x000000045ca825d8> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)
                    at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2339)
                    at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2322)
                    at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
                    at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.load(SandboxResolvingClassLoader.java:81)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader.loadClass(SandboxResolvingClassLoader.java:49)
                    - locked <0x00000002e82c5a18> (a org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader)
                    at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
                    - locked <0x00000002e82c5a60> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)
                    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)
                    at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
                    - locked <0x00000002e82c5aa8> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)
                    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
                    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
                    at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
                    - locked <0x00000002e82c5b18> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader)
                    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)
                    at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
                    - locked <0x00000002e822e248> (a org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader)
                    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
                    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545)
                    at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185)
                    at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170)
                    at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126)
                    at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676)
                    at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313)
                    at org.codehaus.groovy.control.ResolveVisitor.resolveNestedClass(ResolveVisitor.java:341)
                    at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:308)
                    at org.codehaus.groovy.control.ResolveVisitor.resolveFromDefaultImports(ResolveVisitor.java:454)
                    at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:311)
                    at org.codehaus.groovy.control.ResolveVisitor.resolveFromStaticInnerClasses(ResolveVisitor.java:420)
                    at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:312)
                    at org.codehaus.groovy.control.ResolveVisitor.resolveFromModule(ResolveVisitor.java:638)
                    at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:309)
                    at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:276)
                    at org.codehaus.groovy.control.ResolveVisitor.transformPropertyExpression(ResolveVisitor.java:833)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:696)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.GStringExpression.transformExpression(GStringExpression.java:59)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformBinaryExpression(ResolveVisitor.java:1033)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:700)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1058)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704)
                    at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51)
                    at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711)
                    at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1079)
                    at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142)
                    at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
                    at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166)
                    at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1336)
                    at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:104)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:115)
                    at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitConstructorOrMethod(ClassCodeExpressionTransformer.java:53)
                    at org.codehaus.groovy.control.ResolveVisitor.visitConstructorOrMethod(ResolveVisitor.java:201)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitMethod(ClassCodeVisitorSupport.java:126)
                    at org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1081)
                    at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:53)
                    at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1279)
                    at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176)
                    at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663)
                    at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
                    at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
                    at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554)
                    at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
                    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
                    - locked <0x00000002e82c6300> (a java.util.HashMap)
                    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
                    at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761)
                    - locked <0x00000002e82c6300> (a java.util.HashMap)
                    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718)
                    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
                    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:775)
                    at org.jenkinsci.plugins.workflow.cps.global.UserDefinedGlobalVariable.getValue(UserDefinedGlobalVariable.java:57)
                    at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:113)
                    at sun.reflect.GeneratedMethodAccessor561.invoke(Unknown Source)
                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                    at java.lang.reflect.Method.invoke(Method.java:498)
                    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
                    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
                    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
                    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
                    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
                    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
                    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
                    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
                    at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157)
                    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142)
                    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
                    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
                    at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
                    at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
                    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
                    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
                    at sun.reflect.GeneratedMethodAccessor358.invoke(Unknown Source)
                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                    at java.lang.reflect.Method.invoke(Method.java:498)
                    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
                    at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
                    at com.cloudbees.groovy.cps.Next.step(Next.java:83)
                    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
                    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
                    at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
                    at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
                    at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
                    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
                    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
                    at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
                    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
                    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
                    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
                    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
                    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:136)
                    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
                    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
                    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                    at java.lang.Thread.run(Thread.java:748) {noformat}
            teilo James Nord made changes -
            Assignee James Nord [ teilo ]
            brainsam Alexander Moiseenko made changes -
            Link This issue is related to JENKINS-23784 [ JENKINS-23784 ]

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              brainsam Alexander Moiseenko
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated: