Major A user encountered a hung startup after around a minute of activity. Successive thread dumps showed that initialization was paused here:
"ModelList.init" daemon prio=10 … in Object.wait() … java.lang.Thread.State: RUNNABLE at org.codehaus.groovy.util.ReferenceBundle.<clinit>(ReferenceBundle.java:37) at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.<init>(MetaClassRegistryImpl.java:53) at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.<init>(MetaClassRegistryImpl.java:61) at groovy.lang.GroovySystem.<clinit>(GroovySystem.java:29) at org.codehaus.groovy.runtime.InvokerHelper.<clinit>(InvokerHelper.java:49) at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:32) at groovy.lang.Binding.<init>(Binding.java:34) at groovy.lang.GroovyShell.<init>(GroovyShell.java:82) at com.cloudbees.hudson.plugins.modeling.ModelList.init(ModelList.java:71) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at hudson.init.InitializerFinder.invoke(InitializerFinder.java:120) at hudson.init.InitializerFinder$TaskImpl.run(InitializerFinder.java:184) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259) at jenkins.model.Jenkins$6.runTask(Jenkins.java:840) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636)
The line of code in question (assuming Groovy 1.8.5):
ReferenceManager callBack = ReferenceManager.createCallBackedManager(queue);
There were tons of request threads, mostly hung in something like:
"Handling GET /…some page… : http-8080-2" daemon prio=10 … in Object.wait() … java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002aaabb3400a8> (a com.google.common.collect.ComputingConcurrentHashMap$ComputingValueReference) at java.lang.Object.wait(Object.java:502) at com.google.common.collect.ComputingConcurrentHashMap$ComputingValueReference.waitForValue(ComputingConcurrentHashMap.java:328) - locked <0x00002aaabb3400a8> (a com.google.common.collect.ComputingConcurrentHashMap$ComputingValueReference) at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.getOrCompute(ComputingConcurrentHashMap.java:160) at com.google.common.collect.ComputingConcurrentHashMap.getOrCompute(ComputingConcurrentHashMap.java:69) at com.google.common.collect.ComputingConcurrentHashMap$ComputingMapAdapter.get(ComputingConcurrentHashMap.java:393) at org.kohsuke.stapler.CachingScriptLoader.findScript(CachingScriptLoader.java:61) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:112) …
But one was different:
"Handling GET / : http-8080-1" daemon prio=10 … in Object.wait() … java.lang.Thread.State: RUNNABLE at org.codehaus.groovy.util.ReferenceManager.<clinit>(ReferenceManager.java:153) at org.codehaus.groovy.util.ManagedReference.<clinit>(ManagedReference.java:25) at org.codehaus.groovy.reflection.ReflectionCache.getCachedClass(ReflectionCache.java:107) at org.codehaus.groovy.reflection.ReflectionCache.<clinit>(ReflectionCache.java:52) at org.codehaus.groovy.classgen.asm.BytecodeHelper.box(BytecodeHelper.java:585) at org.codehaus.groovy.classgen.asm.BytecodeHelper.box(BytecodeHelper.java:577) at org.codehaus.groovy.classgen.asm.OperandStack.box(OperandStack.java:183) at org.codehaus.groovy.classgen.asm.InvocationWriter.makeCall(InvocationWriter.java:220) at org.codehaus.groovy.classgen.asm.InvocationWriter.makeInvokeMethodCall(InvocationWriter.java:66) at org.codehaus.groovy.classgen.asm.InvocationWriter.writeInvokeMethod(InvocationWriter.java:292) at org.codehaus.groovy.classgen.AsmClassGenerator.visitMethodCallExpression(AsmClassGenerator.java:655) at org.codehaus.groovy.ast.expr.MethodCallExpression.visit(MethodCallExpression.java:75) at org.codehaus.groovy.classgen.asm.StatementWriter.writeExpressionStatement(StatementWriter.java:599) at org.codehaus.groovy.classgen.asm.OptimizingStatementWriter.writeExpressionStatement(OptimizingStatementWriter.java:354) at org.codehaus.groovy.classgen.AsmClassGenerator.visitExpressionStatement(AsmClassGenerator.java:501) at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:40) at org.codehaus.groovy.classgen.asm.StatementWriter.writeBlockStatement(StatementWriter.java:80) at org.codehaus.groovy.classgen.asm.OptimizingStatementWriter.writeBlockStatement(OptimizingStatementWriter.java:155) at org.codehaus.groovy.classgen.AsmClassGenerator.visitBlockStatement(AsmClassGenerator.java:447) at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:69) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:101) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:112) at org.codehaus.groovy.classgen.AsmClassGenerator.visitStdMethod(AsmClassGenerator.java:312) at org.codehaus.groovy.classgen.AsmClassGenerator.visitConstructorOrMethod(AsmClassGenerator.java:269) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructor(ClassCodeVisitorSupport.java:119) at org.codehaus.groovy.classgen.AsmClassGenerator.visitConstructor(AsmClassGenerator.java:384) at org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1052) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:50) at org.codehaus.groovy.classgen.AsmClassGenerator.visitClass(AsmClassGenerator.java:173) at org.codehaus.groovy.control.CompilationUnit$14.call(CompilationUnit.java:767) at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:967) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:546) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:524) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:501) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:306) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:287) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:267) at org.kohsuke.stapler.jelly.groovy.GroovyClassLoaderTearOff.parse(GroovyClassLoaderTearOff.java:82) at org.kohsuke.stapler.jelly.groovy.GroovyClassTearOff.parseScript(GroovyClassTearOff.java:57) at org.kohsuke.stapler.jelly.groovy.GroovyClassTearOff.parseScript(GroovyClassTearOff.java:46) at org.kohsuke.stapler.AbstractTearOff.resolveScript(AbstractTearOff.java:72) at org.kohsuke.stapler.jelly.JellyClassTearOff.resolveScript(JellyClassTearOff.java:80) at org.kohsuke.stapler.jelly.JellyClassTearOff.resolveScript(JellyClassTearOff.java:47) at org.kohsuke.stapler.AbstractTearOff.loadScript(AbstractTearOff.java:78) at org.kohsuke.stapler.CachingScriptLoader$1.apply(CachingScriptLoader.java:30) at org.kohsuke.stapler.CachingScriptLoader$1.apply(CachingScriptLoader.java:27) at com.google.common.collect.ComputingConcurrentHashMap$ComputingValueReference.compute(ComputingConcurrentHashMap.java:355) at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.compute(ComputingConcurrentHashMap.java:184) - locked <0x00002aaabb340170> (a com.google.common.collect.MapMakerInternalMap$StrongEntry) at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.getOrCompute(ComputingConcurrentHashMap.java:153) at com.google.common.collect.ComputingConcurrentHashMap.getOrCompute(ComputingConcurrentHashMap.java:69) at com.google.common.collect.ComputingConcurrentHashMap$ComputingMapAdapter.get(ComputingConcurrentHashMap.java:393) at org.kohsuke.stapler.CachingScriptLoader.findScript(CachingScriptLoader.java:61) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:112) …
which is in:
SOFT_BUNDLE = new ReferenceBundle(manager, ReferenceType.SOFT);
Seems like a class loading deadlock: two static initializers referring to one another, and two threads each initially referring to one of these classes.
Groovy 2.0.5 has the same code. Cannot find any existing reports of this, so it is probably a rare race condition.