-
Bug
-
Resolution: Fixed
-
Blocker
I've noticed issues with jenkins being blocked for several minutes in recent versions (even before 1.490, it started probably like 10 versions in the past).
Here's thread dump showing such deadlocks: https://gist.github.com/4109900
Note that my home page is using the dashboard and several threads are blocked on:
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:638) - waiting to lock <0x5e3afde0> (a hudson.model.RunMap) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:621) at jenkins.model.lazy.AbstractLazyLoadRunMap.getById(AbstractLazyLoadRunMap.java:498) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:472) at hudson.model.AbstractProject.getNearestOldBuild(AbstractProject.java:1025) at hudson.maven.MavenModuleSetBuild.getModuleLastBuilds(MavenModuleSetBuild.java:434) at hudson.maven.MavenModuleSetBuild.getResult(MavenModuleSetBuild.java:189) at hudson.model.Run.getIconColor(Run.java:640) at hudson.plugins.view.dashboard.stats.StatBuilds.getBuildStat(StatBuilds.java:50)
- is duplicated by
-
JENKINS-17571 The main Jenkins dashboard page is slow to reload, seems to cause unusually high CPU load in the process
-
- Resolved
-
- is related to
-
JENKINS-8754 ROADMAP: Improve Start-up Time
-
- Closed
-
[JENKINS-15858] Jenkinks UI slow due to constant build record loading
Got the same problem with 1.489 (running in a tomcat 6). Here is the relevant segment from the thread dump.
Found one Java-level deadlock: ============================= "Handling GET /jenkins//api/xml : http-8080-37": waiting to lock monitor 0x00000000103b4a88 (object 0x0000000780a10130, a hudson.model.RunMap), which is held by "Handling GET /jenkins/ : http-8080-6" "Handling GET /jenkins/ : http-8080-6": waiting to lock monitor 0x0000000011f37498 (object 0x00000007871608c8, a hudson.maven.MavenBuild), which is held by "Executor #0 for master : executing DSODP_branch_DEV-1.3 #144" "Executor #0 for master : executing DSODP_branch_DEV-1.3 #144": waiting to lock monitor 0x00000000103b4a88 (object 0x0000000780a10130, a hudson.model.RunMap), which is held by "Handling GET /jenkins/ : http-8080-6"
Java stack information for the threads listed above: =================================================== "Handling GET /jenkins//api/xml : http-8080-37": at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:638) - waiting to lock <0x0000000780a10130> (a hudson.model.RunMap) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:621) at jenkins.model.lazy.AbstractLazyLoadRunMap.getById(AbstractLazyLoadRunMap.java:498) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:472) at hudson.model.AbstractProject.getNearestOldBuild(AbstractProject.java:1025) at hudson.maven.MavenModuleSetBuild.getModuleLastBuilds(MavenModuleSetBuild.java:434) at hudson.maven.MavenModuleSetBuild.getResult(MavenModuleSetBuild.java:189) at hudson.model.Job.getLastFailedBuild(Job.java:823) at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.kohsuke.stapler.export.MethodProperty.getValue(MethodProperty.java:66) at org.kohsuke.stapler.export.Property.writeTo(Property.java:114) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182) at org.kohsuke.stapler.export.Property.writeValue(Property.java:232) at org.kohsuke.stapler.export.Property.writeValue(Property.java:182) at org.kohsuke.stapler.export.Property.writeValue(Property.java:137) at org.kohsuke.stapler.export.Property.writeTo(Property.java:114) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182) at org.kohsuke.stapler.export.Model.writeTo(Model.java:154) at hudson.model.Api.doXml(Api.java:103) at sun.reflect.GeneratedMethodAccessor391.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:288) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:151) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:90) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:573) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:203) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:573) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:487) at org.kohsuke.stapler.Stapler.service(Stapler.java:164) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:91) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:864) at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:579) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1665) at java.lang.Thread.run(Unknown Source) "Handling GET /jenkins/ : http-8080-6": at hudson.model.Actionable.getActions(Actionable.java:73) - waiting to lock <0x00000007871608c8> (a hudson.maven.MavenBuild) at hudson.model.Actionable.getAction(Actionable.java:117) at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:351) at hudson.model.Run.onLoad(Run.java:303) at hudson.model.RunMap.retrieve(RunMap.java:221) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:638) - locked <0x0000000780a10130> (a hudson.model.RunMap) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:621) at jenkins.model.lazy.AbstractLazyLoadRunMap.getById(AbstractLazyLoadRunMap.java:498) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:472) at hudson.model.AbstractProject.getNearestOldBuild(AbstractProject.java:1025) at hudson.maven.MavenModuleSetBuild.getModuleLastBuilds(MavenModuleSetBuild.java:434) at hudson.maven.MavenModuleSetBuild.getResult(MavenModuleSetBuild.java:189) at hudson.model.Job.getLastFailedBuild(Job.java:823) at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125) at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314) at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185) at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75) at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83) at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57) at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51) at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80) at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:72) at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119) at org.kohsuke.stapler.jelly.groovy.JellyBuilder.doInvokeMethod(JellyBuilder.java:283) at org.kohsuke.stapler.jelly.groovy.Namespace$ProxyImpl.invoke(Namespace.java:92) at $Proxy31.projectView(Unknown Source) at lib.JenkinsTagLib$projectView.call(Unknown Source) at hudson.model.View.main.run(main.groovy:14) at org.kohsuke.stapler.jelly.groovy.GroovierJellyScript.run(GroovierJellyScript.java:66) at org.kohsuke.stapler.jelly.groovy.GroovierJellyScript.run(GroovierJellyScript.java:59) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119) at org.kohsuke.stapler.jelly.CompressTag.doTag(CompressTag.java:44) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53) at org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:107) at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:127) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:562) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:624) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:487) at org.kohsuke.stapler.Stapler.service(Stapler.java:164) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:91) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:864) at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:579) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1665) at java.lang.Thread.run(Unknown Source) "Executor #0 for master : executing DSODP_branch_DEV-1.3 #144": at jenkins.model.lazy.AbstractLazyLoadRunMap.removeValue(AbstractLazyLoadRunMap.java:686) - waiting to lock <0x0000000780a10130> (a hudson.model.RunMap) at hudson.model.RunMap.removeValue(RunMap.java:124) at hudson.model.RunMap.remove(RunMap.java:89) at hudson.model.AbstractProject.removeRun(AbstractProject.java:980) at hudson.model.AbstractProject.removeRun(AbstractProject.java:139) at hudson.model.Run.removeRunFromParent(Run.java:1311) at hudson.model.Run.delete(Run.java:1306) - locked <0x00000007871608c8> (a hudson.maven.MavenBuild) at hudson.model.AbstractBuild.delete(AbstractBuild.java:471) - locked <0x00000007871608c8> (a hudson.maven.MavenBuild) at hudson.maven.MavenModuleSetBuild.delete(MavenModuleSetBuild.java:397) - locked <0x0000000786dc0ed0> (a hudson.maven.MavenModuleSetBuild) at hudson.tasks.LogRotator.perform(LogRotator.java:157) at hudson.model.Job.logRotate(Job.java:381) at hudson.maven.MavenModuleSet.logRotate(MavenModuleSet.java:752) at hudson.model.Run.execute(Run.java:1581) at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:477) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:236)
This isn't specific to dashboard-view - JENKINS-15929 is the same problem with the jclouds plugin. I've bisected the introduction of the error to 1.489, and am assigning this to kohsuke.
The two complete thread dumps (gists) do not indicate a deadlock, but rather a performance issue with a MavenBuild having been evicted from memory (or never loaded to begin with) now being loaded, and another thread waiting for that. Such issues might be solved by either deleting excessive historical build records; or running with a large enough heap that more build records can be loaded at once; or changing the Dashboard View plugin to avoid looking up old records:
at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:472) at hudson.model.AbstractProject.getNearestOldBuild(AbstractProject.java:1025) at hudson.maven.MavenModuleSetBuild.getModuleLastBuilds(MavenModuleSetBuild.java:434) at hudson.maven.MavenModuleSetBuild.getResult(MavenModuleSetBuild.java:189) at hudson.model.Run.getIconColor(Run.java:640) at hudson.plugins.view.dashboard.stats.StatBuilds.getBuildStat(StatBuilds.java:50)
Here StatBuilds is doing something very bad.
@edeppe your snippet (please attach a complete thread dump in the future) looks to be unrelated—a true deadlock. Please file it separately.
Code changed in jenkins
User: Jesse Glick
Path:
test/src/main/java/org/jvnet/hudson/test/RunLoadCounter.java
http://jenkins-ci.org/commit/jenkins/03ca63c62c1f3e20f405b655bbc610ed21931a86
Log:
Lazy-loading utility test class designed for fix of JENKINS-15858.
Compare: https://github.com/jenkinsci/jenkins/compare/1551ad6e4683...03ca63c62c1f
@jglick I had a look at your PR and it seems good. I don't know if @vmassol can do it otherwise I'll try to test it on my instance this WE
@aheritier field testing would be great! My fix is based on a test only.
FTR I still have a lot of performance issues with 1.519. I don't use the dashboard plugin anymore since I thought it was causing the issue but apparently it's not since I have hangs on almost all UI views. Very very frequently a view will take as much as 5 minutes before displying making our jenkins UI almost unusable (http://ci.xwiki.org).
I've just taken a thread dump of jenkins when I tried to access http://ci.xwiki.org/job/xwiki-platform%20Quality%20Checks/569/ which took about 3 minutes to display, see attached jstack-vmassol-20130625.txt
And 2 more thread dumps when UI views are slow to load:
jstack-vmassol-20130625-2.txt
jstack-vmassol-20130625-3.txt
FTR, note that this morning we've had to restart our Jenkins (http://ci.xwiki.org) because of OOM:
Exception: org.apache.commons.jelly.JellyTagException: jar:file:/home/maven/.hudson/war/WEB-INF/lib/jenkins-core-1.519.jar!/lib/layout/layout.jelly:85:72: <st:include> java.lang.OutOfMemoryError: PermGen space
PermGen space out of space should be tracked separately. If you think you've given sufficient perm gen heap size and Jenkins is overusing permgen space, please follow this guide and obtain the heap dump and send it to us offline.
I looked at the additional thread dumps vmassol-20130625*.txt and I think Jesse's earlier analysis still applies — they are showing that while the UI thread is stuck, it's busy loading records from the disk.
I see that three thread dumps involve two jobs "xwiki-platform Quality Checks" and "xwiki-platform". They are both Maven projects with lots of modules, so I can imagine that if the cache is cold, this can result in a considerable delay. When I access this instance, I experience that the first page load time of those two jobs are considerable, yet if I reload the page it renders quickly enough.
Between these and the perm gen problem, I suspect that JVM in question simplify doesn't have enough heap size to keep the cache warm enough. Again, I'd love to see the heap dump that I requested above to see if there's something wasting the heap.
Another thought that occurred to me is if it helps to provide an option to make the build records a strong reference, instead of the weak reference. It shifts the memory pressure from build records to other soft references, but for some users it might be an useful trade off.
Updated the title of the ticket to reflect the status.
There's no dead lock involved.
@vmassol you can try https://buildhive.cloudbees.com/job/jenkinsci/job/dashboard-view-plugin/18/org.jenkins-ci.plugins$dashboard-view/artifact/org.jenkins-ci.plugins/dashboard-view/2.7-SNAPSHOT/dashboard-view-2.7-SNAPSHOT.hpi if you want to still use the Dashboard View plugin.
Code changed in jenkins
User: Jesse Glick
Path:
test/src/main/java/org/jvnet/hudson/test/RunLoadCounter.java
http://jenkins-ci.org/commit/jenkins-test-harness/9dc74d4132ad4e40874c638dc3f1773c770d87e0
Log:
Lazy-loading utility test class designed for fix of JENKINS-15858.
Originally-Committed-As: 03ca63c62c1f3e20f405b655bbc610ed21931a86
Same thing on my side : https://gist.github.com/4109911 (1.491)