Type: Bug
Resolution: Unresolved
Priority: Major
Component/s: disk-usage-plugin, pipeline-multibranch-defaults-plugin, scm2job-plugin
Labels:
None

Jenkins becomes very slow with the below message in the health-checker.log

The branch indexing for My-Multi-Branch-SNAPSHOT succeeds without issues most of the time. But at times, it gets into a deadlock like below.

Over time, jenkins completes hangs up and we end up restarting to restore the server.

Requesting help in identifying the root cause of the issue and solution for the same.

Jenkins version: 2.7.4

Branch indexing log: It hangs forever

Branch Indexing Log

Started by timer > git rev-parse --is-inside-work-tree # timeout=10 Setting origin to ssh://git@mystash/myproject.git > git config remote.origin.url ssh://git@mystash/myproject.git # timeout=10 Fetching & pruning origin... Fetching upstream changes from origin > git --version # timeout=10 using GIT_SSH to set credentials credentails for GIT > git fetch --tags --progress origin +refs/heads/:refs/remotes/origin/ --prune Getting remote branches... Seen branch in repository origin/develop Seen branch in repository origin/master Seen branch in repository origin/release Seen 3 remote branches Checking branch master No changes detected in master (still at 7c4f090ce09229b6900258a79755a72eb16f5a1e) Checking branch develop

health-checker.log

Starting health checks at Fri Jan 25 04:48:02 PST 2019

Health check results at Fri Jan 25 04:48:02 PST 2019:

* disk-space: Result{isHealthy=true}

* mesos: Result{isHealthy=true}

* plugins: Result{isHealthy=true, message=No failed plugins}

* temporary-space: Result{isHealthy=true}

* thread-deadlock: Result{isHealthy=false, message=[Handling GET /job/ENF/ from 10.111.9.36 : RequestHandlerThread225 View/index.jelly LastSuccessColumn/column.jelly locked on hudson.model.RunMap@1e716e7e (owned by Executor #-1 for master : executing BranchIndexing[My-Multi-Branch-SNAPSHOT]):

at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:356)

at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:233)

at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:978)

at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:144)

at jenkins.model.PeepholePermalink.resolve(PeepholePermalink.java:95)

at hudson.model.Job.getLastSuccessfulBuild(Job.java:909)

at com.github.mjdetullio.jenkins.plugins.multibranch.TemplateDrivenMultiBranchProject.getLastSuccessfulBuild(TemplateDrivenMultiBranchProject.java:406)

at sun.reflect.GeneratedMethodAccessor360.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)

at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)

at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)

at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)

at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)

at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)

at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)

at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)

at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)

at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)

at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)

at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)

at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)

at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)

at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)

at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)

at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)

at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)

at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)

at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)

at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

at org.kohsuke.stapler.jelly.groovy.JellyBuilder.doInvokeMethod(JellyBuilder.java:276)

at org.kohsuke.stapler.jelly.groovy.Namespace$ProxyImpl.invoke(Namespace.java:92)

at com.sun.proxy.$Proxy85.projectView(Unknown Source)

at lib.JenkinsTagLib$projectView$1.call(Unknown Source)

at hudson.model.View.main.run(main.groovy:14)

at org.kohsuke.stapler.jelly.groovy.GroovierJellyScript.run(GroovierJellyScript.java:74)

at org.kohsuke.stapler.jelly.groovy.GroovierJellyScript.run(GroovierJellyScript.java:62)

at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)

at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)

at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)