[JENKINS-44958] File handle leaks in *.log after hard kill - Jenkins Jira

XML

Word

Printable

Details

Type: Bug
Status: Resolved (View Workflow)
Priority: Minor
Resolution: Won't Fix
Component/s: workflow-cps-plugin
Labels:
None
Environment:
Jenkins ver. 2.46.3

Similar Issues:

Show

Description

I use a Jenkins Pipeline to run my individual tests in parallel. I just noticed my test job that I run in parallel is failing with "Too many open files" errors.

Next, I ssh into the master and did an lsof -p using the pid of Jenkins... i see over 4k file descriptors. The output has thousands of lines that look like this:

java 7 root 4089w REG 202,96 28 2375888 /var/jenkins_home/jobs/test_pipeline/builds/15331/222.log
java 7 root 4090w REG 202,96 328 787837 /var/jenkins_home/jobs/test_pipeline/builds/15343/166.log
java 7 root 4091w REG 202,96 328 787851 /var/jenkins_home/jobs/test_pipeline/builds/15343/180.log
java 7 root 4094w REG 202,96 671 787839 /var/jenkins_home/jobs/test_pipeline/builds/15343/167.log
java 7 root 4095w REG 202,96 29 2361097 /var/jenkins_home/jobs/test_pipeline/builds/15331/157.log

Each file has content that roughly looks like this:

Starting building: ESC[8mha:////4CQAnoPVmHzYGq9xQUbp+LIf+xF38/zSfY1nZ0iZCPexAAAArB+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3Rdl
LBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAyZEgZF/az8JP2U/Pyi4pLUAt2ULKBh+fElqcUl+sYWliYGhvoAbOU0btYAAAA=ESC[0mmy_test #389401

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

image-2017-06-27-12-41-39-630.png
13 kB
2017-06-27 19:41
image-2017-09-09-09-48-57-939.png
24 kB
2017-09-09 16:48
image-2017-09-09-09-49-15-788.png
23 kB
2017-09-09 16:49

Issue Links

relates to

JENKINS-38381 [JEP-210] Optimize log handling in Pipeline and Durable Task

Resolved

Activity

Ascending order - Click to sort in descending order

Jesse Glick added a comment - 2017-06-22 19:31

Not sure how to reproduce this particular problem (first I have heard of it), but ~~JENKINS-38381~~ would solve it by virtue of never creating these files to begin with.

Jesse Glick added a comment - 2017-06-22 19:31 Not sure how to reproduce this particular problem (first I have heard of it), but JENKINS-38381 would solve it by virtue of never creating these files to begin with.

Jon B added a comment - 2017-06-27 19:42

I've noticed that as long as I preemptively restart Jenkins server at least daily, I won't get an unpredictable crash. The attached graph shows the number of file descriptors open throughout the day and you can see how it leaks up until the point where I manually restart Jenkins:

Is there a better option? I'm not a Jenkins expert by any means.. but perhaps you can advise something I can do within my Jenkins pipeline that will cause java to release the file descriptors after the pipeline has finished.

Jon B added a comment - 2017-06-27 19:42 I've noticed that as long as I preemptively restart Jenkins server at least daily, I won't get an unpredictable crash. The attached graph shows the number of file descriptors open throughout the day and you can see how it leaks up until the point where I manually restart Jenkins: Is there a better option? I'm not a Jenkins expert by any means.. but perhaps you can advise something I can do within my Jenkins pipeline that will cause java to release the file descriptors after the pipeline has finished.

Jon B added a comment - 2017-06-28 22:17 - edited

I did a little more investigation today: It appears this happens when I kill jobs and/or pipelines while they are running.

Jenkins API has an HTTP POST ability to kill in flight jobs with "/kill" at the end of the URL. I use the kill feature with some custom logic that says "Before I start a certain job with input param A and B, first kill any other Jenkins jobs or pipelines which are in flight that are already running with those same A and B inputs."

Therefore, throughout the day, we see a low percentage of kills. It must be for this reason why we are leaking file descriptors all day.

Jon B added a comment - 2017-06-28 22:17 - edited I did a little more investigation today: It appears this happens when I kill jobs and/or pipelines while they are running. Jenkins API has an HTTP POST ability to kill in flight jobs with "/kill" at the end of the URL. I use the kill feature with some custom logic that says "Before I start a certain job with input param A and B, first kill any other Jenkins jobs or pipelines which are in flight that are already running with those same A and B inputs." Therefore, throughout the day, we see a low percentage of kills. It must be for this reason why we are leaking file descriptors all day.

Jesse Glick added a comment - 2017-06-29 20:25

Do not use kill this way!!! This is a last resort when a build is completely unresponsive, and it will not perform proper cleanup.

Still a bug, but a much lower-priority one.

Jesse Glick added a comment - 2017-06-29 20:25 Do not use kill this way!!! This is a last resort when a build is completely unresponsive, and it will not perform proper cleanup. Still a bug, but a much lower-priority one.

Jon B added a comment - 2017-09-09 16:50 - edited

My Jenkins server got wedged at 100% CPU 3 times this week. Despite my not using "kill" anymore, I suspect the reason the master is getting hosed is because there are too many open file handles. I installed jglick's open file handles extension and looked at the output of it when there were no active jobs running starts with this first line:

"391 descriptors are open"

Hm.. that's kind of weird. No jobs running and 391 open file handles .. or descriptors... Let's examine one of them:

#8 /root/.groovy/grapes/org.apache.httpcomponents/httpclient/jars/httpclient-4.5.2.jar by thread:Executor #-1 for master : executing branch_out_of_date_check #67139 on Sat Sep 09 08:41:24 PDT 2017
	at java.util.zip.ZipFile.<init>(ZipFile.java:150)
	at java.util.jar.JarFile.<init>(JarFile.java:166)
	at java.util.jar.JarFile.<init>(JarFile.java:103)
	at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:930)
	at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:791)
	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:876)
	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:869)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:868)
	at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:841)
	at sun.misc.URLClassPath$3.run(URLClassPath.java:565)
	at sun.misc.URLClassPath$3.run(URLClassPath.java:555)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.misc.URLClassPath.getLoader(URLClassPath.java:554)
	at sun.misc.URLClassPath.getLoader(URLClassPath.java:519)
	at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484)
	at sun.misc.URLClassPath.access$100(URLClassPath.java:65)
	at sun.misc.URLClassPath$1.next(URLClassPath.java:266)
	at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:277)
	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)
	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader$3.next(URLClassLoader.java:598)
	at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)
	at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
	at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
	at org.codehaus.groovy.transform.ASTTransformationVisitor.doAddGlobalTransforms(ASTTransformationVisitor.java:201)
	at org.codehaus.groovy.transform.ASTTransformationVisitor.addGlobalTransformsAfterGrab(ASTTransformationVisitor.java:188)
	at groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:381)
	at org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:321)
	at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
	at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:156)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545)
	at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185)
	at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170)
	at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126)
	at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676)
	at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313)
	at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1236)
	at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176)
	at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663)
	at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
	at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:516)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:479)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:419)

Hm... ok.. lets look at another one:

#9 /root/.groovy/grapes/org.apache.httpcomponents/httpcore/jars/httpcore-4.4.4.jar by thread:Executor #-1 for master : executing callbackgateway-nodeploy #195 on Sat Sep 09 09:24:43 PDT 2017
	at java.util.zip.ZipFile.<init>(ZipFile.java:150)
	at java.util.jar.JarFile.<init>(JarFile.java:166)
	at java.util.jar.JarFile.<init>(JarFile.java:103)
	at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:930)
	at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:791)
	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:876)
	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:869)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:868)
	at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:841)
	at sun.misc.URLClassPath$3.run(URLClassPath.java:565)
	at sun.misc.URLClassPath$3.run(URLClassPath.java:555)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.misc.URLClassPath.getLoader(URLClassPath.java:554)
	at sun.misc.URLClassPath.getLoader(URLClassPath.java:519)
	at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484)
	at sun.misc.URLClassPath.access$100(URLClassPath.java:65)
	at sun.misc.URLClassPath$1.next(URLClassPath.java:266)
	at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:277)
	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)
	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader$3.next(URLClassLoader.java:598)
	at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)
	at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
	at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
	at org.codehaus.groovy.transform.ASTTransformationVisitor.doAddGlobalTransforms(ASTTransformationVisitor.java:201)
	at org.codehaus.groovy.transform.ASTTransformationVisitor.addGlobalTransformsAfterGrab(ASTTransformationVisitor.java:188)
	at groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:381)
	at org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:321)
	at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
	at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:156)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545)
	at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185)
	at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170)
	at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126)
	at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676)
	at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313)
	at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1236)
	at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176)
	at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663)
	at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
	at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:516)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:479)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:419)

Here's a 3rd example:

#10 /root/.groovy/grapes/commons-logging/commons-logging/jars/commons-logging-1.2.jar by thread:Executor #-1 for master : executing update_jenkins_slaves_list #12498 on Sat Sep 09 09:23:00 PDT 2017
	at java.util.zip.ZipFile.<init>(ZipFile.java:150)
	at java.util.jar.JarFile.<init>(JarFile.java:166)
	at java.util.jar.JarFile.<init>(JarFile.java:103)
	at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:930)
	at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:791)
	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:876)
	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:869)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:868)
	at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:841)
	at sun.misc.URLClassPath$3.run(URLClassPath.java:565)
	at sun.misc.URLClassPath$3.run(URLClassPath.java:555)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.misc.URLClassPath.getLoader(URLClassPath.java:554)
	at sun.misc.URLClassPath.getLoader(URLClassPath.java:519)
	at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484)
	at sun.misc.URLClassPath.access$100(URLClassPath.java:65)
	at sun.misc.URLClassPath$1.next(URLClassPath.java:266)
	at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:277)
	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)
	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader$3.next(URLClassLoader.java:598)
	at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)
	at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
	at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
	at org.codehaus.groovy.transform.ASTTransformationVisitor.doAddGlobalTransforms(ASTTransformationVisitor.java:201)
	at org.codehaus.groovy.transform.ASTTransformationVisitor.addGlobalTransformsAfterGrab(ASTTransformationVisitor.java:188)
	at groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:381)
	at org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:321)
	at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
	at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:156)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545)
	at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185)
	at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170)
	at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126)
	at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676)
	at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313)
	at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1236)
	at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176)
	at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663)
	at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
	at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:516)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:479)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:419)

Here's one last example.. and this one is the most common one I see in my logs:

#4 /var/jenkins_home/jobs/django-nodeploy/builds/2650/586.log by thread:Running CpsFlowExecution[Owner[django-nodeploy/2650:doorstep-django-nodeploy #2650]] on Fri Sep 08 21:03:54 PDT 2017
	at java.io.FileOutputStream.<init>(FileOutputStream.java:214)
	at org.jenkinsci.plugins.workflow.support.actions.LogActionImpl.stream(LogActionImpl.java:88)
	at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:73)
	at org.jenkinsci.plugins.workflow.steps.StepDescriptor.checkContextAvailability(StepDescriptor.java:258)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:203)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:150)
	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
	at sun.reflect.GeneratedMethodAccessor418.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:123)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:153)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:157)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127)
	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:173)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:162)
	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:162)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:330)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

It is worthwhile noting that my Jenkins installation gets lots of CI traffic.. many developers submit jobs and the master/slave pool is busy during the middle of the afternoon.

I don't have any hard evidence to support my theory, but I believe that when I reach around 3k open file descriptors (as reported by the operating system), the Jenkins master goes crazy and pegs to 100% CPU. The above graphs are from my Jenkins master yesterday. You see at 2PM, it went insane and Jenkins basically become almost totally unresponsive. By around 4:20PM, I decided to go to the Jenkins website "/restart" page and issued a restart. Once I did that, everything resumed normally.

Please advise.

Jon B added a comment - 2017-09-09 16:50 - edited My Jenkins server got wedged at 100% CPU 3 times this week. Despite my not using "kill" anymore, I suspect the reason the master is getting hosed is because there are too many open file handles. I installed jglick 's open file handles extension and looked at the output of it when there were no active jobs running starts with this first line: "391 descriptors are open" Hm.. that's kind of weird. No jobs running and 391 open file handles .. or descriptors... Let's examine one of them: #8 /root/.groovy/grapes/org.apache.httpcomponents/httpclient/jars/httpclient-4.5.2.jar by thread:Executor #-1 for master : executing branch_out_of_date_check #67139 on Sat Sep 09 08:41:24 PDT 2017 at java.util.zip.ZipFile.<init>(ZipFile.java:150) at java.util.jar.JarFile.<init>(JarFile.java:166) at java.util.jar.JarFile.<init>(JarFile.java:103) at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:930) at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:791) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:876) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:869) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:868) at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:841) at sun.misc.URLClassPath$3.run(URLClassPath.java:565) at sun.misc.URLClassPath$3.run(URLClassPath.java:555) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath.getLoader(URLClassPath.java:554) at sun.misc.URLClassPath.getLoader(URLClassPath.java:519) at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484) at sun.misc.URLClassPath.access$100(URLClassPath.java:65) at sun.misc.URLClassPath$1.next(URLClassPath.java:266) at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:277) at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601) at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader$3.next(URLClassLoader.java:598) at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623) at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45) at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54) at org.codehaus.groovy.transform.ASTTransformationVisitor.doAddGlobalTransforms(ASTTransformationVisitor.java:201) at org.codehaus.groovy.transform.ASTTransformationVisitor.addGlobalTransformsAfterGrab(ASTTransformationVisitor.java:188) at groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:381) at org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:321) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254) at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787) at java.lang. ClassLoader .loadClass( ClassLoader .java:411) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:156) at java.lang. ClassLoader .loadClass( ClassLoader .java:411) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545) at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185) at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170) at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126) at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676) at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313) at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1236) at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176) at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:516) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:479) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:419) Hm... ok.. lets look at another one: #9 /root/.groovy/grapes/org.apache.httpcomponents/httpcore/jars/httpcore-4.4.4.jar by thread:Executor #-1 for master : executing callbackgateway-nodeploy #195 on Sat Sep 09 09:24:43 PDT 2017 at java.util.zip.ZipFile.<init>(ZipFile.java:150) at java.util.jar.JarFile.<init>(JarFile.java:166) at java.util.jar.JarFile.<init>(JarFile.java:103) at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:930) at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:791) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:876) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:869) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:868) at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:841) at sun.misc.URLClassPath$3.run(URLClassPath.java:565) at sun.misc.URLClassPath$3.run(URLClassPath.java:555) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath.getLoader(URLClassPath.java:554) at sun.misc.URLClassPath.getLoader(URLClassPath.java:519) at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484) at sun.misc.URLClassPath.access$100(URLClassPath.java:65) at sun.misc.URLClassPath$1.next(URLClassPath.java:266) at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:277) at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601) at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader$3.next(URLClassLoader.java:598) at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623) at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45) at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54) at org.codehaus.groovy.transform.ASTTransformationVisitor.doAddGlobalTransforms(ASTTransformationVisitor.java:201) at org.codehaus.groovy.transform.ASTTransformationVisitor.addGlobalTransformsAfterGrab(ASTTransformationVisitor.java:188) at groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:381) at org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:321) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254) at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787) at java.lang. ClassLoader .loadClass( ClassLoader .java:411) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:156) at java.lang. ClassLoader .loadClass( ClassLoader .java:411) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545) at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185) at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170) at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126) at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676) at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313) at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1236) at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176) at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:516) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:479) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:419) Here's a 3rd example: #10 /root/.groovy/grapes/commons-logging/commons-logging/jars/commons-logging-1.2.jar by thread:Executor #-1 for master : executing update_jenkins_slaves_list #12498 on Sat Sep 09 09:23:00 PDT 2017 at java.util.zip.ZipFile.<init>(ZipFile.java:150) at java.util.jar.JarFile.<init>(JarFile.java:166) at java.util.jar.JarFile.<init>(JarFile.java:103) at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:930) at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:791) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:876) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:869) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:868) at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:841) at sun.misc.URLClassPath$3.run(URLClassPath.java:565) at sun.misc.URLClassPath$3.run(URLClassPath.java:555) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath.getLoader(URLClassPath.java:554) at sun.misc.URLClassPath.getLoader(URLClassPath.java:519) at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484) at sun.misc.URLClassPath.access$100(URLClassPath.java:65) at sun.misc.URLClassPath$1.next(URLClassPath.java:266) at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:277) at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601) at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader$3.next(URLClassLoader.java:598) at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623) at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45) at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54) at org.codehaus.groovy.transform.ASTTransformationVisitor.doAddGlobalTransforms(ASTTransformationVisitor.java:201) at org.codehaus.groovy.transform.ASTTransformationVisitor.addGlobalTransformsAfterGrab(ASTTransformationVisitor.java:188) at groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:381) at org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:321) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254) at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787) at java.lang. ClassLoader .loadClass( ClassLoader .java:411) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:156) at java.lang. ClassLoader .loadClass( ClassLoader .java:411) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545) at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185) at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170) at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126) at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676) at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313) at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1236) at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176) at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:516) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:479) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:419) Here's one last example.. and this one is the most common one I see in my logs: #4 / var /jenkins_home/jobs/django-nodeploy/builds/2650/586.log by thread:Running CpsFlowExecution[Owner[django-nodeploy/2650:doorstep-django-nodeploy #2650]] on Fri Sep 08 21:03:54 PDT 2017 at java.io.FileOutputStream.<init>(FileOutputStream.java:214) at org.jenkinsci.plugins.workflow.support.actions.LogActionImpl.stream(LogActionImpl.java:88) at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:73) at org.jenkinsci.plugins.workflow.steps.StepDescriptor.checkContextAvailability(StepDescriptor.java:258) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:203) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:150) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108) at sun.reflect.GeneratedMethodAccessor418.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155) at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:123) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:153) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:157) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127) at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82) at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46) at com.cloudbees.groovy.cps.Next.step(Next.java:83) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:173) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:162) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:162) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:330) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) It is worthwhile noting that my Jenkins installation gets lots of CI traffic.. many developers submit jobs and the master/slave pool is busy during the middle of the afternoon. I don't have any hard evidence to support my theory, but I believe that when I reach around 3k open file descriptors (as reported by the operating system), the Jenkins master goes crazy and pegs to 100% CPU. The above graphs are from my Jenkins master yesterday. You see at 2PM, it went insane and Jenkins basically become almost totally unresponsive. By around 4:20PM, I decided to go to the Jenkins website "/restart" page and issued a restart. Once I did that, everything resumed normally. Please advise.

Jesse Glick added a comment - 2017-09-18 20:33

piratejohnny all your examples involve Grape, which is a feature of workflow-cps-global-lib-plugin so should perhaps be moved there. If you have steps to reproduce from scratch, please specify them.

Jesse Glick added a comment - 2017-09-18 20:33 piratejohnny all your examples involve Grape, which is a feature of workflow-cps-global-lib-plugin so should perhaps be moved there. If you have steps to reproduce from scratch, please specify them.

Ankit Gupta added a comment - 2017-09-21 11:21

jglick - I am running into same issues. Open file handlers are mainly log files from jobs which are no longer running. To note, I have not killed any job which could lead to this behavior as mentioned. I believe it is a bug with the way pipeline jobs are managing logs. I looked at https://issues.jenkins-ci.org/browse/JENKINS-38381 but doesn't really understand what change needs to be made to fix this issue.

Ankit Gupta added a comment - 2017-09-21 11:21 jglick - I am running into same issues. Open file handlers are mainly log files from jobs which are no longer running. To note, I have not killed any job which could lead to this behavior as mentioned. I believe it is a bug with the way pipeline jobs are managing logs. I looked at https://issues.jenkins-ci.org/browse/JENKINS-38381 but doesn't really understand what change needs to be made to fix this issue.

Jesse Glick added a comment - 2017-10-06 19:42

If you figure out how to reproduce from scratch, that would be useful information.

Jesse Glick added a comment - 2017-10-06 19:42 If you figure out how to reproduce from scratch, that would be useful information.

Ankit Gupta added a comment - 2017-10-10 14:36 - edited

Unfortunately, I don't have exact steps to reproduce. If you have any steps which can help us identify the steps to reproduce this, that would help.

Also, is there a workaround to this issue?

Ankit Gupta added a comment - 2017-10-10 14:36 - edited Unfortunately, I don't have exact steps to reproduce. If you have any steps which can help us identify the steps to reproduce this, that would help. Also, is there a workaround to this issue?

Jon B added a comment - 2017-10-11 06:38 - edited

I just use Jenkins less and I also relaunched my master on a bigger AWS instance and I haven't seen it since then.

I'm 99% sure that the biggest change that made it go away was launching the master on a bigger instance.

Jon B added a comment - 2017-10-11 06:38 - edited I just use Jenkins less and I also relaunched my master on a bigger AWS instance and I haven't seen it since then. I'm 99% sure that the biggest change that made it go away was launching the master on a bigger instance.

Tiejun Sha added a comment - 2018-06-28 16:52

This happens not only for pipeline but also for job

Tiejun Sha added a comment - 2018-06-28 16:52 This happens not only for pipeline but also for job

Tiejun Sha added a comment - 2018-06-28 16:54

looks like when builds reach the limitation, jenkins delete expired builds but JVM still keep the link to those logs, that is the issue

ps -ef | grep java

ls -tl /proc/*****/fd | grep deleted

lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 912 -> /var/lib/jenkins/jobs/metrics-report/builds/.66892/log (deleted)

lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 913 -> /var/lib/jenkins/jobs/metrics-report/builds/.66889/log (deleted)

lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 914 -> /var/lib/jenkins/jobs/metrics-report/builds/.66893/log (deleted)

lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 916 -> /var/lib/jenkins/jobs/metrics-report/builds/.66997/log (deleted)

lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 917 -> /var/lib/jenkins/jobs/performance-ios-build/builds/.4574/log (deleted)

lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 918 -> /var/lib/jenkins/jobs/performance-android-build/builds/.14546/log (deleted)

lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 920 -> /var/lib/jenkins/jobs/metrics-report/builds/.66933/log (deleted)

lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 921 -> /var/lib/jenkins/jobs/metrics-report/builds/.66888/log (deleted)

lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 922 -> /var/lib/jenkins/jobs/metrics-report/builds/.66891/log (deleted)

lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 924 -> /var/lib/jenkins/jobs/performance-android-build/builds/.14654/log (deleted)

lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 927 -> /var/lib/jenkins/jobs/metrics-report/builds/.66894/log (deleted)

lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 928 -> /var/lib/jenkins/jobs/performance-android-build/builds/.14568/log (deleted)

lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 929 -> /var/lib/jenkins/jobs/metrics-report/builds/.66896/log (deleted)

lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 931 -> /var/lib/jenkins/jobs/metrics-report/builds/.66895/log (deleted)

lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 932 -> /var/lib/jenkins/jobs/metrics-report/builds/.66939/log (deleted)

Tiejun Sha added a comment - 2018-06-28 16:54 looks like when builds reach the limitation, jenkins delete expired builds but JVM still keep the link to those logs, that is the issue ps -ef | grep java ls -tl /proc/*****/fd | grep deleted lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 912 -> /var/lib/jenkins/jobs/metrics-report/builds/.66892/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 913 -> /var/lib/jenkins/jobs/metrics-report/builds/.66889/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 914 -> /var/lib/jenkins/jobs/metrics-report/builds/.66893/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 916 -> /var/lib/jenkins/jobs/metrics-report/builds/.66997/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 917 -> /var/lib/jenkins/jobs/performance-ios-build/builds/.4574/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 918 -> /var/lib/jenkins/jobs/performance-android-build/builds/.14546/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 920 -> /var/lib/jenkins/jobs/metrics-report/builds/.66933/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 921 -> /var/lib/jenkins/jobs/metrics-report/builds/.66888/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 922 -> /var/lib/jenkins/jobs/metrics-report/builds/.66891/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 924 -> /var/lib/jenkins/jobs/performance-android-build/builds/.14654/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 927 -> /var/lib/jenkins/jobs/metrics-report/builds/.66894/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 928 -> /var/lib/jenkins/jobs/performance-android-build/builds/.14568/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 929 -> /var/lib/jenkins/jobs/metrics-report/builds/.66896/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 931 -> /var/lib/jenkins/jobs/metrics-report/builds/.66895/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 932 -> /var/lib/jenkins/jobs/metrics-report/builds/.66939/log ( deleted )

Sam Van Oort added a comment - 2018-07-02 12:42

freemail165 Are you running the latest core and plugin versions? If so, can you provide a simple case that will reproduce the issue from scratch?

Sam Van Oort added a comment - 2018-07-02 12:42 freemail165 Are you running the latest core and plugin versions? If so, can you provide a simple case that will reproduce the issue from scratch?

Tiejun Sha added a comment - 2018-07-06 03:06

core version means jenkins version? for plugin version, which plugin do u care about?

Tiejun Sha added a comment - 2018-07-06 03:06 core version means jenkins version? for plugin version, which plugin do u care about?

Sam Van Oort added a comment - 2018-07-06 13:50

freemail165 Yes, "core" means Jenkins version. For plugins, we want the workflow-job, workflow-cps, durable-task, workflow-support, workflow-api, and workflow-basic-steps.

Sam Van Oort added a comment - 2018-07-06 13:50 freemail165 Yes, "core" means Jenkins version. For plugins, we want the workflow-job, workflow-cps, durable-task, workflow-support, workflow-api, and workflow-basic-steps.

Tiejun Sha added a comment - 2018-07-06 15:45

Jenkins ver. 2.104

Durable-task 1.17

from installed tab, I didn't find any plugins start from workflow

Tiejun Sha added a comment - 2018-07-06 15:45 Jenkins ver. 2.104 Durable-task 1.17 from installed tab, I didn't find any plugins start from workflow

Jesse Glick added a comment - 2018-10-04 22:22

Should not be an issue after JEP-210.

Jesse Glick added a comment - 2018-10-04 22:22 Should not be an issue after JEP-210.

Rick Liu added a comment - 2018-10-05 00:47

What's JEP-210?
Is it this JEP-210: http://openjdk.java.net/jeps/210

Rick Liu added a comment - 2018-10-05 00:47 What's JEP-210? Is it this JEP-210: http://openjdk.java.net/jeps/210

Jesse Glick added a comment - 2018-10-05 14:25

No, https://jenkins.io/jep/210 (yes I complained when this system was introduced by rtyler that there was going to be confusion with Java Enhancement Proposals).

Jesse Glick added a comment - 2018-10-05 14:25 No, https://jenkins.io/jep/210 (yes I complained when this system was introduced by rtyler that there was going to be confusion with Java Enhancement Proposals).

Sam Van Oort added a comment - 2018-10-05 15:08

totoroliu Since Jesse forgot to communicate the key info: there's a huge change to logging that's just gone into beta release (available via the Experimental Update Center – see https://jenkins.io/doc/developer/publishing/releasing-experimental-updates/ ) that completely removes these extra log files.

I'd argue that qualifies as Resolution: Fixed rather than "Won't Fix" but ::shrug::

Sam Van Oort added a comment - 2018-10-05 15:08 totoroliu Since Jesse forgot to communicate the key info: there's a huge change to logging that's just gone into beta release (available via the Experimental Update Center – see https://jenkins.io/doc/developer/publishing/releasing-experimental-updates/ ) that completely removes these extra log files. I'd argue that qualifies as Resolution: Fixed rather than "Won't Fix" but ::shrug::

Jesse Glick added a comment - 2018-10-05 15:31

Well, will not fix in original code base. That code is now gone.

Jesse Glick added a comment - 2018-10-05 15:31 Well, will not fix in original code base. That code is now gone.

People

Assignee:: Unassigned

Reporter:: Jon B

Votes:: 3 Vote for this issue

Watchers:: 8 Start watching this issue

Dates

Created:: 2017-06-18 23:52

Updated:: 2018-10-05 15:31

Resolved:: 2018-10-04 22:22