Details
-
Bug
-
Status: Resolved (View Workflow)
-
Minor
-
Resolution: Won't Fix
-
None
-
Jenkins ver. 2.46.3
Description
I use a Jenkins Pipeline to run my individual tests in parallel. I just noticed my test job that I run in parallel is failing with "Too many open files" errors.
Next, I ssh into the master and did an lsof -p using the pid of Jenkins... i see over 4k file descriptors. The output has thousands of lines that look like this:
java 7 root 4089w REG 202,96 28 2375888 /var/jenkins_home/jobs/test_pipeline/builds/15331/222.log java 7 root 4090w REG 202,96 328 787837 /var/jenkins_home/jobs/test_pipeline/builds/15343/166.log java 7 root 4091w REG 202,96 328 787851 /var/jenkins_home/jobs/test_pipeline/builds/15343/180.log java 7 root 4094w REG 202,96 671 787839 /var/jenkins_home/jobs/test_pipeline/builds/15343/167.log java 7 root 4095w REG 202,96 29 2361097 /var/jenkins_home/jobs/test_pipeline/builds/15331/157.log
Each file has content that roughly looks like this:
Starting building: ESC[8mha:////4CQAnoPVmHzYGq9xQUbp+LIf+xF38/zSfY1nZ0iZCPexAAAArB+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3Rdl
LBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAyZEgZF/az8JP2U/Pyi4pLUAt2ULKBh+fElqcUl+sYWliYGhvoAbOU0btYAAAA=ESC[0mmy_test #389401
Attachments
Issue Links
- relates to
-
JENKINS-38381 [JEP-210] Optimize log handling in Pipeline and Durable Task
-
- Resolved
-
Activity
I've noticed that as long as I preemptively restart Jenkins server at least daily, I won't get an unpredictable crash. The attached graph shows the number of file descriptors open throughout the day and you can see how it leaks up until the point where I manually restart Jenkins:
Is there a better option? I'm not a Jenkins expert by any means.. but perhaps you can advise something I can do within my Jenkins pipeline that will cause java to release the file descriptors after the pipeline has finished.
I did a little more investigation today: It appears this happens when I kill jobs and/or pipelines while they are running.
Jenkins API has an HTTP POST ability to kill in flight jobs with "/kill" at the end of the URL. I use the kill feature with some custom logic that says "Before I start a certain job with input param A and B, first kill any other Jenkins jobs or pipelines which are in flight that are already running with those same A and B inputs."
Therefore, throughout the day, we see a low percentage of kills. It must be for this reason why we are leaking file descriptors all day.
Do not use kill this way!!! This is a last resort when a build is completely unresponsive, and it will not perform proper cleanup.
Still a bug, but a much lower-priority one.
My Jenkins server got wedged at 100% CPU 3 times this week. Despite my not using "kill" anymore, I suspect the reason the master is getting hosed is because there are too many open file handles. I installed jglick's open file handles extension and looked at the output of it when there were no active jobs running starts with this first line:
"391 descriptors are open"
Hm.. that's kind of weird. No jobs running and 391 open file handles .. or descriptors... Let's examine one of them:
#8 /root/.groovy/grapes/org.apache.httpcomponents/httpclient/jars/httpclient-4.5.2.jar by thread:Executor #-1 for master : executing branch_out_of_date_check #67139 on Sat Sep 09 08:41:24 PDT 2017 at java.util.zip.ZipFile.<init>(ZipFile.java:150) at java.util.jar.JarFile.<init>(JarFile.java:166) at java.util.jar.JarFile.<init>(JarFile.java:103) at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:930) at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:791) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:876) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:869) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:868) at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:841) at sun.misc.URLClassPath$3.run(URLClassPath.java:565) at sun.misc.URLClassPath$3.run(URLClassPath.java:555) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath.getLoader(URLClassPath.java:554) at sun.misc.URLClassPath.getLoader(URLClassPath.java:519) at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484) at sun.misc.URLClassPath.access$100(URLClassPath.java:65) at sun.misc.URLClassPath$1.next(URLClassPath.java:266) at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:277) at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601) at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader$3.next(URLClassLoader.java:598) at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623) at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45) at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54) at org.codehaus.groovy.transform.ASTTransformationVisitor.doAddGlobalTransforms(ASTTransformationVisitor.java:201) at org.codehaus.groovy.transform.ASTTransformationVisitor.addGlobalTransformsAfterGrab(ASTTransformationVisitor.java:188) at groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:381) at org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:321) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254) at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:156) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545) at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185) at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170) at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126) at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676) at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313) at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1236) at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176) at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:516) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:479) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:419)
Hm... ok.. lets look at another one:
#9 /root/.groovy/grapes/org.apache.httpcomponents/httpcore/jars/httpcore-4.4.4.jar by thread:Executor #-1 for master : executing callbackgateway-nodeploy #195 on Sat Sep 09 09:24:43 PDT 2017 at java.util.zip.ZipFile.<init>(ZipFile.java:150) at java.util.jar.JarFile.<init>(JarFile.java:166) at java.util.jar.JarFile.<init>(JarFile.java:103) at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:930) at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:791) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:876) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:869) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:868) at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:841) at sun.misc.URLClassPath$3.run(URLClassPath.java:565) at sun.misc.URLClassPath$3.run(URLClassPath.java:555) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath.getLoader(URLClassPath.java:554) at sun.misc.URLClassPath.getLoader(URLClassPath.java:519) at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484) at sun.misc.URLClassPath.access$100(URLClassPath.java:65) at sun.misc.URLClassPath$1.next(URLClassPath.java:266) at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:277) at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601) at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader$3.next(URLClassLoader.java:598) at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623) at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45) at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54) at org.codehaus.groovy.transform.ASTTransformationVisitor.doAddGlobalTransforms(ASTTransformationVisitor.java:201) at org.codehaus.groovy.transform.ASTTransformationVisitor.addGlobalTransformsAfterGrab(ASTTransformationVisitor.java:188) at groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:381) at org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:321) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254) at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:156) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545) at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185) at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170) at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126) at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676) at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313) at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1236) at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176) at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:516) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:479) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:419)
Here's a 3rd example:
#10 /root/.groovy/grapes/commons-logging/commons-logging/jars/commons-logging-1.2.jar by thread:Executor #-1 for master : executing update_jenkins_slaves_list #12498 on Sat Sep 09 09:23:00 PDT 2017 at java.util.zip.ZipFile.<init>(ZipFile.java:150) at java.util.jar.JarFile.<init>(JarFile.java:166) at java.util.jar.JarFile.<init>(JarFile.java:103) at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:930) at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:791) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:876) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:869) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:868) at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:841) at sun.misc.URLClassPath$3.run(URLClassPath.java:565) at sun.misc.URLClassPath$3.run(URLClassPath.java:555) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath.getLoader(URLClassPath.java:554) at sun.misc.URLClassPath.getLoader(URLClassPath.java:519) at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484) at sun.misc.URLClassPath.access$100(URLClassPath.java:65) at sun.misc.URLClassPath$1.next(URLClassPath.java:266) at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:277) at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601) at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader$3.next(URLClassLoader.java:598) at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623) at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45) at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54) at org.codehaus.groovy.transform.ASTTransformationVisitor.doAddGlobalTransforms(ASTTransformationVisitor.java:201) at org.codehaus.groovy.transform.ASTTransformationVisitor.addGlobalTransformsAfterGrab(ASTTransformationVisitor.java:188) at groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:381) at org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:321) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254) at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:156) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545) at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185) at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170) at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126) at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676) at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313) at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1236) at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176) at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:516) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:479) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:419)
Here's one last example.. and this one is the most common one I see in my logs:
#4 /var/jenkins_home/jobs/django-nodeploy/builds/2650/586.log by thread:Running CpsFlowExecution[Owner[django-nodeploy/2650:doorstep-django-nodeploy #2650]] on Fri Sep 08 21:03:54 PDT 2017 at java.io.FileOutputStream.<init>(FileOutputStream.java:214) at org.jenkinsci.plugins.workflow.support.actions.LogActionImpl.stream(LogActionImpl.java:88) at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:73) at org.jenkinsci.plugins.workflow.steps.StepDescriptor.checkContextAvailability(StepDescriptor.java:258) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:203) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:150) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108) at sun.reflect.GeneratedMethodAccessor418.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155) at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:123) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:153) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:157) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127) at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82) at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46) at com.cloudbees.groovy.cps.Next.step(Next.java:83) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:173) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:162) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:162) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:330) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
It is worthwhile noting that my Jenkins installation gets lots of CI traffic.. many developers submit jobs and the master/slave pool is busy during the middle of the afternoon.
I don't have any hard evidence to support my theory, but I believe that when I reach around 3k open file descriptors (as reported by the operating system), the Jenkins master goes crazy and pegs to 100% CPU. The above graphs are from my Jenkins master yesterday. You see at 2PM, it went insane and Jenkins basically become almost totally unresponsive. By around 4:20PM, I decided to go to the Jenkins website "/restart" page and issued a restart. Once I did that, everything resumed normally.
Please advise.
piratejohnny all your examples involve Grape, which is a feature of workflow-cps-global-lib-plugin so should perhaps be moved there. If you have steps to reproduce from scratch, please specify them.
jglick - I am running into same issues. Open file handlers are mainly log files from jobs which are no longer running. To note, I have not killed any job which could lead to this behavior as mentioned. I believe it is a bug with the way pipeline jobs are managing logs. I looked at https://issues.jenkins-ci.org/browse/JENKINS-38381 but doesn't really understand what change needs to be made to fix this issue.
If you figure out how to reproduce from scratch, that would be useful information.
Unfortunately, I don't have exact steps to reproduce. If you have any steps which can help us identify the steps to reproduce this, that would help.
Also, is there a workaround to this issue?
I just use Jenkins less and I also relaunched my master on a bigger AWS instance and I haven't seen it since then.
I'm 99% sure that the biggest change that made it go away was launching the master on a bigger instance.
looks like when builds reach the limitation, jenkins delete expired builds but JVM still keep the link to those logs, that is the issue
ps -ef | grep java
ls -tl /proc/*****/fd | grep deleted
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 912 -> /var/lib/jenkins/jobs/metrics-report/builds/.66892/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 913 -> /var/lib/jenkins/jobs/metrics-report/builds/.66889/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 914 -> /var/lib/jenkins/jobs/metrics-report/builds/.66893/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 916 -> /var/lib/jenkins/jobs/metrics-report/builds/.66997/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 917 -> /var/lib/jenkins/jobs/performance-ios-build/builds/.4574/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 918 -> /var/lib/jenkins/jobs/performance-android-build/builds/.14546/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 920 -> /var/lib/jenkins/jobs/metrics-report/builds/.66933/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 921 -> /var/lib/jenkins/jobs/metrics-report/builds/.66888/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 922 -> /var/lib/jenkins/jobs/metrics-report/builds/.66891/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 924 -> /var/lib/jenkins/jobs/performance-android-build/builds/.14654/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 927 -> /var/lib/jenkins/jobs/metrics-report/builds/.66894/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 928 -> /var/lib/jenkins/jobs/performance-android-build/builds/.14568/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 929 -> /var/lib/jenkins/jobs/metrics-report/builds/.66896/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 931 -> /var/lib/jenkins/jobs/metrics-report/builds/.66895/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 932 -> /var/lib/jenkins/jobs/metrics-report/builds/.66939/log (deleted)
freemail165 Are you running the latest core and plugin versions? If so, can you provide a simple case that will reproduce the issue from scratch?
core version means jenkins version? for plugin version, which plugin do u care about?
freemail165 Yes, "core" means Jenkins version. For plugins, we want the workflow-job, workflow-cps, durable-task, workflow-support, workflow-api, and workflow-basic-steps.
Durable-task 1.17
from installed tab, I didn't find any plugins start from workflow
No, https://jenkins.io/jep/210 (yes I complained when this system was introduced by rtyler that there was going to be confusion with Java Enhancement Proposals).
totoroliu Since Jesse forgot to communicate the key info: there's a huge change to logging that's just gone into beta release (available via the Experimental Update Center – see https://jenkins.io/doc/developer/publishing/releasing-experimental-updates/ ) that completely removes these extra log files.
I'd argue that qualifies as Resolution: Fixed rather than "Won't Fix" but ::shrug::
Not sure how to reproduce this particular problem (first I have heard of it), but
JENKINS-38381would solve it by virtue of never creating these files to begin with.