Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44958

File handle leaks in *.log after hard kill

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Minor
    • Resolution: Won't Fix
    • Component/s: workflow-cps-plugin
    • Labels:
      None
    • Environment:
      Jenkins ver. 2.46.3
    • Similar Issues:

      Description

      I use a Jenkins Pipeline to run my individual tests in parallel. I just noticed my test job that I run in parallel is failing with "Too many open files" errors.

      Next, I ssh into the master and did an lsof -p using the pid of Jenkins... i see over 4k file descriptors. The output has thousands of lines that look like this:

      java 7 root 4089w REG 202,96 28 2375888 /var/jenkins_home/jobs/test_pipeline/builds/15331/222.log
java 7 root 4090w REG 202,96 328 787837 /var/jenkins_home/jobs/test_pipeline/builds/15343/166.log
java 7 root 4091w REG 202,96 328 787851 /var/jenkins_home/jobs/test_pipeline/builds/15343/180.log
java 7 root 4094w REG 202,96 671 787839 /var/jenkins_home/jobs/test_pipeline/builds/15343/167.log
java 7 root 4095w REG 202,96 29 2361097 /var/jenkins_home/jobs/test_pipeline/builds/15331/157.log

      Each file has content that roughly looks like this:

      Starting building: ESC[8mha:////4CQAnoPVmHzYGq9xQUbp+LIf+xF38/zSfY1nZ0iZCPexAAAArB+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3Rdl
LBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAyZEgZF/az8JP2U/Pyi4pLUAt2ULKBh+fElqcUl+sYWliYGhvoAbOU0btYAAAA=ESC[0mmy_test #389401

       

        Attachments

          Issue Links

          relates to

          Task - A task that needs to be done. JENKINS-38381 [JEP-210] Optimize log handling in Pipeline and Durable Task

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            jglick Jesse Glick added a comment -

            Not sure how to reproduce this particular problem (first I have heard of it), but JENKINS-38381 would solve it by virtue of never creating these files to begin with.

            Show
            jglick Jesse Glick added a comment - Not sure how to reproduce this particular problem (first I have heard of it), but JENKINS-38381 would solve it by virtue of never creating these files to begin with.
            Hide
            Permalink
            piratejohnny Jon B added a comment -

            I've noticed that as long as I preemptively restart Jenkins server at least daily, I won't get an unpredictable crash. The attached graph shows the number of file descriptors open throughout the day and you can see how it leaks up until the point where I manually restart Jenkins:

            Is there a better option? I'm not a Jenkins expert by any means.. but perhaps you can advise something I can do within my Jenkins pipeline that will cause java to release the file descriptors after the pipeline has finished.

            Show
            piratejohnny Jon B added a comment - I've noticed that as long as I preemptively restart Jenkins server at least daily, I won't get an unpredictable crash. The attached graph shows the number of file descriptors open throughout the day and you can see how it leaks up until the point where I manually restart Jenkins: Is there a better option? I'm not a Jenkins expert by any means.. but perhaps you can advise something I can do within my Jenkins pipeline that will cause java to release the file descriptors after the pipeline has finished.
            Hide
            Permalink
            piratejohnny Jon B added a comment - - edited

            I did a little more investigation today: It appears this happens when I kill jobs and/or pipelines while they are running.

            Jenkins API has an HTTP POST ability to kill in flight jobs with "/kill" at the end of the URL. I use the kill feature with some custom logic that says "Before I start a certain job with input param A and B, first kill any other Jenkins jobs or pipelines which are in flight that are already running with those same A and B inputs."

            Therefore, throughout the day, we see a low percentage of kills. It must be for this reason why we are leaking file descriptors all day.

            Show
            piratejohnny Jon B added a comment - - edited I did a little more investigation today: It appears this happens when I kill jobs and/or pipelines while they are running. Jenkins API has an HTTP POST ability to kill in flight jobs with "/kill" at the end of the URL. I use the kill feature with some custom logic that says "Before I start a certain job with input param A and B, first kill any other Jenkins jobs or pipelines which are in flight that are already running with those same A and B inputs." Therefore, throughout the day, we see a low percentage of kills. It must be for this reason why we are leaking file descriptors all day.
            Hide
            Permalink
            jglick Jesse Glick added a comment -

            Do not use kill this way!!! This is a last resort when a build is completely unresponsive, and it will not perform proper cleanup.

            Still a bug, but a much lower-priority one.

            Show
            jglick Jesse Glick added a comment - Do not use kill this way!!! This is a last resort when a build is completely unresponsive, and it will not perform proper cleanup. Still a bug, but a much lower-priority one.
            Hide
            Permalink
            piratejohnny Jon B added a comment - - edited

            My Jenkins server got wedged at 100% CPU 3 times this week. Despite my not using "kill" anymore, I suspect the reason the master is getting hosed is because there are too many open file handles. I installed Jesse Glick's open file handles extension and looked at the output of it when there were no active jobs running starts with this first line:

            "391 descriptors are open"

            Hm.. that's kind of weird. No jobs running and 391 open file handles .. or descriptors... Let's examine one of them:

            #8 /root/.groovy/grapes/org.apache.httpcomponents/httpclient/jars/httpclient-4.5.2.jar by thread:Executor #-1 for master : executing branch_out_of_date_check #67139 on Sat Sep 09 08:41:24 PDT 2017
	at java.util.zip.ZipFile.<init>(ZipFile.java:150)
	at java.util.jar.JarFile.<init>(JarFile.java:166)
	at java.util.jar.JarFile.<init>(JarFile.java:103)
	at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:930)
	at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:791)
	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:876)
	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:869)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:868)
	at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:841)
	at sun.misc.URLClassPath$3.run(URLClassPath.java:565)
	at sun.misc.URLClassPath$3.run(URLClassPath.java:555)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.misc.URLClassPath.getLoader(URLClassPath.java:554)
	at sun.misc.URLClassPath.getLoader(URLClassPath.java:519)
	at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484)
	at sun.misc.URLClassPath.access$100(URLClassPath.java:65)
	at sun.misc.URLClassPath$1.next(URLClassPath.java:266)
	at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:277)
	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)
	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader$3.next(URLClassLoader.java:598)
	at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)
	at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
	at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
	at org.codehaus.groovy.transform.ASTTransformationVisitor.doAddGlobalTransforms(ASTTransformationVisitor.java:201)
	at org.codehaus.groovy.transform.ASTTransformationVisitor.addGlobalTransformsAfterGrab(ASTTransformationVisitor.java:188)
	at groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:381)
	at org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:321)
	at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
	at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:156)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545)
	at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185)
	at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170)
	at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126)
	at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676)
	at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313)
	at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1236)
	at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176)
	at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663)
	at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
	at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:516)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:479)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:419)

            Hm... ok.. lets look at another one:

            #9 /root/.groovy/grapes/org.apache.httpcomponents/httpcore/jars/httpcore-4.4.4.jar by thread:Executor #-1 for master : executing callbackgateway-nodeploy #195 on Sat Sep 09 09:24:43 PDT 2017
	at java.util.zip.ZipFile.<init>(ZipFile.java:150)
	at java.util.jar.JarFile.<init>(JarFile.java:166)
	at java.util.jar.JarFile.<init>(JarFile.java:103)
	at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:930)
	at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:791)
	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:876)
	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:869)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:868)
	at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:841)
	at sun.misc.URLClassPath$3.run(URLClassPath.java:565)
	at sun.misc.URLClassPath$3.run(URLClassPath.java:555)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.misc.URLClassPath.getLoader(URLClassPath.java:554)
	at sun.misc.URLClassPath.getLoader(URLClassPath.java:519)
	at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484)
	at sun.misc.URLClassPath.access$100(URLClassPath.java:65)
	at sun.misc.URLClassPath$1.next(URLClassPath.java:266)
	at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:277)
	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)
	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader$3.next(URLClassLoader.java:598)
	at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)
	at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
	at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
	at org.codehaus.groovy.transform.ASTTransformationVisitor.doAddGlobalTransforms(ASTTransformationVisitor.java:201)
	at org.codehaus.groovy.transform.ASTTransformationVisitor.addGlobalTransformsAfterGrab(ASTTransformationVisitor.java:188)
	at groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:381)
	at org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:321)
	at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
	at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:156)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545)
	at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185)
	at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170)
	at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126)
	at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676)
	at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313)
	at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1236)
	at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176)
	at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663)
	at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
	at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:516)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:479)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:419)

            Here's a 3rd example:

            #10 /root/.groovy/grapes/commons-logging/commons-logging/jars/commons-logging-1.2.jar by thread:Executor #-1 for master : executing update_jenkins_slaves_list #12498 on Sat Sep 09 09:23:00 PDT 2017
	at java.util.zip.ZipFile.<init>(ZipFile.java:150)
	at java.util.jar.JarFile.<init>(JarFile.java:166)
	at java.util.jar.JarFile.<init>(JarFile.java:103)
	at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:930)
	at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:791)
	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:876)
	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:869)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:868)
	at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:841)
	at sun.misc.URLClassPath$3.run(URLClassPath.java:565)
	at sun.misc.URLClassPath$3.run(URLClassPath.java:555)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.misc.URLClassPath.getLoader(URLClassPath.java:554)
	at sun.misc.URLClassPath.getLoader(URLClassPath.java:519)
	at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484)
	at sun.misc.URLClassPath.access$100(URLClassPath.java:65)
	at sun.misc.URLClassPath$1.next(URLClassPath.java:266)
	at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:277)
	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)
	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader$3.next(URLClassLoader.java:598)
	at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)
	at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
	at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
	at org.codehaus.groovy.transform.ASTTransformationVisitor.doAddGlobalTransforms(ASTTransformationVisitor.java:201)
	at org.codehaus.groovy.transform.ASTTransformationVisitor.addGlobalTransformsAfterGrab(ASTTransformationVisitor.java:188)
	at groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:381)
	at org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:321)
	at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
	at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:156)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545)
	at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185)
	at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170)
	at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126)
	at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676)
	at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313)
	at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1236)
	at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176)
	at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663)
	at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
	at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:516)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:479)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:419)

             Here's one last example.. and this one is the most common one I see in my logs:

            #4 /var/jenkins_home/jobs/django-nodeploy/builds/2650/586.log by thread:Running CpsFlowExecution[Owner[django-nodeploy/2650:doorstep-django-nodeploy #2650]] on Fri Sep 08 21:03:54 PDT 2017
	at java.io.FileOutputStream.<init>(FileOutputStream.java:214)
	at org.jenkinsci.plugins.workflow.support.actions.LogActionImpl.stream(LogActionImpl.java:88)
	at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:73)
	at org.jenkinsci.plugins.workflow.steps.StepDescriptor.checkContextAvailability(StepDescriptor.java:258)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:203)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:150)
	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
	at sun.reflect.GeneratedMethodAccessor418.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:123)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:153)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:157)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127)
	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:173)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:162)
	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:162)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:330)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

             

            It is worthwhile noting that my Jenkins installation gets lots of CI traffic.. many developers submit jobs and the master/slave pool is busy during the middle of the afternoon.

            I don't have any hard evidence to support my theory, but I believe that when I reach around 3k open file descriptors (as reported by the operating system), the Jenkins master goes crazy and pegs to 100% CPU. The above graphs are from my Jenkins master yesterday. You see at 2PM, it went insane and Jenkins basically become almost totally unresponsive. By around 4:20PM, I decided to go to the Jenkins website "/restart" page and issued a restart. Once I did that, everything resumed normally.

            Please advise.

            Show
            piratejohnny Jon B added a comment - - edited My Jenkins server got wedged at 100% CPU 3 times this week. Despite my not using "kill" anymore, I suspect the reason the master is getting hosed is because there are too many open file handles. I installed Jesse Glick 's open file handles extension and looked at the output of it when there were no active jobs running starts with this first line: "391 descriptors are open" Hm.. that's kind of weird. No jobs running and 391 open file handles .. or descriptors... Let's examine one of them: #8 /root/.groovy/grapes/org.apache.httpcomponents/httpclient/jars/httpclient-4.5.2.jar by thread:Executor #-1 for master : executing branch_out_of_date_check #67139 on Sat Sep 09 08:41:24 PDT 2017 at java.util.zip.ZipFile.<init>(ZipFile.java:150) at java.util.jar.JarFile.<init>(JarFile.java:166) at java.util.jar.JarFile.<init>(JarFile.java:103) at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:930) at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:791) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:876) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:869) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:868) at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:841) at sun.misc.URLClassPath$3.run(URLClassPath.java:565) at sun.misc.URLClassPath$3.run(URLClassPath.java:555) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath.getLoader(URLClassPath.java:554) at sun.misc.URLClassPath.getLoader(URLClassPath.java:519) at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484) at sun.misc.URLClassPath.access$100(URLClassPath.java:65) at sun.misc.URLClassPath$1.next(URLClassPath.java:266) at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:277) at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601) at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader$3.next(URLClassLoader.java:598) at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623) at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45) at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54) at org.codehaus.groovy.transform.ASTTransformationVisitor.doAddGlobalTransforms(ASTTransformationVisitor.java:201) at org.codehaus.groovy.transform.ASTTransformationVisitor.addGlobalTransformsAfterGrab(ASTTransformationVisitor.java:188) at groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:381) at org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:321) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254) at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787) at java.lang. ClassLoader .loadClass( ClassLoader .java:411) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:156) at java.lang. ClassLoader .loadClass( ClassLoader .java:411) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545) at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185) at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170) at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126) at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676) at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313) at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1236) at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176) at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:516) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:479) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:419) Hm... ok.. lets look at another one: #9 /root/.groovy/grapes/org.apache.httpcomponents/httpcore/jars/httpcore-4.4.4.jar by thread:Executor #-1 for master : executing callbackgateway-nodeploy #195 on Sat Sep 09 09:24:43 PDT 2017 at java.util.zip.ZipFile.<init>(ZipFile.java:150) at java.util.jar.JarFile.<init>(JarFile.java:166) at java.util.jar.JarFile.<init>(JarFile.java:103) at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:930) at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:791) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:876) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:869) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:868) at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:841) at sun.misc.URLClassPath$3.run(URLClassPath.java:565) at sun.misc.URLClassPath$3.run(URLClassPath.java:555) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath.getLoader(URLClassPath.java:554) at sun.misc.URLClassPath.getLoader(URLClassPath.java:519) at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484) at sun.misc.URLClassPath.access$100(URLClassPath.java:65) at sun.misc.URLClassPath$1.next(URLClassPath.java:266) at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:277) at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601) at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader$3.next(URLClassLoader.java:598) at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623) at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45) at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54) at org.codehaus.groovy.transform.ASTTransformationVisitor.doAddGlobalTransforms(ASTTransformationVisitor.java:201) at org.codehaus.groovy.transform.ASTTransformationVisitor.addGlobalTransformsAfterGrab(ASTTransformationVisitor.java:188) at groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:381) at org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:321) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254) at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787) at java.lang. ClassLoader .loadClass( ClassLoader .java:411) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:156) at java.lang. ClassLoader .loadClass( ClassLoader .java:411) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545) at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185) at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170) at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126) at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676) at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313) at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1236) at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176) at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:516) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:479) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:419) Here's a 3rd example: #10 /root/.groovy/grapes/commons-logging/commons-logging/jars/commons-logging-1.2.jar by thread:Executor #-1 for master : executing update_jenkins_slaves_list #12498 on Sat Sep 09 09:23:00 PDT 2017 at java.util.zip.ZipFile.<init>(ZipFile.java:150) at java.util.jar.JarFile.<init>(JarFile.java:166) at java.util.jar.JarFile.<init>(JarFile.java:103) at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:930) at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:791) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:876) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:869) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:868) at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:841) at sun.misc.URLClassPath$3.run(URLClassPath.java:565) at sun.misc.URLClassPath$3.run(URLClassPath.java:555) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath.getLoader(URLClassPath.java:554) at sun.misc.URLClassPath.getLoader(URLClassPath.java:519) at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484) at sun.misc.URLClassPath.access$100(URLClassPath.java:65) at sun.misc.URLClassPath$1.next(URLClassPath.java:266) at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:277) at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601) at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader$3.next(URLClassLoader.java:598) at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623) at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45) at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54) at org.codehaus.groovy.transform.ASTTransformationVisitor.doAddGlobalTransforms(ASTTransformationVisitor.java:201) at org.codehaus.groovy.transform.ASTTransformationVisitor.addGlobalTransformsAfterGrab(ASTTransformationVisitor.java:188) at groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:381) at org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:321) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254) at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787) at java.lang. ClassLoader .loadClass( ClassLoader .java:411) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:156) at java.lang. ClassLoader .loadClass( ClassLoader .java:411) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545) at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185) at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170) at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126) at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676) at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:313) at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1236) at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176) at org.codehaus.groovy.control.CompilationUnit$12.call(CompilationUnit.java:663) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:516) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:479) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:419)  Here's one last example.. and this one is the most common one I see in my logs: #4 / var /jenkins_home/jobs/django-nodeploy/builds/2650/586.log by thread:Running CpsFlowExecution[Owner[django-nodeploy/2650:doorstep-django-nodeploy #2650]] on Fri Sep 08 21:03:54 PDT 2017 at java.io.FileOutputStream.<init>(FileOutputStream.java:214) at org.jenkinsci.plugins.workflow.support.actions.LogActionImpl.stream(LogActionImpl.java:88) at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:73) at org.jenkinsci.plugins.workflow.steps.StepDescriptor.checkContextAvailability(StepDescriptor.java:258) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:203) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:150) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108) at sun.reflect.GeneratedMethodAccessor418.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155) at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:123) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:153) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:157) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127) at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82) at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46) at com.cloudbees.groovy.cps.Next.step(Next.java:83) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:173) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:162) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:162) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:330) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748)   It is worthwhile noting that my Jenkins installation gets lots of CI traffic.. many developers submit jobs and the master/slave pool is busy during the middle of the afternoon. I don't have any hard evidence to support my theory, but I believe that when I reach around 3k open file descriptors (as reported by the operating system), the Jenkins master goes crazy and pegs to 100% CPU. The above graphs are from my Jenkins master yesterday. You see at 2PM, it went insane and Jenkins basically become almost totally unresponsive. By around 4:20PM, I decided to go to the Jenkins website "/restart" page and issued a restart. Once I did that, everything resumed normally. Please advise.
            Hide
            Permalink
            jglick Jesse Glick added a comment -

            Jon B all your examples involve Grape, which is a feature of workflow-cps-global-lib-plugin so should perhaps be moved there. If you have steps to reproduce from scratch, please specify them.

            Show
            jglick Jesse Glick added a comment - Jon B all your examples involve Grape, which is a feature of workflow-cps-global-lib-plugin  so should perhaps be moved there. If you have steps to reproduce from scratch, please specify them.
            Hide
            Permalink
            agupta Ankit Gupta added a comment -

            Jesse Glick - I am running into same issues. Open file handlers are mainly log files from jobs which are no longer running. To note, I have not killed any job which could lead to this behavior as mentioned. I believe it is a bug with the way pipeline jobs are managing logs. I looked at https://issues.jenkins-ci.org/browse/JENKINS-38381 but doesn't really understand what change needs to be made to fix this issue.

            Show
            agupta Ankit Gupta added a comment - Jesse Glick - I am running into same issues. Open file handlers are mainly log files from jobs which are no longer running. To note, I have not killed any job which could lead to this behavior as mentioned. I believe it is a bug with the way pipeline jobs are managing logs. I looked at https://issues.jenkins-ci.org/browse/JENKINS-38381  but doesn't really understand what change needs to be made to fix this issue.
            Hide
            Permalink
            jglick Jesse Glick added a comment -

            If you figure out how to reproduce from scratch, that would be useful information.

            Show
            jglick Jesse Glick added a comment - If you figure out how to reproduce from scratch, that would be useful information.
            Hide
            Permalink
            agupta Ankit Gupta added a comment - - edited

            Unfortunately, I don't have exact steps to reproduce. If you have any steps which can help us identify the steps to reproduce this, that would help. 

            Also, is there a workaround to this issue?

            Show
            agupta Ankit Gupta added a comment - - edited Unfortunately, I don't have exact steps to reproduce. If you have any steps which can help us identify the steps to reproduce this, that would help.  Also, is there a workaround to this issue?
            Hide
            Permalink
            piratejohnny Jon B added a comment - - edited

            I just use Jenkins less and I also relaunched my master on a bigger AWS instance and I haven't seen it since then.

            I'm 99% sure that the biggest change that made it go away was launching the master on a bigger instance.

            Show
            piratejohnny Jon B added a comment - - edited I just use Jenkins less and I also relaunched my master on a bigger AWS instance and I haven't seen it since then. I'm 99% sure that the biggest change that made it go away was launching the master on a bigger instance.
            Hide
            Permalink
            freemail165 Tiejun Sha added a comment -

            This happens not only for pipeline but also for job

            Show
            freemail165 Tiejun Sha added a comment - This happens not only for pipeline but also for job
            Hide
            Permalink
            freemail165 Tiejun Sha added a comment -

            looks like when builds reach the limitation, jenkins delete expired builds but JVM still keep the link to those logs, that is the issue

            ps -ef | grep java 

            ls -tl /proc/*****/fd | grep deleted

             

            lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 912 -> /var/lib/jenkins/jobs/metrics-report/builds/.66892/log (deleted)

            lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 913 -> /var/lib/jenkins/jobs/metrics-report/builds/.66889/log (deleted)

            lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 914 -> /var/lib/jenkins/jobs/metrics-report/builds/.66893/log (deleted)

            lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 916 -> /var/lib/jenkins/jobs/metrics-report/builds/.66997/log (deleted)

            lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 917 -> /var/lib/jenkins/jobs/performance-ios-build/builds/.4574/log (deleted)

            lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 918 -> /var/lib/jenkins/jobs/performance-android-build/builds/.14546/log (deleted)

            lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 920 -> /var/lib/jenkins/jobs/metrics-report/builds/.66933/log (deleted)

            lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 921 -> /var/lib/jenkins/jobs/metrics-report/builds/.66888/log (deleted)

            lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 922 -> /var/lib/jenkins/jobs/metrics-report/builds/.66891/log (deleted)

            lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 924 -> /var/lib/jenkins/jobs/performance-android-build/builds/.14654/log (deleted)

            lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 927 -> /var/lib/jenkins/jobs/metrics-report/builds/.66894/log (deleted)

            lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 928 -> /var/lib/jenkins/jobs/performance-android-build/builds/.14568/log (deleted)

            lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 929 -> /var/lib/jenkins/jobs/metrics-report/builds/.66896/log (deleted)

            lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 931 -> /var/lib/jenkins/jobs/metrics-report/builds/.66895/log (deleted)

            lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 932 -> /var/lib/jenkins/jobs/metrics-report/builds/.66939/log (deleted)

            Show
            freemail165 Tiejun Sha added a comment - looks like when builds reach the limitation, jenkins delete expired builds but JVM still keep the link to those logs, that is the issue ps -ef | grep java  ls -tl /proc/*****/fd | grep deleted   lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 912 -> /var/lib/jenkins/jobs/metrics-report/builds/.66892/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 913 -> /var/lib/jenkins/jobs/metrics-report/builds/.66889/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 914 -> /var/lib/jenkins/jobs/metrics-report/builds/.66893/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 916 -> /var/lib/jenkins/jobs/metrics-report/builds/.66997/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 917 -> /var/lib/jenkins/jobs/performance-ios-build/builds/.4574/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 918 -> /var/lib/jenkins/jobs/performance-android-build/builds/.14546/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 920 -> /var/lib/jenkins/jobs/metrics-report/builds/.66933/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 921 -> /var/lib/jenkins/jobs/metrics-report/builds/.66888/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 922 -> /var/lib/jenkins/jobs/metrics-report/builds/.66891/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 924 -> /var/lib/jenkins/jobs/performance-android-build/builds/.14654/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 927 -> /var/lib/jenkins/jobs/metrics-report/builds/.66894/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 928 -> /var/lib/jenkins/jobs/performance-android-build/builds/.14568/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 929 -> /var/lib/jenkins/jobs/metrics-report/builds/.66896/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 931 -> /var/lib/jenkins/jobs/metrics-report/builds/.66895/log ( deleted ) lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 932 -> /var/lib/jenkins/jobs/metrics-report/builds/.66939/log ( deleted )
            Hide
            Permalink
            svanoort Sam Van Oort added a comment -

            Tiejun Sha Are you running the latest core and plugin versions?  If so, can you provide a simple case that will reproduce the issue from scratch?

            Show
            svanoort Sam Van Oort added a comment - Tiejun Sha Are you running the latest core and plugin versions?  If so, can you provide a simple case that will reproduce the issue from scratch?
            Hide
            Permalink
            freemail165 Tiejun Sha added a comment -

            core version means jenkins version? for plugin version, which plugin do u care about?

            Show
            freemail165 Tiejun Sha added a comment - core version means jenkins version? for plugin version, which plugin do u care about?
            Hide
            Permalink
            svanoort Sam Van Oort added a comment -

            Tiejun Sha Yes, "core" means Jenkins version.  For plugins, we want the workflow-job, workflow-cps, durable-task, workflow-support, workflow-api, and workflow-basic-steps.

            Show
            svanoort Sam Van Oort added a comment - Tiejun Sha Yes, "core" means Jenkins version.  For plugins, we want the workflow-job, workflow-cps, durable-task, workflow-support, workflow-api, and workflow-basic-steps.
            Hide
            Permalink
            freemail165 Tiejun Sha added a comment -

            Jenkins ver. 2.104

            Durable-task 1.17

            from installed tab, I didn't find any plugins start from workflow

            Show
            freemail165 Tiejun Sha added a comment - Jenkins ver. 2.104 Durable-task 1.17 from installed tab, I didn't find any plugins start from workflow
            Hide
            Permalink
            jglick Jesse Glick added a comment -

            Should not be an issue after JEP-210.

            Show
            jglick Jesse Glick added a comment - Should not be an issue after JEP-210.
            Hide
            Permalink
            totoroliu Rick Liu added a comment -

            What's JEP-210?
            Is it this JEP-210: http://openjdk.java.net/jeps/210

            Show
            totoroliu Rick Liu added a comment - What's JEP-210? Is it this JEP-210: http://openjdk.java.net/jeps/210
            Hide
            Permalink
            jglick Jesse Glick added a comment -

            No, https://jenkins.io/jep/210 (yes I complained when this system was introduced by R. Tyler Croy that there was going to be confusion with Java Enhancement Proposals).

            Show
            jglick Jesse Glick added a comment - No, https://jenkins.io/jep/210 (yes I complained when this system was introduced by R. Tyler Croy that there was going to be confusion with Java Enhancement Proposals).
            Hide
            Permalink
            svanoort Sam Van Oort added a comment -

            Rick Liu Since Jesse forgot to communicate the key info: there's a huge change to logging that's just gone into beta release (available via the Experimental Update Center – see https://jenkins.io/doc/developer/publishing/releasing-experimental-updates/ ) that completely removes these extra log files.

            I'd argue that qualifies as Resolution: Fixed rather than "Won't Fix" but ::shrug::

            Show
            svanoort Sam Van Oort added a comment - Rick Liu Since Jesse forgot to communicate the key info: there's a huge change to logging that's just gone into beta release (available via the Experimental Update Center – see https://jenkins.io/doc/developer/publishing/releasing-experimental-updates/ ) that completely removes these extra log files. I'd argue that qualifies as Resolution: Fixed rather than "Won't Fix" but ::shrug::
            Hide
            Permalink
            jglick Jesse Glick added a comment -

            Well, will not fix in original code base. That code is now gone.

            Show
            jglick Jesse Glick added a comment - Well, will not fix in original code base. That code is now gone.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              piratejohnny Jon B
              Votes:
              3 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: