Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56947

File handle leak in EnvActionImpl when accessing environment of completed builds

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      We recently upgraded workflow-job-plugin from 2.24 => 2.32 and started seeing file handle leaks in our builds.

      We installed the file handle leak plugin and saw a lot of file handles being locked by other builds. This only seemed to happen when we were accessing the previous build's environment information.

      We narrowed it down to it using the EnvActionImpl to access environment variables. To workaround the issue, we called getEnvironment(TaskListener.NULL) to access the environment before accessing the variables.

      I'm wondering if this change is intentional. I believe the cause of this issue is that the EnvActionImpl getListener() uses the job's FlowExecution getListener(). This listener is set to always initialize whenever it is null. Whenever, a WorkFlowRun finished (calls finish()), the listener is set to null, so whenever something else accesses the build environment, the listener is re-initialized.

       

      Function causing file leak:

      @NonCPS
      private static def getFailureCause(build) {
          return build.previousBuild.getAction(org.jenkinsci.plugins.workflow.cps.EnvActionImpl)?.failureCause
        }
      

      File handle detector output:

      #99 /var/lib/jenkins/jobs/MikeTest/builds/6/log by thread:Running CpsFlowExecution[Owner[MikeTest/7:MikeTest #7]] on Fri Apr 05 17:29:43 PDT 2019
      	at java.io.FileOutputStream.<init>(FileOutputStream.java:214)
      	at org.jenkinsci.plugins.workflow.log.FileLogStorage.open(FileLogStorage.java:87)
      	at org.jenkinsci.plugins.workflow.log.FileLogStorage.access$000(FileLogStorage.java:61)
      	at org.jenkinsci.plugins.workflow.log.FileLogStorage$IndexOutputStream.<init>(FileLogStorage.java:146)
      	at org.jenkinsci.plugins.workflow.log.FileLogStorage.overallListener(FileLogStorage.java:115)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getListener(WorkflowRun.java:219)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.getListener(WorkflowRun.java:954)
      	at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getListener(EnvActionImpl.java:75)
      	at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getProperty(EnvActionImpl.java:99)
      	at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getProperty(EnvActionImpl.java:53)
      	at org.codehaus.groovy.runtime.InvokerHelper.getProperty(InvokerHelper.java:174)
      	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.getProperty(ScriptBytecodeAdapter.java:456)
      	at org.kohsuke.groovy.sandbox.impl.Checker$6.call(Checker.java:290)
      	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onGetProperty(GroovyInterceptor.java:68)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetProperty(SandboxInterceptor.java:279)
      	at org.kohsuke.groovy.sandbox.impl.Checker$6.call(Checker.java:288)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:292)
      	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.getProperty(SandboxInvoker.java:29)
      	at com.cloudbees.groovy.cps.impl.PropertyAccessBlock.rawGet(PropertyAccessBlock.java:20)
      	at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.get(PropertyishBlock.java:74)
      	at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
      	at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.fixName(PropertyishBlock.java:66)
      	at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
      	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
      	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:136)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:182)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

        Attachments

          Issue Links

            Activity

            michaelw Michael Wong created issue -
            Hide
            dnusbaum Devin Nusbaum added a comment - - edited

            I think this was fixed in workflow-cps 2.74 as part of JENKINS-59083, see the second changelog entry and PR 317 for details.

            Show
            dnusbaum Devin Nusbaum added a comment - - edited I think this was fixed in workflow-cps 2.74 as part of JENKINS-59083 , see the second changelog entry and PR 317 for details.
            dnusbaum Devin Nusbaum made changes -
            Field Original Value New Value
            Link This issue duplicates JENKINS-59083 [ JENKINS-59083 ]
            dnusbaum Devin Nusbaum made changes -
            Resolution Duplicate [ 3 ]
            Status Open [ 1 ] Closed [ 6 ]

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              michaelw Michael Wong
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: