Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56947

File handle leak in EnvActionImpl when accessing environment of completed builds

    XMLWordPrintable

Details

    Description

      We recently upgraded workflow-job-plugin from 2.24 => 2.32 and started seeing file handle leaks in our builds.

      We installed the file handle leak plugin and saw a lot of file handles being locked by other builds. This only seemed to happen when we were accessing the previous build's environment information.

      We narrowed it down to it using the EnvActionImpl to access environment variables. To workaround the issue, we called getEnvironment(TaskListener.NULL) to access the environment before accessing the variables.

      I'm wondering if this change is intentional. I believe the cause of this issue is that the EnvActionImpl getListener() uses the job's FlowExecution getListener(). This listener is set to always initialize whenever it is null. Whenever, a WorkFlowRun finished (calls finish()), the listener is set to null, so whenever something else accesses the build environment, the listener is re-initialized.

       

      Function causing file leak:

      @NonCPS
      private static def getFailureCause(build) {
          return build.previousBuild.getAction(org.jenkinsci.plugins.workflow.cps.EnvActionImpl)?.failureCause
        }
      

      File handle detector output:

      #99 /var/lib/jenkins/jobs/MikeTest/builds/6/log by thread:Running CpsFlowExecution[Owner[MikeTest/7:MikeTest #7]] on Fri Apr 05 17:29:43 PDT 2019
      	at java.io.FileOutputStream.<init>(FileOutputStream.java:214)
      	at org.jenkinsci.plugins.workflow.log.FileLogStorage.open(FileLogStorage.java:87)
      	at org.jenkinsci.plugins.workflow.log.FileLogStorage.access$000(FileLogStorage.java:61)
      	at org.jenkinsci.plugins.workflow.log.FileLogStorage$IndexOutputStream.<init>(FileLogStorage.java:146)
      	at org.jenkinsci.plugins.workflow.log.FileLogStorage.overallListener(FileLogStorage.java:115)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getListener(WorkflowRun.java:219)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.getListener(WorkflowRun.java:954)
      	at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getListener(EnvActionImpl.java:75)
      	at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getProperty(EnvActionImpl.java:99)
      	at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getProperty(EnvActionImpl.java:53)
      	at org.codehaus.groovy.runtime.InvokerHelper.getProperty(InvokerHelper.java:174)
      	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.getProperty(ScriptBytecodeAdapter.java:456)
      	at org.kohsuke.groovy.sandbox.impl.Checker$6.call(Checker.java:290)
      	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onGetProperty(GroovyInterceptor.java:68)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetProperty(SandboxInterceptor.java:279)
      	at org.kohsuke.groovy.sandbox.impl.Checker$6.call(Checker.java:288)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:292)
      	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.getProperty(SandboxInvoker.java:29)
      	at com.cloudbees.groovy.cps.impl.PropertyAccessBlock.rawGet(PropertyAccessBlock.java:20)
      	at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.get(PropertyishBlock.java:74)
      	at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
      	at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.fixName(PropertyishBlock.java:66)
      	at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
      	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
      	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:136)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:182)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

      Attachments

        Issue Links

          Activity

            michaelw Michael Wong created issue -
            dnusbaum Devin Nusbaum added a comment - - edited

            I think this was fixed in workflow-cps 2.74 as part of JENKINS-59083, see the second changelog entry and PR 317 for details.

            dnusbaum Devin Nusbaum added a comment - - edited I think this was fixed in workflow-cps 2.74 as part of JENKINS-59083 , see the second changelog entry and PR 317 for details.
            dnusbaum Devin Nusbaum made changes -
            Field Original Value New Value
            Link This issue duplicates JENKINS-59083 [ JENKINS-59083 ]
            dnusbaum Devin Nusbaum made changes -
            Resolution Duplicate [ 3 ]
            Status Open [ 1 ] Closed [ 6 ]

            People

              Unassigned Unassigned
              michaelw Michael Wong
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: