Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45057

"too many files open": file handles leak, job output file not closed

      Jenkins seems to keep a open file handle to the log file (job output) for every single build, even those who have been discarded by the "Discard old build policy".

       

      This is a sample of the lsof output (whole file attached)

      java 8870 jenkins 941w REG 252,0 1840 1332171 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/.50063/log (deleted)
      java 8870 jenkins 942w REG 252,0 2023 402006 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/.50044/log (deleted)
      java 8870 jenkins 943w REG 252,0 2193 1332217 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/50101/log
      java 8870 jenkins 944w REG 252,0 2512 1332247 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/50106/log
      java 8870 jenkins 945w REG 252,0 1840 1703994 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/.50067/log (deleted)
      java 8870 jenkins 946w REG 252,0 2350 1332230 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/.50092/log (deleted)
      java 8870 jenkins 947w REG 252,0 1840 402034 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/.50049/log (deleted)
      java 8870 jenkins 948w REG 252,0 1840 927855 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/.50080/log (deleted)
      java 8870 jenkins 949w REG 252,0 2195 1332245 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/.50095/log (deleted)
      java 8870 jenkins 950w REG 252,0 2326 1332249 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/50107/log
      java 8870 jenkins 952w REG 252,0 2195 1332227 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/50102/log
      java 8870 jenkins 953w REG 252,0 2154 1332254 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/50109/log
      java 8870 jenkins 954w REG 252,0 2356 1332282 /data/jenkins/jobs/automation/jobs/emr-termination-policy/builds/50105/log
      

       

          [JENKINS-45057] "too many files open": file handles leak, job output file not closed

          Bruno Bonacci created issue -

          Oleg Nenashev added a comment -

          Which Build Discarder do you use in your job?

          Oleg Nenashev added a comment - Which Build Discarder do you use in your job?

          Bruno Bonacci added a comment -

          I'm using the default build discarder.

          Bruno Bonacci added a comment - I'm using the default build discarder.

          Daniel Beck added a comment -

          Please provide a list of installed plugins, and a sample configuration file of an affected job. Does this happen with all jobs?

          Daniel Beck added a comment - Please provide a list of installed plugins, and a sample configuration file of an affected job. Does this happen with all jobs?

          Jonas Jonsson added a comment - - edited

          Here's a very simple way of getting this:

          Create a simple FreeStyle job (NOTHING else but default settings) in Jenkins that only contains the following System Groovy Script:
          /*
           * See if Jenkins/Groovy leaves files open.
           */
          import hudson.model.*

          def thr = Thread.currentThread()
          def build = thr?.executable
          def jobName = build.parent.builds[0].properties.get("envVars").get("JOB_NAME")
          def jobNr = build.parent.builds[0].properties.get("envVars").get("BUILD_NUMBER")
          println "This is " + jobName + " running for the $jobNr:th time"
           

          That's it.  For every time I run this job, I get three (3!!) new open files in /proc/$PID_OF_JENKINS that points to the "log" file of the job.

          Linux (Ubuntu-14.04.5 LTS) 4.4.0-79 kernel
          Java version: 1.8.0_131-b11
          Jenkins-version: 2.66
          Groovy-plugin: 2.0
          System groovy version: 1.8.6

          Jonas Jonsson added a comment - - edited Here's a very simple way of getting this: Create a simple FreeStyle job (NOTHING else but default settings) in Jenkins that only contains the following System Groovy Script : /*  * See if Jenkins/Groovy leaves files open.  */ import hudson.model.* def thr = Thread.currentThread() def build = thr?.executable def jobName = build.parent.builds [0] .properties.get("envVars").get("JOB_NAME") def jobNr = build.parent.builds [0] .properties.get("envVars").get("BUILD_NUMBER") println "This is " + jobName + " running for the $jobNr:th time"   That's it.  For every time I run this job, I get three (3!!) new open files in /proc/$PID_OF_JENKINS that points to the "log" file of the job. Linux (Ubuntu-14.04.5 LTS) 4.4.0-79 kernel Java version: 1.8.0_131-b11 Jenkins-version: 2.66 Groovy-plugin: 2.0 System groovy version: 1.8.6

          Jonas Jonsson added a comment -

          The rationale for JENKINS-42934 was to avoid using close() on files, was this change taken a bit to far?

          Jonas Jonsson added a comment - The rationale for JENKINS-42934 was to avoid using close() on files, was this change taken a bit to far?
          Jonas Jonsson made changes -
          Link New: This issue relates to JENKINS-42934 [ JENKINS-42934 ]

          Jonas Jonsson added a comment -

          Our problems started when stepping the Jenkins-version from 2.51 to 2.58.

          Currently our production Jenkins must be restarted after about ten days.

          Jonas Jonsson added a comment - Our problems started when stepping the Jenkins-version from 2.51 to 2.58. Currently our production Jenkins must be restarted after about ten days.

          Daniel Beck added a comment -

          jonasatwork Would be helpful if you could narrow this down further.

          Daniel Beck added a comment - jonasatwork Would be helpful if you could narrow this down further.
          Jonas Jonsson made changes -
          Component/s New: groovy-plugin [ 15549 ]

            jglick Jesse Glick
            bbonacci Bruno Bonacci
            Votes:
            13 Vote for this issue
            Watchers:
            28 Start watching this issue

              Created:
              Updated:
              Resolved: