Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52658

google-storage-plugin cause file handle leak

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: google-storage-plugin
    • Labels:
      None
    • Environment:
      Jenkins ver. 2.104
      Google Cloud Storage Plugin 1.1
    • Similar Issues:

      Description

      We got the issue as "too many files open"

      After some searching/investigation, In the end , we realized the issue came from google-storage-plugin

      repro steps are as below:

      1) Create a freestyle project

      2) add log upload as post-build action

      3) set "Max # of builds to keep" to 3

      4) rebuild the project a few times

      5) get processId with ps -ef | grep java | grep -v daemon | grep -v grep | awk '

      { print $2}

      '

      6) lsof -p $processId | grep deleted

      I can get below

      java    6480 jenkins 780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.1/log (*deleted)

      java    6480 jenkins 780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.2/log (*deleted)

      java    6480 jenkins 780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.3/log (*deleted)

      java    6480 jenkins 780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.4/log (*deleted)

       

      BTW, for a finished job #5 but hasn't been deleted, we can get below result.

      java still hold the open file of the job log which seems to be a issue

       lsof -p 6480 | grep file-handle-leak

      java    6480 jenkins *809r   REG                8,1   466655   10234118 /var/lib/jenkins/jobs/file-handle-leak/builds/5/log

        Attachments

          Activity

          Hide
          freemail165 Tiejun Sha added a comment - - edited

          On the same Jenkins master, we have pipeline which is using below for log upload, issue not happen

          node('master') {
              dir ("../builds/${BUILD_ID}") {
                  sh(script: "cp log build-log.txt")
                  script {
                        googleStorageUpload bucket: "gs://*****/${JOB_NAME}/${BUILD_ID}", credentialsId: '*****', pattern: 'build-log.txt'
                  }
              }
          }

          Show
          freemail165 Tiejun Sha added a comment - - edited On the same Jenkins master, we have pipeline which is using below for log upload, issue not happen node('master') {     dir ("../builds/${BUILD_ID}") {         sh(script: "cp log build-log.txt")         script {               googleStorageUpload bucket: "gs://***** /${JOB_NAME}/${BUILD_ID}", credentialsId: ' *****', pattern: 'build-log.txt'         }     } }
          Hide
          a_goulti Alexandra Goultiaeva added a comment -

          Thanks for the bug report, and for detailed steps to reproduce.

          I can confirm that I can reproduce this, looking deeper into this.

          Show
          a_goulti Alexandra Goultiaeva added a comment - Thanks for the bug report, and for detailed steps to reproduce. I can confirm that I can reproduce this, looking deeper into this.
          Hide
          freemail165 Tiejun Sha added a comment - - edited

          BTW, we have another jenkins master which seems work fine with below version 
          ==> I didn't try above steps, but we never met "too many files open" issue there
           
          Jenkins ver. 2.34 
          Google Cloud Storage Plugin 1.1

          Show
          freemail165 Tiejun Sha added a comment - - edited BTW, we have another jenkins master which seems work fine with below version  ==> I didn't try above steps, but we never met "too many files open" issue there   Jenkins ver. 2.34  Google Cloud Storage Plugin 1.1
          Hide
          a_goulti Alexandra Goultiaeva added a comment -

          Yes, the problem was that the Stdout Upload step never closed the input stream it was reading from.

          The fix is out for review as a PR here: https://github.com/jenkinsci/google-storage-plugin/pull/46

           

          Thanks again for reporting with an easy repro!

          Show
          a_goulti Alexandra Goultiaeva added a comment - Yes, the problem was that the Stdout Upload step never closed the input stream it was reading from. The fix is out for review as a PR here: https://github.com/jenkinsci/google-storage-plugin/pull/46   Thanks again for reporting with an easy repro!
          Hide
          freemail165 Tiejun Sha added a comment -

          any idea for below:
          1) why pipeline works well on the same jenkins master
          2) why the same version plugin works on another jenkins master

          Show
          freemail165 Tiejun Sha added a comment - any idea for below: 1) why pipeline works well on the same jenkins master 2) why the same version plugin works on another jenkins master
          Hide
          freemail165 Tiejun Sha added a comment -

          BTW, what is the ETA for this change to be released?
          I assuming that it will be 1.2, is that correct?

          Show
          freemail165 Tiejun Sha added a comment - BTW, what is the ETA for this change to be released? I assuming that it will be 1.2, is that correct?
          Hide
          freemail165 Tiejun Sha added a comment -

          seems i know the reason for pipeline, u can see my above script,
          build-log.txt was copied by myself, while in job, it was generated by the plugin

          Show
          freemail165 Tiejun Sha added a comment - seems i know the reason for pipeline, u can see my above script, build-log.txt was copied by myself, while in job, it was generated by the plugin
          Hide
          a_goulti Alexandra Goultiaeva added a comment -

          You are correct about the pipeline. Only the Stdout upload mode was affected by this bug, the file handler got leaked when it was taking in Stdout directly from Jenkins.

          I don't see why it would work on another master; I'm guessing it doesn't, but you don't run into the "too many open files" for a different reasons. Maybe that machine gets to do less uploads? Something else is using less file handles? Restarted more often? Different system with different limits?

          You can try to grep for "/log" in the output of "lsof" to see if old build files are being kept open. Searching for "deleted" only shows the ones that your "Discard Old Build plugin" happened to remove.

           

          Regarding timing, I see my change was reviewed so it's just a matter of releasing it. I will try to get a release out next week.

          Show
          a_goulti Alexandra Goultiaeva added a comment - You are correct about the pipeline. Only the Stdout upload mode was affected by this bug, the file handler got leaked when it was taking in Stdout directly from Jenkins. I don't see why it would work on another master; I'm guessing it doesn't, but you don't run into the "too many open files" for a different reasons. Maybe that machine gets to do less uploads? Something else is using less file handles? Restarted more often? Different system with different limits? You can try to grep for "/log" in the output of "lsof" to see if old build files are being kept open. Searching for "deleted" only shows the ones that your "Discard Old Build plugin" happened to remove.   Regarding timing, I see my change was reviewed so it's just a matter of releasing it. I will try to get a release out next week.
          Hide
          a_goulti Alexandra Goultiaeva added a comment -

          The fix has been released as of Aug 3

          Show
          a_goulti Alexandra Goultiaeva added a comment - The fix has been released as of Aug 3

            People

            Assignee:
            a_goulti Alexandra Goultiaeva
            Reporter:
            freemail165 Tiejun Sha
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: