Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52658

google-storage-plugin cause file handle leak

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: google-storage-plugin
    • Labels:
      None
    • Environment:
      Jenkins ver. 2.104
      Google Cloud Storage Plugin 1.1
    • Similar Issues:

      Description

      We got the issue as "too many files open"

      After some searching/investigation, In the end , we realized the issue came from google-storage-plugin

      repro steps are as below:

      1) Create a freestyle project

      2) add log upload as post-build action

      3) set "Max # of builds to keep" to 3

      4) rebuild the project a few times

      5) get processId with ps -ef | grep java | grep -v daemon | grep -v grep | awk '

      { print $2}

      '

      6) lsof -p $processId | grep deleted

      I can get below

      java    6480 jenkins 780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.1/log (*deleted)

      java    6480 jenkins 780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.2/log (*deleted)

      java    6480 jenkins 780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.3/log (*deleted)

      java    6480 jenkins 780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.4/log (*deleted)

       

      BTW, for a finished job #5 but hasn't been deleted, we can get below result.

      java still hold the open file of the job log which seems to be a issue

       lsof -p 6480 | grep file-handle-leak

      java    6480 jenkins *809r   REG                8,1   466655   10234118 /var/lib/jenkins/jobs/file-handle-leak/builds/5/log

        Attachments

          Activity

          freemail165 Tiejun Sha created issue -
          freemail165 Tiejun Sha made changes -
          Field Original Value New Value
          Description We got the issue as "too many files open"

          After some searching/investigation, In the end , we realized the issue came from google-storage-plugin

          repro steps are as below:

          1) Create a freestyle project

          2) add log upload as post-build action

          3) set "Max # of builds to keep" to 3

          4) rebuild the project a few times

          5) get processId with ps -ef | grep java | grep -v daemon | grep -v grep | awk '{ print $2}'

          6) lsof -p $processId | grep deleted

          I can get below

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.1/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.2/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.3/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.4/log (*deleted*)

           

          BTW, for a finished job #13455 but hasn't been deleted, we can get below result.

          java still hold the open file of the job log which seems to be a issue

          ### I tried with a pipeline job, after pipeline finished , this won't happen

           lsof -p 6480 | grep 13455

           

          java    6480 jenkins *809r   REG                8,1   466655   10234118 /var/lib/jenkins/jobs/file-handle-leak/builds/*13455*/log
          We got the issue as "too many files open"

          After some searching/investigation, In the end , we realized the issue came from google-storage-plugin

          repro steps are as below:

          1) Create a freestyle project

          2) add log upload as post-build action

          3) set "Max # of builds to keep" to 3

          4) rebuild the project a few times

          5) get processId with ps -ef | grep java | grep -v daemon | grep -v grep | awk '{ print $2}'

          6) lsof -p $processId | grep deleted

          I can get below

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.1/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.2/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.3/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.4/log (*deleted*)

           

          BTW, for a finished job #5 but hasn't been deleted, we can get below result.

          java still hold the open file of the job log which seems to be a issue

          ### I tried with a pipeline job, after pipeline finished , this won't happen

           lsof -p 6480 | grep file-handle-leak


          java    6480 jenkins *809r   REG                8,1   466655   10234118 /var/lib/jenkins/jobs/file-handle-leak/builds/5/log
          freemail165 Tiejun Sha made changes -
          Description We got the issue as "too many files open"

          After some searching/investigation, In the end , we realized the issue came from google-storage-plugin

          repro steps are as below:

          1) Create a freestyle project

          2) add log upload as post-build action

          3) set "Max # of builds to keep" to 3

          4) rebuild the project a few times

          5) get processId with ps -ef | grep java | grep -v daemon | grep -v grep | awk '{ print $2}'

          6) lsof -p $processId | grep deleted

          I can get below

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.1/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.2/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.3/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.4/log (*deleted*)

           

          BTW, for a finished job #5 but hasn't been deleted, we can get below result.

          java still hold the open file of the job log which seems to be a issue

          ### I tried with a pipeline job, after pipeline finished , this won't happen

           lsof -p 6480 | grep file-handle-leak


          java    6480 jenkins *809r   REG                8,1   466655   10234118 /var/lib/jenkins/jobs/file-handle-leak/builds/5/log
          We got the issue as "too many files open"

          After some searching/investigation, In the end , we realized the issue came from google-storage-plugin

          repro steps are as below:

          1) Create a freestyle project

          2) add log upload as post-build action

          3) set "Max # of builds to keep" to 3

          4) rebuild the project a few times

          5) get processId with ps -ef | grep java | grep -v daemon | grep -v grep | awk '{ print $2}'

          6) lsof -p $processId | grep deleted

          I can get below

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.1/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.2/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.3/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.4/log (*deleted*)

           

          BTW, for a finished job #5 but hasn't been deleted, we can get below result.

          java still hold the open file of the job log which seems to be a issue

           lsof -p 6480 | grep file-handle-leak


          java    6480 jenkins *809r   REG                8,1   466655   10234118 /var/lib/jenkins/jobs/file-handle-leak/builds/5/log
          a_goulti Alexandra Goultiaeva made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Closed [ 6 ]

            People

            Assignee:
            a_goulti Alexandra Goultiaeva
            Reporter:
            freemail165 Tiejun Sha
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: