Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52658

google-storage-plugin cause file handle leak

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • google-storage-plugin
    • None
    • Jenkins ver. 2.104
      Google Cloud Storage Plugin 1.1

      We got the issue as "too many files open"

      After some searching/investigation, In the end , we realized the issue came from google-storage-plugin

      repro steps are as below:

      1) Create a freestyle project

      2) add log upload as post-build action

      3) set "Max # of builds to keep" to 3

      4) rebuild the project a few times

      5) get processId with ps -ef | grep java | grep -v daemon | grep -v grep | awk '

      { print $2}

      '

      6) lsof -p $processId | grep deleted

      I can get below

      java    6480 jenkins 780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.1/log (*deleted)

      java    6480 jenkins 780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.2/log (*deleted)

      java    6480 jenkins 780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.3/log (*deleted)

      java    6480 jenkins 780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.4/log (*deleted)

       

      BTW, for a finished job #5 but hasn't been deleted, we can get below result.

      java still hold the open file of the job log which seems to be a issue

       lsof -p 6480 | grep file-handle-leak

      java    6480 jenkins *809r   REG                8,1   466655   10234118 /var/lib/jenkins/jobs/file-handle-leak/builds/5/log

          [JENKINS-52658] google-storage-plugin cause file handle leak

          Tiejun Sha created issue -
          Tiejun Sha made changes -
          Description Original: We got the issue as "too many files open"

          After some searching/investigation, In the end , we realized the issue came from google-storage-plugin

          repro steps are as below:

          1) Create a freestyle project

          2) add log upload as post-build action

          3) set "Max # of builds to keep" to 3

          4) rebuild the project a few times

          5) get processId with ps -ef | grep java | grep -v daemon | grep -v grep | awk '{ print $2}'

          6) lsof -p $processId | grep deleted

          I can get below

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.1/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.2/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.3/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.4/log (*deleted*)

           

          BTW, for a finished job #13455 but hasn't been deleted, we can get below result.

          java still hold the open file of the job log which seems to be a issue

          ### I tried with a pipeline job, after pipeline finished , this won't happen

           lsof -p 6480 | grep 13455

           

          java    6480 jenkins *809r   REG                8,1   466655   10234118 /var/lib/jenkins/jobs/file-handle-leak/builds/*13455*/log           		New: We got the issue as "too many files open"

          After some searching/investigation, In the end , we realized the issue came from google-storage-plugin

          repro steps are as below:

          1) Create a freestyle project

          2) add log upload as post-build action

          3) set "Max # of builds to keep" to 3

          4) rebuild the project a few times

          5) get processId with ps -ef | grep java | grep -v daemon | grep -v grep | awk '{ print $2}'

          6) lsof -p $processId | grep deleted

          I can get below

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.1/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.2/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.3/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.4/log (*deleted*)

           

          BTW, for a finished job #5 but hasn't been deleted, we can get below result.

          java still hold the open file of the job log which seems to be a issue

          ### I tried with a pipeline job, after pipeline finished , this won't happen

           lsof -p 6480 | grep file-handle-leak


          java    6480 jenkins *809r   REG                8,1   466655   10234118 /var/lib/jenkins/jobs/file-handle-leak/builds/5/log
          Tiejun Sha made changes -
          Description Original: We got the issue as "too many files open"

          After some searching/investigation, In the end , we realized the issue came from google-storage-plugin

          repro steps are as below:

          1) Create a freestyle project

          2) add log upload as post-build action

          3) set "Max # of builds to keep" to 3

          4) rebuild the project a few times

          5) get processId with ps -ef | grep java | grep -v daemon | grep -v grep | awk '{ print $2}'

          6) lsof -p $processId | grep deleted

          I can get below

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.1/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.2/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.3/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.4/log (*deleted*)

           

          BTW, for a finished job #5 but hasn't been deleted, we can get below result.

          java still hold the open file of the job log which seems to be a issue

          ### I tried with a pipeline job, after pipeline finished , this won't happen

           lsof -p 6480 | grep file-handle-leak


          java    6480 jenkins *809r   REG                8,1   466655   10234118 /var/lib/jenkins/jobs/file-handle-leak/builds/5/log           		New: We got the issue as "too many files open"

          After some searching/investigation, In the end , we realized the issue came from google-storage-plugin

          repro steps are as below:

          1) Create a freestyle project

          2) add log upload as post-build action

          3) set "Max # of builds to keep" to 3

          4) rebuild the project a few times

          5) get processId with ps -ef | grep java | grep -v daemon | grep -v grep | awk '{ print $2}'

          6) lsof -p $processId | grep deleted

          I can get below

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.1/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.2/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.3/log (*deleted*)

          java    6480 jenkins *780r   REG                8,1     2178   19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.4/log (*deleted*)

           

          BTW, for a finished job #5 but hasn't been deleted, we can get below result.

          java still hold the open file of the job log which seems to be a issue

           lsof -p 6480 | grep file-handle-leak


          java    6480 jenkins *809r   REG                8,1   466655   10234118 /var/lib/jenkins/jobs/file-handle-leak/builds/5/log
          Alexandra Goultiaeva made changes -
          Resolution New: Fixed [ 1 ]
          Status Original: Open [ 1 ] New: Closed [ 6 ]

            a_goulti Alexandra Goultiaeva
            freemail165 Tiejun Sha
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: