[JENKINS-52658] google-storage-plugin cause file handle leak

Type: Bug
Resolution: Fixed
Priority: Critical
Component/s: google-storage-plugin
Labels:
None
Environment:
Jenkins ver. 2.104
Google Cloud Storage Plugin 1.1

Similar Issues:
Powered by SuggestiMate

Show

We got the issue as "too many files open"

After some searching/investigation, In the end , we realized the issue came from google-storage-plugin

repro steps are as below:

1) Create a freestyle project

2) add log upload as post-build action

3) set "Max # of builds to keep" to 3

4) rebuild the project a few times

5) get processId with ps -ef | grep java | grep -v daemon | grep -v grep | awk '

{ print $2}

6) lsof -p $processId | grep deleted

I can get below

java 6480 jenkins 780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.1/log (*deleted)

java 6480 jenkins 780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.2/log (*deleted)

java 6480 jenkins 780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.3/log (*deleted)

java 6480 jenkins 780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.4/log (*deleted)

BTW, for a finished job #5 but hasn't been deleted, we can get below result.

java still hold the open file of the job log which seems to be a issue

lsof -p 6480 | grep file-handle-leak

java 6480 jenkins *809r REG 8,1 466655 10234118 /var/lib/jenkins/jobs/file-handle-leak/builds/5/log

Tiejun Sha created issue - 2018-07-19 15:10

Tiejun Sha added a comment - 2018-07-19 16:15 - edited

On the same Jenkins master, we have pipeline which is using below for log upload, issue not happen

node('master') {
dir ("../builds/${BUILD_ID}") {
sh(script: "cp log build-log.txt")
script {
googleStorageUpload bucket: "gs://*****/${JOB_NAME}/${BUILD_ID}", credentialsId: '*****', pattern: 'build-log.txt'
}
}
}

Tiejun Sha added a comment - 2018-07-19 16:15 - edited On the same Jenkins master, we have pipeline which is using below for log upload, issue not happen node('master') { dir ("../builds/${BUILD_ID}") { sh(script: "cp log build-log.txt") script { googleStorageUpload bucket: "gs://***** /${JOB_NAME}/${BUILD_ID}", credentialsId: ' *****', pattern: 'build-log.txt' } } }

Tiejun Sha made changes - 2018-07-19 16:16

Description

Original: We got the issue as "too many files open"

After some searching/investigation, In the end , we realized the issue came from google-storage-plugin

repro steps are as below:

1) Create a freestyle project

2) add log upload as post-build action

3) set "Max # of builds to keep" to 3

4) rebuild the project a few times

5) get processId with ps -ef | grep java | grep -v daemon | grep -v grep | awk '{ print $2}'

6) lsof -p $processId | grep deleted

I can get below

java 6480 jenkins *780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.1/log (*deleted*)

java 6480 jenkins *780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.2/log (*deleted*)

java 6480 jenkins *780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.3/log (*deleted*)

java 6480 jenkins *780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.4/log (*deleted*)

BTW, for a finished job #13455 but hasn't been deleted, we can get below result.

java still hold the open file of the job log which seems to be a issue

### I tried with a pipeline job, after pipeline finished , this won't happen

lsof -p 6480 | grep 13455

java 6480 jenkins *809r REG 8,1 466655 10234118 /var/lib/jenkins/jobs/file-handle-leak/builds/*13455*/log

New: We got the issue as "too many files open"

After some searching/investigation, In the end , we realized the issue came from google-storage-plugin

repro steps are as below:

1) Create a freestyle project

2) add log upload as post-build action

3) set "Max # of builds to keep" to 3

4) rebuild the project a few times

5) get processId with ps -ef | grep java | grep -v daemon | grep -v grep | awk '{ print $2}'

6) lsof -p $processId | grep deleted

I can get below

java 6480 jenkins *780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.1/log (*deleted*)

java 6480 jenkins *780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.2/log (*deleted*)

java 6480 jenkins *780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.3/log (*deleted*)

java 6480 jenkins *780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.4/log (*deleted*)

BTW, for a finished job #5 but hasn't been deleted, we can get below result.

java still hold the open file of the job log which seems to be a issue

### I tried with a pipeline job, after pipeline finished , this won't happen

lsof -p 6480 | grep file-handle-leak

java 6480 jenkins *809r REG 8,1 466655 10234118 /var/lib/jenkins/jobs/file-handle-leak/builds/5/log

Tiejun Sha made changes - 2018-07-19 16:17

Description

Original: We got the issue as "too many files open"

After some searching/investigation, In the end , we realized the issue came from google-storage-plugin

repro steps are as below:

1) Create a freestyle project

2) add log upload as post-build action

3) set "Max # of builds to keep" to 3

4) rebuild the project a few times

5) get processId with ps -ef | grep java | grep -v daemon | grep -v grep | awk '{ print $2}'

6) lsof -p $processId | grep deleted

I can get below

java 6480 jenkins *780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.1/log (*deleted*)

java 6480 jenkins *780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.2/log (*deleted*)

java 6480 jenkins *780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.3/log (*deleted*)

java 6480 jenkins *780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.4/log (*deleted*)

BTW, for a finished job #5 but hasn't been deleted, we can get below result.

java still hold the open file of the job log which seems to be a issue

### I tried with a pipeline job, after pipeline finished , this won't happen

lsof -p 6480 | grep file-handle-leak

java 6480 jenkins *809r REG 8,1 466655 10234118 /var/lib/jenkins/jobs/file-handle-leak/builds/5/log

New: We got the issue as "too many files open"

After some searching/investigation, In the end , we realized the issue came from google-storage-plugin

repro steps are as below:

1) Create a freestyle project

2) add log upload as post-build action

3) set "Max # of builds to keep" to 3

4) rebuild the project a few times

5) get processId with ps -ef | grep java | grep -v daemon | grep -v grep | awk '{ print $2}'

6) lsof -p $processId | grep deleted

I can get below

java 6480 jenkins *780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.1/log (*deleted*)

java 6480 jenkins *780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.2/log (*deleted*)

java 6480 jenkins *780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.3/log (*deleted*)

java 6480 jenkins *780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.4/log (*deleted*)

BTW, for a finished job #5 but hasn't been deleted, we can get below result.

java still hold the open file of the job log which seems to be a issue

lsof -p 6480 | grep file-handle-leak

java 6480 jenkins *809r REG 8,1 466655 10234118 /var/lib/jenkins/jobs/file-handle-leak/builds/5/log

Alexandra Goultiaeva added a comment - 2018-07-19 17:08

Thanks for the bug report, and for detailed steps to reproduce.

I can confirm that I can reproduce this, looking deeper into this.

Alexandra Goultiaeva added a comment - 2018-07-19 17:08 Thanks for the bug report, and for detailed steps to reproduce. I can confirm that I can reproduce this, looking deeper into this.

Tiejun Sha added a comment - 2018-07-19 17:21 - edited

BTW, we have another jenkins master which seems work fine with below version
==> I didn't try above steps, but we never met "too many files open" issue there

Jenkins ver. 2.34
Google Cloud Storage Plugin 1.1

Tiejun Sha added a comment - 2018-07-19 17:21 - edited BTW, we have another jenkins master which seems work fine with below version ==> I didn't try above steps, but we never met "too many files open" issue there Jenkins ver. 2.34 Google Cloud Storage Plugin 1.1

Alexandra Goultiaeva added a comment - 2018-07-19 21:17

Yes, the problem was that the Stdout Upload step never closed the input stream it was reading from.

The fix is out for review as a PR here: https://github.com/jenkinsci/google-storage-plugin/pull/46

Thanks again for reporting with an easy repro!

Alexandra Goultiaeva added a comment - 2018-07-19 21:17 Yes, the problem was that the Stdout Upload step never closed the input stream it was reading from. The fix is out for review as a PR here: https://github.com/jenkinsci/google-storage-plugin/pull/46 Thanks again for reporting with an easy repro!

Tiejun Sha added a comment - 2018-07-19 21:38

any idea for below:
1) why pipeline works well on the same jenkins master
2) why the same version plugin works on another jenkins master

Tiejun Sha added a comment - 2018-07-19 21:38 any idea for below: 1) why pipeline works well on the same jenkins master 2) why the same version plugin works on another jenkins master

Tiejun Sha added a comment - 2018-07-19 21:53

BTW, what is the ETA for this change to be released?
I assuming that it will be 1.2, is that correct?

Tiejun Sha added a comment - 2018-07-19 21:53 BTW, what is the ETA for this change to be released? I assuming that it will be 1.2, is that correct?

Tiejun Sha added a comment - 2018-07-19 22:30

seems i know the reason for pipeline, u can see my above script,
build-log.txt was copied by myself, while in job, it was generated by the plugin

Tiejun Sha added a comment - 2018-07-19 22:30 seems i know the reason for pipeline, u can see my above script, build-log.txt was copied by myself, while in job, it was generated by the plugin

Assignee:: Alexandra Goultiaeva

Reporter:: Tiejun Sha

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Created:: 2018-07-19 15:10

Updated:: 2018-08-20 14:41

Resolved:: 2018-08-20 14:41

Jenkins

Details

Description

Attachments

Activity

Collapse comment: Tiejun Sha added a comment - 2018-07-19 16:15, Edited by Tiejun Sha - 2018-07-19 16:29

Expand comment: Tiejun Sha added a comment - 2018-07-19 16:15, Edited by Tiejun Sha - 2018-07-19 16:29

Collapse comment: Alexandra Goultiaeva added a comment - 2018-07-19 17:08

Expand comment: Alexandra Goultiaeva added a comment - 2018-07-19 17:08

Collapse comment: Tiejun Sha added a comment - 2018-07-19 17:21, Edited by Tiejun Sha - 2018-07-19 17:22

Expand comment: Tiejun Sha added a comment - 2018-07-19 17:21, Edited by Tiejun Sha - 2018-07-19 17:22

Collapse comment: Alexandra Goultiaeva added a comment - 2018-07-19 21:17

Expand comment: Alexandra Goultiaeva added a comment - 2018-07-19 21:17

Collapse comment: Tiejun Sha added a comment - 2018-07-19 21:38

Expand comment: Tiejun Sha added a comment - 2018-07-19 21:38

Collapse comment: Tiejun Sha added a comment - 2018-07-19 21:53

Expand comment: Tiejun Sha added a comment - 2018-07-19 21:53

Collapse comment: Tiejun Sha added a comment - 2018-07-19 22:30

Expand comment: Tiejun Sha added a comment - 2018-07-19 22:30

People

Dates