-
Bug
-
Resolution: Fixed
-
Critical
-
None
-
Jenkins ver. 2.104
Google Cloud Storage Plugin 1.1
We got the issue as "too many files open"
After some searching/investigation, In the end , we realized the issue came from google-storage-plugin
repro steps are as below:
1) Create a freestyle project
2) add log upload as post-build action
3) set "Max # of builds to keep" to 3
4) rebuild the project a few times
5) get processId with ps -ef | grep java | grep -v daemon | grep -v grep | awk '
{ print $2}'
6) lsof -p $processId | grep deleted
I can get below
java 6480 jenkins 780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.1/log (*deleted)
java 6480 jenkins 780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.2/log (*deleted)
java 6480 jenkins 780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.3/log (*deleted)
java 6480 jenkins 780r REG 8,1 2178 19014020 /var/lib/jenkins/jobs/file-handle-leak/builds/.4/log (*deleted)
BTW, for a finished job #5 but hasn't been deleted, we can get below result.
java still hold the open file of the job log which seems to be a issue
lsof -p 6480 | grep file-handle-leak
java 6480 jenkins *809r REG 8,1 466655 10234118 /var/lib/jenkins/jobs/file-handle-leak/builds/5/log
[JENKINS-52658] google-storage-plugin cause file handle leak
Thanks for the bug report, and for detailed steps to reproduce.
I can confirm that I can reproduce this, looking deeper into this.
BTW, we have another jenkins master which seems work fine with below version
==> I didn't try above steps, but we never met "too many files open" issue there
Jenkins ver. 2.34
Google Cloud Storage Plugin 1.1
Tiejun Sha
added a comment - - edited BTW, we have another jenkins master which seems work fine with below version
==> I didn't try above steps, but we never met "too many files open" issue there
Jenkins ver. 2.34
Google Cloud Storage Plugin 1.1 Yes, the problem was that the Stdout Upload step never closed the input stream it was reading from.
The fix is out for review as a PR here: https://github.com/jenkinsci/google-storage-plugin/pull/46
Thanks again for reporting with an easy repro!
Tiejun Sha
added a comment - any idea for below:
1) why pipeline works well on the same jenkins master
2) why the same version plugin works on another jenkins master
Tiejun Sha
added a comment - any idea for below:
1) why pipeline works well on the same jenkins master
2) why the same version plugin works on another jenkins master Tiejun Sha
added a comment - BTW, what is the ETA for this change to be released?
I assuming that it will be 1.2, is that correct?
Tiejun Sha
added a comment - BTW, what is the ETA for this change to be released?
I assuming that it will be 1.2, is that correct? Tiejun Sha
added a comment - seems i know the reason for pipeline, u can see my above script,
build-log.txt was copied by myself, while in job, it was generated by the plugin
Tiejun Sha
added a comment - seems i know the reason for pipeline, u can see my above script,
build-log.txt was copied by myself, while in job, it was generated by the plugin You are correct about the pipeline. Only the Stdout upload mode was affected by this bug, the file handler got leaked when it was taking in Stdout directly from Jenkins.
I don't see why it would work on another master; I'm guessing it doesn't, but you don't run into the "too many open files" for a different reasons. Maybe that machine gets to do less uploads? Something else is using less file handles? Restarted more often? Different system with different limits?
You can try to grep for "/log" in the output of "lsof" to see if old build files are being kept open. Searching for "deleted" only shows the ones that your "Discard Old Build plugin" happened to remove.
Regarding timing, I see my change was reviewed so it's just a matter of releasing it. I will try to get a release out next week.
On the same Jenkins master, we have pipeline which is using below for log upload, issue not happen
node('master') {
dir ("../builds/${BUILD_ID}") {
sh(script: "cp log build-log.txt")
script {
googleStorageUpload bucket: "gs://*****/${JOB_NAME}/${BUILD_ID}", credentialsId: '*****', pattern: 'build-log.txt'
}
}
}