-
Bug
-
Resolution: Won't Fix
-
Minor
-
None
-
Jenkins ver. 2.46.3
I use a Jenkins Pipeline to run my individual tests in parallel. I just noticed my test job that I run in parallel is failing with "Too many open files" errors.
Next, I ssh into the master and did an lsof -p using the pid of Jenkins... i see over 4k file descriptors. The output has thousands of lines that look like this:
java 7 root 4089w REG 202,96 28 2375888 /var/jenkins_home/jobs/test_pipeline/builds/15331/222.log java 7 root 4090w REG 202,96 328 787837 /var/jenkins_home/jobs/test_pipeline/builds/15343/166.log java 7 root 4091w REG 202,96 328 787851 /var/jenkins_home/jobs/test_pipeline/builds/15343/180.log java 7 root 4094w REG 202,96 671 787839 /var/jenkins_home/jobs/test_pipeline/builds/15343/167.log java 7 root 4095w REG 202,96 29 2361097 /var/jenkins_home/jobs/test_pipeline/builds/15331/157.log
Each file has content that roughly looks like this:
Starting building: ESC[8mha:////4CQAnoPVmHzYGq9xQUbp+LIf+xF38/zSfY1nZ0iZCPexAAAArB+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3Rdl
LBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAyZEgZF/az8JP2U/Pyi4pLUAt2ULKBh+fElqcUl+sYWliYGhvoAbOU0btYAAAA=ESC[0mmy_test #389401
- relates to
-
JENKINS-38381 [JEP-210] Optimize log handling in Pipeline and Durable Task
-
- Resolved
-
[JENKINS-44958] File handle leaks in *.log after hard kill
Sam Van Oort
added a comment - freemail165 Are you running the latest core and plugin versions? If so, can you provide a simple case that will reproduce the issue from scratch?
Sam Van Oort
added a comment - freemail165 Are you running the latest core and plugin versions? If so, can you provide a simple case that will reproduce the issue from scratch? 
Tiejun Sha
added a comment - core version means jenkins version? for plugin version, which plugin do u care about?
Tiejun Sha
added a comment - core version means jenkins version? for plugin version, which plugin do u care about? Sam Van Oort
added a comment - freemail165 Yes, "core" means Jenkins version. For plugins, we want the workflow-job, workflow-cps, durable-task, workflow-support, workflow-api, and workflow-basic-steps.
Sam Van Oort
added a comment - freemail165 Yes, "core" means Jenkins version. For plugins, we want the workflow-job, workflow-cps, durable-task, workflow-support, workflow-api, and workflow-basic-steps. Tiejun Sha
added a comment - Durable-task 1.17
from installed tab, I didn't find any plugins start from workflow
Tiejun Sha
added a comment - Jenkins ver. 2.104
Durable-task 1.17
from installed tab, I didn't find any plugins start from workflow 
Rick Liu
added a comment - What's JEP-210?
Is it this JEP-210: http://openjdk.java.net/jeps/210 No, https://jenkins.io/jep/210 (yes I complained when this system was introduced by rtyler that there was going to be confusion with Java Enhancement Proposals).
Sam Van Oort
added a comment - totoroliu Since Jesse forgot to communicate the key info: there's a huge change to logging that's just gone into beta release (available via the Experimental Update Center – see https://jenkins.io/doc/developer/publishing/releasing-experimental-updates/ ) that completely removes these extra log files.
I'd argue that qualifies as Resolution: Fixed rather than "Won't Fix" but ::shrug::
Sam Van Oort
added a comment - totoroliu Since Jesse forgot to communicate the key info: there's a huge change to logging that's just gone into beta release (available via the Experimental Update Center – see https://jenkins.io/doc/developer/publishing/releasing-experimental-updates/ ) that completely removes these extra log files.
I'd argue that qualifies as Resolution: Fixed rather than "Won't Fix" but ::shrug:: 
looks like when builds reach the limitation, jenkins delete expired builds but JVM still keep the link to those logs, that is the issue
ps -ef | grep java
ls -tl /proc/*****/fd | grep deleted
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 912 -> /var/lib/jenkins/jobs/metrics-report/builds/.66892/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 913 -> /var/lib/jenkins/jobs/metrics-report/builds/.66889/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 914 -> /var/lib/jenkins/jobs/metrics-report/builds/.66893/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 916 -> /var/lib/jenkins/jobs/metrics-report/builds/.66997/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 917 -> /var/lib/jenkins/jobs/performance-ios-build/builds/.4574/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 918 -> /var/lib/jenkins/jobs/performance-android-build/builds/.14546/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 920 -> /var/lib/jenkins/jobs/metrics-report/builds/.66933/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 921 -> /var/lib/jenkins/jobs/metrics-report/builds/.66888/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 922 -> /var/lib/jenkins/jobs/metrics-report/builds/.66891/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 924 -> /var/lib/jenkins/jobs/performance-android-build/builds/.14654/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 927 -> /var/lib/jenkins/jobs/metrics-report/builds/.66894/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 928 -> /var/lib/jenkins/jobs/performance-android-build/builds/.14568/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 929 -> /var/lib/jenkins/jobs/metrics-report/builds/.66896/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 931 -> /var/lib/jenkins/jobs/metrics-report/builds/.66895/log (deleted)
lr-x------ 1 jenkins jenkins 64 Jun 20 22:23 932 -> /var/lib/jenkins/jobs/metrics-report/builds/.66939/log (deleted)