Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48280

File handle leak

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • core
    • None
    • Jenkins 2.73.3

      After upgrade to 2.73.3 we noticed that after a few days Jenkins becomes unresponsive and gives "too many open files" error. The only way to overcome the issue is to restart the instance.

      The File Leak Detector Plugin does show some open handles for completed jobs but after a few hours those are pruned by themselves from the list. The total amount of open file handles never exceeds 2000 according to the plugin.

      However when we run 'lsof' we see the increase of open file handles over time (see the screenshot attached: the big drop on Nov 27 is due to the manual Jenkins restart).

      Here's the summary of 2 lsof runs with 1 day between them. The list of top files:

      Nov-17:

      100632 slave.log
32294 log
7685 timestamps
4193 random
3635 urandom

      Nov-18:

      708532 log
297707 timestamps
98280 slave.log
90675 Common.groovy
85995 BobHelper.groovy

      Common.groovy and BobHelper.groovy are used by some jobs.

        1. dpi-jenkins.png
          dpi-jenkins.png
          28 kB

          [JENKINS-48280] File handle leak

          Volodymyr Sobotovych created issue -
          Volodymyr Sobotovych made changes -
          Attachment New: Screen Shot 2017-11-29 at 9.45.10 AM.png [ 40525 ]
          Volodymyr Sobotovych made changes -
          Attachment Original: Screen Shot 2017-11-29 at 9.45.10 AM.png [ 40525 ]
          Volodymyr Sobotovych made changes -
          Attachment New: dpi-jenkins.png [ 40526 ]
          Volodymyr Sobotovych made changes -
          Description Original: After upgrade to 2.73.3 we noticed that after a few days Jenkins becomes unresponsive and gives "too many open files" error. The only way to overcome the issue is to restart the instance.

          The File Leak Detector Plugin does show some open handles for completed jobs but after a few hours those are pruned by themselves from the list. The total amount of open file handles never exceeds 2000 according to the plugin.

          However when we run 'lsof' we see the increase of open file handles over time (see the screenshot attached).

          Here's the summary of 2 lsof runs with 1 day between them. The list of top files:

          Nov-17:
          {code:java}
          100632 slave.log
          32294 log
          7685 timestamps
          4193 random
          3635 urandom{code}

          Nov-18:
          {code}
          708532 log
          297707 timestamps
          98280 slave.log
          90675 Common.groovy
          85995 BobHelper.groovy
          {code}

          Common.groovy and BobHelper.groovy are used by some jobs.           		New: After upgrade to 2.73.3 we noticed that after a few days Jenkins becomes unresponsive and gives "too many open files" error. The only way to overcome the issue is to restart the instance.

          The File Leak Detector Plugin does show some open handles for completed jobs but after a few hours those are pruned by themselves from the list. The total amount of open file handles never exceeds 2000 according to the plugin.

          However when we run 'lsof' we see the increase of open file handles over time (see the screenshot attached: the big drop on Nov 27 is due to the manual Jenkins restart).

          Here's the summary of 2 lsof runs with 1 day between them. The list of top files:

          Nov-17:
          {code:java}
          100632 slave.log
          32294 log
          7685 timestamps
          4193 random
          3635 urandom{code}

          Nov-18:
          {code}
          708532 log
          297707 timestamps
          98280 slave.log
          90675 Common.groovy
          85995 BobHelper.groovy
          {code}

          Common.groovy and BobHelper.groovy are used by some jobs.
          Volodymyr Sobotovych made changes -
          Link New: This issue is blocked by JENKINS-48244 [ JENKINS-48244 ]

          Volodymyr Sobotovych added a comment -

          jglick, oleg_nenashev this looks related to https://issues.jenkins-ci.org/browse/JENKINS-45057. Can you have a look?

          Volodymyr Sobotovych added a comment - jglick , oleg_nenashev this looks related to https://issues.jenkins-ci.org/browse/JENKINS-45057 . Can you have a look?
          Volodymyr Sobotovych made changes -
          Link Original: This issue is blocked by JENKINS-48244 [ JENKINS-48244 ]
          Oleg Nenashev made changes -
          Link New: This issue is related to JENKINS-45057 [ JENKINS-45057 ]

          Denis Shvedchenko added a comment -

          Using it in systemgroovystep to collect info about builds, 

          va 7 pentaho 1042r REG 259,0 2296 1085803614 /var/jenkins_home/jobs/EIS/jobs/pgsf_stage/jobs/test_b4596/jobs/TEST02/builds/648/log
COMMAND  PID  TID    USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME
java 7 8557 pentaho 14w REG 259,0 139604708 3227706987 /var/log/jenkins/jenkins.log
> java 7 8557 pentaho 104r REG 0,43 23645 1525 /var/cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar
> java 7 8557 pentaho 669r REG 259,0 489884 1085740354 /var/jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar
> java 7 8557 pentaho 779r REG 259,0 48055 3232726251 /var/jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar
> java 7 8557 pentaho 888r REG 259,0 352291 12583595 /var/jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar
> java 7 8558 pentaho mem REG 259,0 352291 12583595 /var/jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar
> java 7 8558 pentaho mem REG 259,0 48055 3232726251 /var/jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar
> java 7 8558 pentaho mem REG 259,0 489884 1085740354 /var/jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar
> java 7 8558 pentaho mem REG 0,43 23645 1525 /var/cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar
> java 7 8558 pentaho 14w REG 259,0 139604708 3227706987 /var/log/jenkins/jenkins.log
> java 7 8558 pentaho 104r REG 0,43 23645 1525 /var/cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar
> java 7 8558 pentaho 669r REG 259,0 489884 1085740354 /var/jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar
> java 7 8558 pentaho 779r REG 259,0 48055 3232726251 /var/jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar
> java 7 8558 pentaho 888r REG 259,0 352291 12583595 /var/jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar
> java 7 8559 pentaho mem REG 259,0 352291 12583595 /var/jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar
> java 7 8559 pentaho mem REG 259,0 48055 3232726251 /var/jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar
> java 7 8559 pentaho mem REG 259,0 489884 1085740354 /var/jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar
> java 7 8559 pentaho mem REG 0,43 23645 1525 /var/cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar
> java 7 8559 pentaho 14w REG 259,0 139604708 3227706987 /var/log/jenkins/jenkins.log
> java 7 8559 pentaho 104r REG 0,43 23645 1525 /var/cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar
> java 7 8559 pentaho 669r REG 259,0 489884 1085740354 /var/jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar
> java 7 8559 pentaho 779r REG 259,0 48055 3232726251 /var/jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar
> java 7 8559 pentaho 888r REG 259,0 352291 12583595 /var/jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar
> java 7 8560 pentaho mem REG 259,0 352291 12583595 /var/jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar
> java 7 8560 pentaho mem REG 259,0 48055 3232726251 /var/jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar
> java 7 8560 pentaho mem REG 259,0 489884 1085740354 /var/jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar
> java 7 8560 pentaho mem REG 0,43 23645 1525 /var/cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar
> java 7 8560 pentaho 14w REG 259,0 139604708 3227706987 /var/log/jenkins/jenkins.log
> java 7 8560 pentaho 104r REG 0,43 23645 1525 /var/cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar
> java 7 8560 pentaho 669r REG 259,0 489884 1085740354 /var/jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar
> java 7 8560 pentaho 779r REG 259,0 48055 3232726251 /var/jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar
> java 7 8560 pentaho 888r REG 259,0 352291 12583595 /var/jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar

           

          this is lsof output. every execution adds another TID left opened log and logging related jars

           

           

          Denis Shvedchenko added a comment - Using it in systemgroovystep to collect info about builds,  va 7 pentaho 1042r REG 259,0 2296 1085803614 / var /jenkins_home/jobs/EIS/jobs/pgsf_stage/jobs/test_b4596/jobs/TEST02/builds/648/log COMMAND  PID  TID    USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME java 7 8557 pentaho 14w REG 259,0 139604708 3227706987 / var /log/jenkins/jenkins.log > java 7 8557 pentaho 104r REG 0,43 23645 1525 / var /cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar > java 7 8557 pentaho 669r REG 259,0 489884 1085740354 / var /jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar > java 7 8557 pentaho 779r REG 259,0 48055 3232726251 / var /jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar > java 7 8557 pentaho 888r REG 259,0 352291 12583595 / var /jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar > java 7 8558 pentaho mem REG 259,0 352291 12583595 / var /jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar > java 7 8558 pentaho mem REG 259,0 48055 3232726251 / var /jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar > java 7 8558 pentaho mem REG 259,0 489884 1085740354 / var /jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar > java 7 8558 pentaho mem REG 0,43 23645 1525 / var /cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar > java 7 8558 pentaho 14w REG 259,0 139604708 3227706987 / var /log/jenkins/jenkins.log > java 7 8558 pentaho 104r REG 0,43 23645 1525 / var /cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar > java 7 8558 pentaho 669r REG 259,0 489884 1085740354 / var /jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar > java 7 8558 pentaho 779r REG 259,0 48055 3232726251 / var /jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar > java 7 8558 pentaho 888r REG 259,0 352291 12583595 / var /jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar > java 7 8559 pentaho mem REG 259,0 352291 12583595 / var /jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar > java 7 8559 pentaho mem REG 259,0 48055 3232726251 / var /jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar > java 7 8559 pentaho mem REG 259,0 489884 1085740354 / var /jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar > java 7 8559 pentaho mem REG 0,43 23645 1525 / var /cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar > java 7 8559 pentaho 14w REG 259,0 139604708 3227706987 / var /log/jenkins/jenkins.log > java 7 8559 pentaho 104r REG 0,43 23645 1525 / var /cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar > java 7 8559 pentaho 669r REG 259,0 489884 1085740354 / var /jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar > java 7 8559 pentaho 779r REG 259,0 48055 3232726251 / var /jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar > java 7 8559 pentaho 888r REG 259,0 352291 12583595 / var /jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar > java 7 8560 pentaho mem REG 259,0 352291 12583595 / var /jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar > java 7 8560 pentaho mem REG 259,0 48055 3232726251 / var /jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar > java 7 8560 pentaho mem REG 259,0 489884 1085740354 / var /jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar > java 7 8560 pentaho mem REG 0,43 23645 1525 / var /cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar > java 7 8560 pentaho 14w REG 259,0 139604708 3227706987 / var /log/jenkins/jenkins.log > java 7 8560 pentaho 104r REG 0,43 23645 1525 / var /cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar > java 7 8560 pentaho 669r REG 259,0 489884 1085740354 / var /jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar > java 7 8560 pentaho 779r REG 259,0 48055 3232726251 / var /jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar > java 7 8560 pentaho 888r REG 259,0 352291 12583595 / var /jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar   this is lsof output. every execution adds another TID left opened log and logging related jars    

            Unassigned Unassigned
            wheleph Volodymyr Sobotovych
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: