• Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • core
    • None
    • Jenkins 2.73.3

      After upgrade to 2.73.3 we noticed that after a few days Jenkins becomes unresponsive and gives "too many open files" error. The only way to overcome the issue is to restart the instance.

      The File Leak Detector Plugin does show some open handles for completed jobs but after a few hours those are pruned by themselves from the list. The total amount of open file handles never exceeds 2000 according to the plugin.

      However when we run 'lsof' we see the increase of open file handles over time (see the screenshot attached: the big drop on Nov 27 is due to the manual Jenkins restart).

      Here's the summary of 2 lsof runs with 1 day between them. The list of top files:

      Nov-17:

      100632 slave.log
      32294 log
      7685 timestamps
      4193 random
      3635 urandom

      Nov-18:

      708532 log
      297707 timestamps
      98280 slave.log
      90675 Common.groovy
      85995 BobHelper.groovy
      

      Common.groovy and BobHelper.groovy are used by some jobs.

          [JENKINS-48280] File handle leak

          jglick, oleg_nenashev this looks related to https://issues.jenkins-ci.org/browse/JENKINS-45057. Can you have a look?

          Volodymyr Sobotovych added a comment - jglick , oleg_nenashev this looks related to https://issues.jenkins-ci.org/browse/JENKINS-45057 . Can you have a look?

          Using it in systemgroovystep to collect info about builds, 

          va 7 pentaho 1042r REG 259,0 2296 1085803614 /var/jenkins_home/jobs/EIS/jobs/pgsf_stage/jobs/test_b4596/jobs/TEST02/builds/648/log
          COMMAND  PID  TID    USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME
          java 7 8557 pentaho 14w REG 259,0 139604708 3227706987 /var/log/jenkins/jenkins.log
          > java 7 8557 pentaho 104r REG 0,43 23645 1525 /var/cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar
          > java 7 8557 pentaho 669r REG 259,0 489884 1085740354 /var/jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar
          > java 7 8557 pentaho 779r REG 259,0 48055 3232726251 /var/jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar
          > java 7 8557 pentaho 888r REG 259,0 352291 12583595 /var/jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar
          > java 7 8558 pentaho mem REG 259,0 352291 12583595 /var/jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar
          > java 7 8558 pentaho mem REG 259,0 48055 3232726251 /var/jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar
          > java 7 8558 pentaho mem REG 259,0 489884 1085740354 /var/jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar
          > java 7 8558 pentaho mem REG 0,43 23645 1525 /var/cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar
          > java 7 8558 pentaho 14w REG 259,0 139604708 3227706987 /var/log/jenkins/jenkins.log
          > java 7 8558 pentaho 104r REG 0,43 23645 1525 /var/cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar
          > java 7 8558 pentaho 669r REG 259,0 489884 1085740354 /var/jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar
          > java 7 8558 pentaho 779r REG 259,0 48055 3232726251 /var/jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar
          > java 7 8558 pentaho 888r REG 259,0 352291 12583595 /var/jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar
          > java 7 8559 pentaho mem REG 259,0 352291 12583595 /var/jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar
          > java 7 8559 pentaho mem REG 259,0 48055 3232726251 /var/jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar
          > java 7 8559 pentaho mem REG 259,0 489884 1085740354 /var/jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar
          > java 7 8559 pentaho mem REG 0,43 23645 1525 /var/cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar
          > java 7 8559 pentaho 14w REG 259,0 139604708 3227706987 /var/log/jenkins/jenkins.log
          > java 7 8559 pentaho 104r REG 0,43 23645 1525 /var/cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar
          > java 7 8559 pentaho 669r REG 259,0 489884 1085740354 /var/jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar
          > java 7 8559 pentaho 779r REG 259,0 48055 3232726251 /var/jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar
          > java 7 8559 pentaho 888r REG 259,0 352291 12583595 /var/jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar
          > java 7 8560 pentaho mem REG 259,0 352291 12583595 /var/jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar
          > java 7 8560 pentaho mem REG 259,0 48055 3232726251 /var/jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar
          > java 7 8560 pentaho mem REG 259,0 489884 1085740354 /var/jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar
          > java 7 8560 pentaho mem REG 0,43 23645 1525 /var/cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar
          > java 7 8560 pentaho 14w REG 259,0 139604708 3227706987 /var/log/jenkins/jenkins.log
          > java 7 8560 pentaho 104r REG 0,43 23645 1525 /var/cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar
          > java 7 8560 pentaho 669r REG 259,0 489884 1085740354 /var/jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar
          > java 7 8560 pentaho 779r REG 259,0 48055 3232726251 /var/jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar
          > java 7 8560 pentaho 888r REG 259,0 352291 12583595 /var/jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar
          

           

          this is lsof output. every execution adds another TID left opened log and logging related jars

           

           

          Denis Shvedchenko added a comment - Using it in systemgroovystep to collect info about builds,  va 7 pentaho 1042r REG 259,0 2296 1085803614 / var /jenkins_home/jobs/EIS/jobs/pgsf_stage/jobs/test_b4596/jobs/TEST02/builds/648/log COMMAND  PID  TID    USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME java 7 8557 pentaho 14w REG 259,0 139604708 3227706987 / var /log/jenkins/jenkins.log > java 7 8557 pentaho 104r REG 0,43 23645 1525 / var /cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar > java 7 8557 pentaho 669r REG 259,0 489884 1085740354 / var /jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar > java 7 8557 pentaho 779r REG 259,0 48055 3232726251 / var /jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar > java 7 8557 pentaho 888r REG 259,0 352291 12583595 / var /jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar > java 7 8558 pentaho mem REG 259,0 352291 12583595 / var /jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar > java 7 8558 pentaho mem REG 259,0 48055 3232726251 / var /jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar > java 7 8558 pentaho mem REG 259,0 489884 1085740354 / var /jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar > java 7 8558 pentaho mem REG 0,43 23645 1525 / var /cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar > java 7 8558 pentaho 14w REG 259,0 139604708 3227706987 / var /log/jenkins/jenkins.log > java 7 8558 pentaho 104r REG 0,43 23645 1525 / var /cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar > java 7 8558 pentaho 669r REG 259,0 489884 1085740354 / var /jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar > java 7 8558 pentaho 779r REG 259,0 48055 3232726251 / var /jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar > java 7 8558 pentaho 888r REG 259,0 352291 12583595 / var /jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar > java 7 8559 pentaho mem REG 259,0 352291 12583595 / var /jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar > java 7 8559 pentaho mem REG 259,0 48055 3232726251 / var /jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar > java 7 8559 pentaho mem REG 259,0 489884 1085740354 / var /jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar > java 7 8559 pentaho mem REG 0,43 23645 1525 / var /cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar > java 7 8559 pentaho 14w REG 259,0 139604708 3227706987 / var /log/jenkins/jenkins.log > java 7 8559 pentaho 104r REG 0,43 23645 1525 / var /cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar > java 7 8559 pentaho 669r REG 259,0 489884 1085740354 / var /jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar > java 7 8559 pentaho 779r REG 259,0 48055 3232726251 / var /jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar > java 7 8559 pentaho 888r REG 259,0 352291 12583595 / var /jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar > java 7 8560 pentaho mem REG 259,0 352291 12583595 / var /jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar > java 7 8560 pentaho mem REG 259,0 48055 3232726251 / var /jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar > java 7 8560 pentaho mem REG 259,0 489884 1085740354 / var /jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar > java 7 8560 pentaho mem REG 0,43 23645 1525 / var /cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar > java 7 8560 pentaho 14w REG 259,0 139604708 3227706987 / var /log/jenkins/jenkins.log > java 7 8560 pentaho 104r REG 0,43 23645 1525 / var /cache/jenkins/war/WEB-INF/lib/log4j-over-slf4j-1.7.25.jar > java 7 8560 pentaho 669r REG 259,0 489884 1085740354 / var /jenkins_home/plugins/docker-build-step/WEB-INF/lib/log4j-1.2.17.jar > java 7 8560 pentaho 779r REG 259,0 48055 3232726251 / var /jenkins_home/plugins/log-parser/WEB-INF/lib/log-parser.jar > java 7 8560 pentaho 888r REG 259,0 352291 12583595 / var /jenkins_home/plugins/graphiteIntegrator/WEB-INF/lib/log4j-1.2.9.jar   this is lsof output. every execution adds another TID left opened log and logging related jars    

          Oleg Nenashev added a comment -

          I am not sure whether it is an issue in the Jenkins core or on Groovy plugin

          Oleg Nenashev added a comment - I am not sure whether it is an issue in the Jenkins core or on Groovy plugin

          Oleg Nenashev added a comment -

          Is there a File Leak detector output with allocation traces?

          Oleg Nenashev added a comment - Is there a File Leak detector output with allocation traces?

          oleg_nenashev

          no, File Leak detector has no references to that files

          Denis Shvedchenko added a comment - oleg_nenashev no, File Leak detector has no references to that files

            Unassigned Unassigned
            wheleph Volodymyr Sobotovych
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: