Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-8881

After Log Rolling, Pervious, Uncompressed Log File is Deleted But _Not_ Closed and its Disk Space Not Freed

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Critical Critical
    • core
    • None
    • Linux, CentOS 5.5, kernel 2.6.18-194.el5; x86_64; Java 1.6.0_10-b33

      When Jenkins performs log-rolling, it compresses the completed log file leaving a date-stamped, ".gz"-suffixed equivalent while moving on to a new "jenkins.log" file. It also unlinks the old plain-text (non-compressed) log file. However, it does not close its descriptor to that log file and thus its disk space is not freed until Jenkins itself is restarted.

      On a Linux system, such ghost files can be discovered via entries in /proc/NNN/fd/*:

      1. ll /proc/[0-9][0-9]/fd |egrep '/var.(deleted)'

          [JENKINS-8881] After Log Rolling, Pervious, Uncompressed Log File is Deleted But _Not_ Closed and its Disk Space Not Freed

          This behavior is a big problem for us. Does no one else encounter it?

          Witness:

          2168# df -h
          Filesystem Size Used Avail Use% Mounted on
          ...
          /dev/mapper/vg1-var 7.8G 5.9G 1.6G 80% /var
          ...

          2169# /etc/init.d/jenkins stop
          Shutting down Jenkins [ OK ]

          2171# df -h
          Filesystem Size Used Avail Use% Mounted on
          ...
          /dev/mapper/vg1-var 7.8G 586M 6.8G 8% /var
          ...

          2172# /etc/init.d/jenkins start
          Starting Jenkins [ OK ]

          Randall Schulz added a comment - This behavior is a big problem for us. Does no one else encounter it? Witness: 2168# df -h Filesystem Size Used Avail Use% Mounted on ... /dev/mapper/vg1-var 7.8G 5.9G 1.6G 80% /var ... 2169# /etc/init.d/jenkins stop Shutting down Jenkins [ OK ] 2171# df -h Filesystem Size Used Avail Use% Mounted on ... /dev/mapper/vg1-var 7.8G 586M 6.8G 8% /var ... 2172# /etc/init.d/jenkins start Starting Jenkins [ OK ]

          A quick exchange with kohsuke that he asked me to transcribe here:

          RSchulz ‣ kohsuke: Can you comment on http://issues.jenkins-ci.org/browse/JENKINS-8881 ?
          jenkins-admin ‣ JENKINS-8881: ...
          kohsuke ‣ how did you install it?
          kohsuke ‣ RPM?
          RSchulz ‣ Yes.
          kohsuke ‣ Sounds like a bug in our packaging
          RSchulz ‣ Really?
          kohsuke ‣ Jenkins process needs to get SIGALM to reopen the log file
          kohsuke ‣ The logrotate config needs to include that instruction
          RSchulz ‣ Oh. I rarely associate signal processing with Java code.
          kohsuke ‣ When, that's because most Java developers don't know anything about Unix.
          kohsuke ‣ Not that I know much, but I try...
          RSchulz ‣ Well, I'm a Unix person from way, way back.
          kohsuke ‣ If you can contribute the proper logrotate setting, I can very easily integrate that to the RPM
          RSchulz ‣ I'm a Unix person from well before logrotate!
          kohsuke ‣ Otherwise just paste this conversation into the comment and hope someone else would be able to write it
          RSchulz ‣ It's odd that Jenkins compresses the log file and moves on to a new one but doesn't close the descriptor to the previous file (or files).nkins isn't doing that
          kohsuke ‣ logrotate is doing it
          RSchulz ‣ I'm pretty unclear on the relationship between them and how they interact (or fail to interact properly).

          Randall Schulz added a comment - A quick exchange with kohsuke that he asked me to transcribe here: RSchulz ‣ kohsuke: Can you comment on http://issues.jenkins-ci.org/browse/JENKINS-8881 ? jenkins-admin ‣ JENKINS-8881 : ... kohsuke ‣ how did you install it? kohsuke ‣ RPM? RSchulz ‣ Yes. kohsuke ‣ Sounds like a bug in our packaging RSchulz ‣ Really? kohsuke ‣ Jenkins process needs to get SIGALM to reopen the log file kohsuke ‣ The logrotate config needs to include that instruction RSchulz ‣ Oh. I rarely associate signal processing with Java code. kohsuke ‣ When, that's because most Java developers don't know anything about Unix. kohsuke ‣ Not that I know much, but I try... RSchulz ‣ Well, I'm a Unix person from way, way back. kohsuke ‣ If you can contribute the proper logrotate setting, I can very easily integrate that to the RPM RSchulz ‣ I'm a Unix person from well before logrotate! kohsuke ‣ Otherwise just paste this conversation into the comment and hope someone else would be able to write it RSchulz ‣ It's odd that Jenkins compresses the log file and moves on to a new one but doesn't close the descriptor to the previous file (or files).nkins isn't doing that kohsuke ‣ logrotate is doing it RSchulz ‣ I'm pretty unclear on the relationship between them and how they interact (or fail to interact properly).

          scooper4711 added a comment -

          I believe that this issue is related to JENKINS-5784.
          I had a situation with an 8 gig deleted and unclosed log file like this bug reports, and I did a kill -SIGALRM on the jenkins process and the file was then closed and the space freed up.

          scooper4711 added a comment - I believe that this issue is related to JENKINS-5784 . I had a situation with an 8 gig deleted and unclosed log file like this bug reports, and I did a kill -SIGALRM on the jenkins process and the file was then closed and the space freed up.

          akiko_pusu added a comment -

          Hello, I saw the same situation on my two servers installed Jenkins.
          OS environment is also the same to CneetOS 5.5.

          1. lsb_release -a
            LSB Version: :core-3.1-ia32:core-3.1-noarch:graphics-3.1-ia32:graphics-3.1-noarch
            Distributor ID: CentOS
            Description: CentOS release 5.5 (Final)
            Release: 5.5
            Codename: Final

          In addition to, I've installed Jenkins via RPM (exactly, update from hudson to jenkins with yum update).

          I believe this will be fixed with following this issue: JENKINS-5784.
          https://issues.jenkins-ci.org/browse/JENKINS-5784

          At first of this week I've updated one of my jenkins to 1.418 with yum update, but /etc/logrotate.d/jenkins is the same to previous version.

          And I think using "kill -HUP" is better for CentOS, as other logrotate config file does.

          I'm sorry if this bug related to logrotate was already fixed.

          akiko_pusu added a comment - Hello, I saw the same situation on my two servers installed Jenkins. OS environment is also the same to CneetOS 5.5. lsb_release -a LSB Version: :core-3.1-ia32:core-3.1-noarch:graphics-3.1-ia32:graphics-3.1-noarch Distributor ID: CentOS Description: CentOS release 5.5 (Final) Release: 5.5 Codename: Final In addition to, I've installed Jenkins via RPM (exactly, update from hudson to jenkins with yum update). I believe this will be fixed with following this issue: JENKINS-5784 . https://issues.jenkins-ci.org/browse/JENKINS-5784 At first of this week I've updated one of my jenkins to 1.418 with yum update, but /etc/logrotate.d/jenkins is the same to previous version. And I think using "kill -HUP" is better for CentOS, as other logrotate config file does. I'm sorry if this bug related to logrotate was already fixed.

          akiko_pusu added a comment -

          I reviewed my another CentOS CI server which using not jenkins but "hudson". And the server is also CentOS and hudson was installed RPM.

          This server has been running without any problem around 1 years.
          FYI: /etc/logrotate.d/hudson is using "kill -SIGALRM `cat /var/run/hudson.pid`", not using "kill -s SIGALRM".

          And the size of my Jenkins server's log (jenkins.log) is always reported as 0KB and I can't see log content with less or tail, head command.

          The hudson server's one is xxxxxxxKB and I can grance it with less or tail command.

          akiko_pusu added a comment - I reviewed my another CentOS CI server which using not jenkins but "hudson". And the server is also CentOS and hudson was installed RPM. This server has been running without any problem around 1 years. FYI: /etc/logrotate.d/hudson is using "kill -SIGALRM `cat /var/run/hudson.pid`", not using "kill -s SIGALRM". And the size of my Jenkins server's log (jenkins.log) is always reported as 0KB and I can't see log content with less or tail, head command. The hudson server's one is xxxxxxxKB and I can grance it with less or tail command.

          akiko_pusu added a comment -

          Sorry for many times.

          I found that this problem on my environment was caused by wrong log rotate action.
          The rotation task is done as cron job and some cron jobs may be failed if the command is not using absolute path.

          In my case, "kill -s SIGALRM `cat /var/run/jenkins.pid" was failed and "/bin/kill -s SIGALRM `cat /var/run/jenkins.pid" works correctly.

          akiko_pusu added a comment - Sorry for many times. I found that this problem on my environment was caused by wrong log rotate action. The rotation task is done as cron job and some cron jobs may be failed if the command is not using absolute path. In my case, "kill -s SIGALRM `cat /var/run/jenkins.pid" was failed and "/bin/kill -s SIGALRM `cat /var/run/jenkins.pid" works correctly.

          evernat added a comment -

          Is this a duplicate of JENKINS-5784, which is already fixed?

          evernat added a comment - Is this a duplicate of JENKINS-5784 , which is already fixed?

          evernat added a comment -

          No response, so resolving as supposed duplicate of JENKINS-5784

          evernat added a comment - No response, so resolving as supposed duplicate of JENKINS-5784

            abayer Andrew Bayer
            randallschulz Randall Schulz
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: