Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-2551

Option to compress build logs

    XMLWordPrintable

Details

    • Improvement
    • Status: Resolved (View Workflow)
    • Major
    • Resolution: Fixed
    • other
    • Platform: All, OS: All

    Description

      Old builds can take up a significant amount of space on disk, even though they
      often take up very little space when tarred and zipped. For example, I have a
      build setup where the archived build output takes up 164MB, but can be tarred
      and zipped down to 8MB. This problem is most prevalent when there are lots of
      small files being archived, as with JUnit and Javadoc output.

      In the mailing list, the argument against doing this is:

      https://hudson.dev.java.net/servlets/ReadMsg?list=users&msgNo=10279

      "In most of the situations, I don't think it makes much sense for Hudson
      to do such desperate disk conservation measure. It's much cheaper to fix
      the problem by throwing more HDDs at the problem, and on modern file
      systems, you can have file-system level compression, too."

      Throwing more HDDs is a good solution in many cases, but here it can often be a
      20X win to compress the output. That means a 20X difference in the number of
      HDDs required.

      The problem with file-system level compression is that it will typically also
      apply to the workspace folder, which can have a nontrivial impact on build
      performance.

      Attachments

        Issue Links

          Activity

            danielbeck Daniel Beck added a comment -

            I wrote a plugin that basically does what is requested here:

            https://github.com/daniel-beck/compress-buildlog-plugin

            It'll gzip the build log after the build completes if a checkbox is enabled on the job config page.

            It's very bare bones, but for some of you this might be sufficient.

            danielbeck Daniel Beck added a comment - I wrote a plugin that basically does what is requested here: https://github.com/daniel-beck/compress-buildlog-plugin It'll gzip the build log after the build completes if a checkbox is enabled on the job config page. It's very bare bones, but for some of you this might be sufficient.
            jglick Jesse Glick added a comment -

            Sounds like that would suffice. Nice work!

            jglick Jesse Glick added a comment - Sounds like that would suffice. Nice work!
            oleg_nenashev Oleg Nenashev added a comment -

            The plugin has not been released yet

            oleg_nenashev Oleg Nenashev added a comment - The plugin has not been released yet
            oleg_nenashev Oleg Nenashev added a comment -

            Forked and released Daniel's plugin

            oleg_nenashev Oleg Nenashev added a comment - Forked and released Daniel's plugin
            marc321 Mr Cinquero added a comment - - edited

            The plugin disables the console tail functionality. And it does not compress any leftover logs. And you need to enable it by hand for each single job.

            Best solution so far (on Linux boxes): set up a daily script job:

            set -Eex
            cd "$JENKINS_HOME/jobs"
            find * -regex '^\([^/]+/builds/[0-9]+/log\|[^/]+/modules/[^/]+/builds/[0-9]+/log\)$' -cmin +600 -exec gzip -9v '{}' \;
            

            This compresses all uncompressed build logs which haven't been touched in the last 10 hours (to avoid processing active logs).

            marc321 Mr Cinquero added a comment - - edited The plugin disables the console tail functionality. And it does not compress any leftover logs. And you need to enable it by hand for each single job. Best solution so far (on Linux boxes): set up a daily script job: set -Eex cd "$JENKINS_HOME/jobs" find * -regex '^\([^/]+/builds/[0-9]+/log\|[^/]+/modules/[^/]+/builds/[0-9]+/log\)$' -cmin +600 -exec gzip -9v '{}' \; This compresses all uncompressed build logs which haven't been touched in the last 10 hours (to avoid processing active logs).

            People

              Unassigned Unassigned
              plinehan plinehan
              Votes:
              21 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: