• Icon: Improvement Improvement
    • Resolution: Fixed
    • Icon: Major Major
    • other
    • Platform: All, OS: All

      Old builds can take up a significant amount of space on disk, even though they
      often take up very little space when tarred and zipped. For example, I have a
      build setup where the archived build output takes up 164MB, but can be tarred
      and zipped down to 8MB. This problem is most prevalent when there are lots of
      small files being archived, as with JUnit and Javadoc output.

      In the mailing list, the argument against doing this is:

      https://hudson.dev.java.net/servlets/ReadMsg?list=users&msgNo=10279

      "In most of the situations, I don't think it makes much sense for Hudson
      to do such desperate disk conservation measure. It's much cheaper to fix
      the problem by throwing more HDDs at the problem, and on modern file
      systems, you can have file-system level compression, too."

      Throwing more HDDs is a good solution in many cases, but here it can often be a
      20X win to compress the output. That means a 20X difference in the number of
      HDDs required.

      The problem with file-system level compression is that it will typically also
      apply to the workspace folder, which can have a nontrivial impact on build
      performance.

          [JENKINS-2551] Option to compress build logs

          Code changed in jenkins
          User: Martin Schroeder
          Path:
          core/src/main/java/hudson/console/AnnotatedLargeText.java
          core/src/main/java/hudson/model/Run.java
          core/src/main/resources/hudson/model/Run/console.jelly
          http://jenkins-ci.org/commit/jenkins/346fc998b16f2021cbd6b3bb1cd1878a2b19ff5b
          Log:
          Enabled transparent compression support.

          This only works if the "transparent GZIP support" patch has been
          applied against Stapler. Otherwise, this patch will not compile
          as the new "LargeText" constructor will not be found.

          Additionally, the console.jelly was modified to make use of the
          stream instead of the raw file, which is necessary to get the
          correct uncompressed size of the file for skipping bytes.

          JENKINS-2551
          JENKINS-10400
          JENKINS-13655

          Signed-off-by: Martin Schroeder <martin.h.schroeder@intel.com>

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Martin Schroeder Path: core/src/main/java/hudson/console/AnnotatedLargeText.java core/src/main/java/hudson/model/Run.java core/src/main/resources/hudson/model/Run/console.jelly http://jenkins-ci.org/commit/jenkins/346fc998b16f2021cbd6b3bb1cd1878a2b19ff5b Log: Enabled transparent compression support. This only works if the "transparent GZIP support" patch has been applied against Stapler. Otherwise, this patch will not compile as the new "LargeText" constructor will not be found. Additionally, the console.jelly was modified to make use of the stream instead of the raw file, which is necessary to get the correct uncompressed size of the file for skipping bytes. JENKINS-2551 JENKINS-10400 JENKINS-13655 Signed-off-by: Martin Schroeder <martin.h.schroeder@intel.com>

          Alex Lorenz added a comment -

          We are sponsoring the fix of this issue with $100.

          Please see details here:

          http://www.freedomsponsors.org/core/offer/235/add-ability-to-zipcompress-old-builds

          Alex Lorenz added a comment - We are sponsoring the fix of this issue with $100. Please see details here: http://www.freedomsponsors.org/core/offer/235/add-ability-to-zipcompress-old-builds

          Jesse Glick added a comment -

          There are a bunch of interrelated issues with vague descriptions and excess scope so I am narrowing this to a specific issue: log file compression. There is already support for reading compressed logs, but apparently no UI option to write a compressed log: you have to compress historical logs manually.

          For compression of build artifacts, please use JENKINS-6229. For compression of other files such as test results, see JENKINS-3268 but file separate bugs since each plugin really has to deal with this independently: there is no chance that Jenkins will ZIP the entire build directory (this would be highly incompatible).

          Jesse Glick added a comment - There are a bunch of interrelated issues with vague descriptions and excess scope so I am narrowing this to a specific issue: log file compression. There is already support for reading compressed logs, but apparently no UI option to write a compressed log: you have to compress historical logs manually. For compression of build artifacts, please use JENKINS-6229 . For compression of other files such as test results, see JENKINS-3268 but file separate bugs since each plugin really has to deal with this independently: there is no chance that Jenkins will ZIP the entire build directory (this would be highly incompatible).

          Jesse Glick added a comment -

          Note that an alternate RFE (please file separately!) would be to offer a UI option to compress the entire directory of an old build, but this means Jenkins cannot load the build record directly: such compressed builds would have to be listed in some separate UI in the project, could not be used from Copy Artifact, etc. In order to work with such builds, you would have to ask to uncompress them, restoring them to live status. This kind of feature does not belong in core: a plugin can do it all. (To allow a specific build directory to be loaded even though it is not in the initial AbstractLazyLoadRunMap.idOnDisk, purgeCache would probably work, or calling put might suffice.)

          Jesse Glick added a comment - Note that an alternate RFE (please file separately!) would be to offer a UI option to compress the entire directory of an old build, but this means Jenkins cannot load the build record directly: such compressed builds would have to be listed in some separate UI in the project, could not be used from Copy Artifact, etc. In order to work with such builds, you would have to ask to uncompress them, restoring them to live status. This kind of feature does not belong in core: a plugin can do it all. (To allow a specific build directory to be loaded even though it is not in the initial AbstractLazyLoadRunMap.idOnDisk , purgeCache would probably work, or calling put might suffice.)

          Ronald Chen added a comment -

          So easy, so valuable.

          Ronald Chen added a comment - So easy, so valuable.

          Daniel Beck added a comment -

          I wrote a plugin that basically does what is requested here:

          https://github.com/daniel-beck/compress-buildlog-plugin

          It'll gzip the build log after the build completes if a checkbox is enabled on the job config page.

          It's very bare bones, but for some of you this might be sufficient.

          Daniel Beck added a comment - I wrote a plugin that basically does what is requested here: https://github.com/daniel-beck/compress-buildlog-plugin It'll gzip the build log after the build completes if a checkbox is enabled on the job config page. It's very bare bones, but for some of you this might be sufficient.

          Jesse Glick added a comment -

          Sounds like that would suffice. Nice work!

          Jesse Glick added a comment - Sounds like that would suffice. Nice work!

          Oleg Nenashev added a comment -

          The plugin has not been released yet

          Oleg Nenashev added a comment - The plugin has not been released yet

          Oleg Nenashev added a comment -

          Forked and released Daniel's plugin

          Oleg Nenashev added a comment - Forked and released Daniel's plugin

          Mr Cinquero added a comment - - edited

          The plugin disables the console tail functionality. And it does not compress any leftover logs. And you need to enable it by hand for each single job.

          Best solution so far (on Linux boxes): set up a daily script job:

          set -Eex
          cd "$JENKINS_HOME/jobs"
          find * -regex '^\([^/]+/builds/[0-9]+/log\|[^/]+/modules/[^/]+/builds/[0-9]+/log\)$' -cmin +600 -exec gzip -9v '{}' \;
          

          This compresses all uncompressed build logs which haven't been touched in the last 10 hours (to avoid processing active logs).

          Mr Cinquero added a comment - - edited The plugin disables the console tail functionality. And it does not compress any leftover logs. And you need to enable it by hand for each single job. Best solution so far (on Linux boxes): set up a daily script job: set -Eex cd "$JENKINS_HOME/jobs" find * -regex '^\([^/]+/builds/[0-9]+/log\|[^/]+/modules/[^/]+/builds/[0-9]+/log\)$' -cmin +600 -exec gzip -9v '{}' \; This compresses all uncompressed build logs which haven't been touched in the last 10 hours (to avoid processing active logs).

            Unassigned Unassigned
            plinehan plinehan
            Votes:
            21 Vote for this issue
            Watchers:
            19 Start watching this issue

              Created:
              Updated:
              Resolved: