Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47192

RFC: Compress pipeline stage logs?

XMLWordPrintable

      We are extensively using multibranch pipelines to build our system and we're struggling to manage the sheer quantity of build logs generated.

      On inspection we're seeing roughly 250M of logs (uncompressed) for every build. The compress-build-log-plugin would seem to be the answer and indeed having added this to the properties of the pipeline the log file is compressed to about 2.5M. But, the individual stage logs are not compressed and so are consuming roughly 125M as the logs are entirely replicated so that the pipeline UI can show logs for each step.

      Should the compress-build-log-plugin be extended to also find all the *.log files and compress them as well? Not sure what the pipeline UI would think to this though.

          [JENKINS-47192] RFC: Compress pipeline stage logs?

          Daniel Beck added a comment -

          Pipeline doesn't support transparently decompressing compressed logs in the fly. That needs to be added first, otherwise there's no point in gzipping the logs.

          Seems to be around LogActionImpl in workflow-support.

          Daniel Beck added a comment - Pipeline doesn't support transparently decompressing compressed logs in the fly. That needs to be added first, otherwise there's no point in gzipping the logs. Seems to be around LogActionImpl in workflow-support.

          Mark Syms added a comment -

          I thought that might be the case. Worth doing though?

          Mark Syms added a comment - I thought that might be the case. Worth doing though?

          Fajran Rusadi added a comment -

          Hi, I would also like to have this feature implemented. It would save me a ton of storage space if I could compress the log files and still able to read them.

          I did try to change the LogActionImpl class and it worked on my local test – https://github.com/fajran/workflow-support-plugin/tree/read-compressed-log This one was built on top of workflow-support-2.16.

          Do you think it is a good idea to submit a pull request? Looks like there are some works being done related to the logging.

          Fajran Rusadi added a comment - Hi, I would also like to have this feature implemented. It would save me a ton of storage space if I could compress the log files and still able to read them. I did try to change the LogActionImpl class and it worked on my local test – https://github.com/fajran/workflow-support-plugin/tree/read-compressed-log This one was built on top of workflow-support-2.16. Do you think it is a good idea to submit a pull request? Looks like there are some works being done related to the logging.

          Jesse Glick added a comment -

          See JENKINS-38381.

          Jesse Glick added a comment - See JENKINS-38381 .

            Unassigned Unassigned
            marksyms Mark Syms
            Votes:
            2 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: