Code changed in hudson
User: : huybrechts
Path:
trunk/hudson/main/core/src/main/java/hudson/model/Run.java
trunk/hudson/main/core/src/main/resources/hudson/model/Run/console.jelly
trunk/hudson/main/core/src/main/resources/hudson/model/Run/consoleText.jelly
http://fisheye4.cenqua.com/changelog/hudson/?cs=15363
Log:
JENKINS-2551 support for reading compressed build logs

Hudson will read log.gz files if available, although it doesn't produce them.
So you can write some script to compress old build logs...

I'm putting the artifact on fixed. Feel free to reopen if you desire more
functionality...

This is great news! Thanks!

I'm reopening for the following functionality:

It would still be really useful if Hudson supported "archive" folders in .zip or
.tar.gz format in addition to logs in .gz format. I don't mind zipping them up myself,
as long as Hudson can read them.

As an example from my build setup:

====================
$ [cd into particular job directory]
$ du -hs .
2.2M
$ gzip log
$ du -hs .
1.5M
$ tar --remove-files -czvf archive.tar.gz archive
576K
====================

As you can see, this would give me a ~3X savings in disk space beyond what compressing
the log file can do.

I appreciate the good work y'all are doing. Keep it up!

• • • Issue 3268 has been marked as a duplicate of this issue. ***

Wow cool, I didn't know about this feature, thanks! It would be neat to generalize it to a function that you call to get a file in a build directory, and it will do the same logic there, returning either a stream of the file or .gz if it exists. This way it would be trivial to support this anywhere, just change the function call.

The low hanging fruit for us would be extending this to junitResult.xml, which results in a ~93% compression in my case and would save us many gigabytes! Any thoughts?

Then someone could just write a post-build plugin that you can enable at the system or project level to compress any supported files after a build, so anyone desiring this can easily enable it.

I do not believe this functionality is working. I hope I am wrong.

Env Details:
Hudson: 1.380
Java Version: jre1.6.0_22
OS: RHEL 5.3
gzip version: gzip 1.3.5 (2002-09-30)

For example, I have gone into a build directory:

/var/lib/hudson/jobs/emb_UEQ_2.1_hudson/builds/2010-10-20_20-46-39/

And issued the command
#gzip log

#ls -l log.gz
-rwxrwxrwx 1 hudson hudson 62966 Nov 1 18:38 log.gz

When I go to the build page for this corresponding build I don't see any log files available. When searching /var/log/hudson/hudson.log I see the following error:

WARNING: Caught exception evaluating: it.writeLogTo(offset,output). Reason: java.io.FileNotFoundException: /var/lib/hudson/jobs/emb_UEQ_2.1_hudson/builds/2010-10-20_20-46-39/log (No such file or directory)
java.io.FileNotFoundException: /var/lib/hudson/jobs/emb_UEQ_2.1_hudson/builds/2010-10-20_20-46-39/log (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
at org.kohsuke.stapler.framework.io.LargeText$FileSession.<init>(LargeText.java:322)
at org.kohsuke.stapler.framework.io.LargeText$1.open(LargeText.java:52)
at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:137)
at hudson.console.AnnotatedLargeText.writeHtmlTo(AnnotatedLargeText.java:154)
at hudson.model.Run.writeLogTo(Run.java:1038)
at sun.reflect.GeneratedMethodAccessor619.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258)
at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:72)
at org.apache.commons.jelly.impl.ExpressionScript.run(ExpressionScript.java:66)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
at org.apache.commons.jelly.tags.core.WhitespaceTag.doTag(WhitespaceTag.java:48)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:76)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:75)
at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:96)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:75)
at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:76)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:76)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:76)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:76)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:76)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:96)
at org.kohsuke.stapler.jelly.CompressTag.doTag(CompressTag.java:21)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:40)
at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:40)
at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:30)
at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:67)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:537)
at org.kohsuke.stapler.MetaClass$13.dispatch(MetaClass.java:359)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:537)
at org.kohsuke.stapler.MetaClass$7.doDispatch(MetaClass.java:219)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:30)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:537)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:453)
at org.kohsuke.stapler.Stapler.service(Stapler.java:135)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
at winstone.ServletConfiguration.execute(ServletConfiguration.java:249)
at winstone.RequestDispatcher.forward(RequestDispatcher.java:335)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:378)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:94)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:86)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:244)
at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)
at java.lang.Thread.run(Thread.java:636)

i believe you are right, tried it as well and found this:

http://hudson.361315.n4.nabble.com/Request-store-console-output-in-a-compressed-format-td381416.html

"Hi all,

In case others look back to this thread, compressed log format no
longer works in recent hudson versions (not sure exactly when the
support was removed between 349->373 , but the culprit is this evol:
http://issues.jenkins-ci.org/browse/JENKINS-2137).

It'd be good to see it back, considering it's pretty easy to add.
I hacked it on my platform, adding support for gzipped log to
LargeText.writeLogTo() is very easy, especially if you only consider
the "completed" case (if the log file is zipped, it can only be
complete).

Cheers.
Francois VALDY."

We are sponsoring the fix of this issue with $100.

Please see details here:

http://www.freedomsponsors.org/core/offer/235/add-ability-to-zipcompress-old-builds

There are a bunch of interrelated issues with vague descriptions and excess scope so I am narrowing this to a specific issue: log file compression. There is already support for reading compressed logs, but apparently no UI option to write a compressed log: you have to compress historical logs manually.

For compression of build artifacts, please use JENKINS-6229. For compression of other files such as test results, see JENKINS-3268 but file separate bugs since each plugin really has to deal with this independently: there is no chance that Jenkins will ZIP the entire build directory (this would be highly incompatible).

Note that an alternate RFE (please file separately!) would be to offer a UI option to compress the entire directory of an old build, but this means Jenkins cannot load the build record directly: such compressed builds would have to be listed in some separate UI in the project, could not be used from Copy Artifact, etc. In order to work with such builds, you would have to ask to uncompress them, restoring them to live status. This kind of feature does not belong in core: a plugin can do it all. (To allow a specific build directory to be loaded even though it is not in the initial AbstractLazyLoadRunMap.idOnDisk, purgeCache would probably work, or calling put might suffice.)

So easy, so valuable.

I wrote a plugin that basically does what is requested here:

https://github.com/daniel-beck/compress-buildlog-plugin

It'll gzip the build log after the build completes if a checkbox is enabled on the job config page.

It's very bare bones, but for some of you this might be sufficient.

Sounds like that would suffice. Nice work!

The plugin has not been released yet

Forked and released Daniel's plugin

The plugin disables the console tail functionality. And it does not compress any leftover logs. And you need to enable it by hand for each single job.

Best solution so far (on Linux boxes): set up a daily script job:

set -Eex
cd "$JENKINS_HOME/jobs"
find * -regex '^\([^/]+/builds/[0-9]+/log\|[^/]+/modules/[^/]+/builds/[0-9]+/log\)$' -cmin +600 -exec gzip -9v '{}' \;

This compresses all uncompressed build logs which haven't been touched in the last 10 hours (to avoid processing active logs).