Jenkins has ability to diplay gunzip gzipped logs on the fly (https://issues.jenkins-ci.org/browse/JENKINS-2551) though doesn't natively compress the logs. The BFA doesn't expect that the logs can be compressed and appends a plaintext to the end log:

      1. diff <(hexdump -C /var/lib/jenkins/jobs/sync-images-rhel-7v0/builds/25/log.gz.previous) <(hexdump -C /var/lib/jenkins/jobs/sync-images-rhel-7v0/builds/25/log.gz)
        2800,2801c2800,2805
        < 0000aef0 07 81 6b fd bf f2 2a 01 00 |..k...*..|
        < 0000aef9
        —
        > 0000aef0 07 81 6b fd bf f2 2a 01 00 5b 42 46 41 5d 20 53 |..k...*..[BFA] S|
        > 0000af00 63 61 6e 6e 69 6e 67 20 62 75 69 6c 64 20 66 6f |canning build fo|
        > 0000af10 72 20 6b 6e 6f 77 6e 20 63 61 75 73 65 73 2e 2e |r known causes..|
        > 0000af20 2e 0a 2e 2e 0a 5b 42 46 41 5d 20 44 6f 6e 65 2e |.....[BFA] Done.|
        > 0000af30 20 32 73 0a | 2s.|
        > 0000af34

      This then breaks the displaying of the logs in the sense that every GET results in one core continuously loaded in endless loop.

      "Handling GET /job/sync-images-rhel-7v0/24/console : RequestHandlerThread10" daemon prio=10 tid=0x00007f55c400b000 nid=0x3b4e runnable [0x00007f56077f8000]
      java.lang.Thread.State: RUNNABLE
      at org.kohsuke.stapler.framework.io.LargeText$GzipAwareSession.skip(LargeText.java:437)
      at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:211)
      at hudson.console.AnnotatedLargeText.writeHtmlTo(AnnotatedLargeText.java:156)
      at hudson.model.Run.writeLogTo(Run.java:1351)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258)
      at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
      at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
      at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
      at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
      at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
      at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
      at org.apache.commons.jelly.parser.EscapingExpression.evaluate(EscapingExpression.java:24)
      at org.apache.commons.jelly.impl.ExpressionScript.run(ExpressionScript.java:66)
      at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
      at org.apache.commons.jelly.tags.core.WhitespaceTag.doTag(WhitespaceTag.java:48)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
      at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
      at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
      at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
      at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
      at org.kohsuke.stapler.jelly.CompressTag.doTag(CompressTag.java:44)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
      at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
      at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
      at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
      at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:390)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
      at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
      at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:631)
      at org.kohsuke.stapler.Stapler.service(Stapler.java:225)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
      at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
      at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:79)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
      at org.eclipse.jetty.server.Server.handle(Server.java:370)
      at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
      at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
      at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
      at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
      at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
      at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
      at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
      at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
      at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)

          [JENKINS-24171] BFA is blindly adding text to gzipped log

          Jaroslav Henner created issue -
          Jaroslav Henner made changes -
          Component/s New: logging [ 16621 ]
          Jaroslav Henner made changes -
          Component/s New: core [ 15593 ]
          Component/s Original: logging [ 16621 ]
          Jaroslav Henner made changes -
          Description Original: Jenkins has ability to diplay gunzip gzipped logs on the fly (https://issues.jenkins-ci.org/browse/JENKINS-2551) but it doesn't natively compress the logs. The BFA doesn't expect that and appends the plaintext to the log:

          # diff <(hexdump -C /var/lib/jenkins/jobs/sync-images-rhel-7v0/builds/25/log.gz.previous) <(hexdump -C /var/lib/jenkins/jobs/sync-images-rhel-7v0/builds/25/log.gz)
          2800,2801c2800,2805
          < 0000aef0 07 81 6b fd bf f2 2a 01 00 |..k...*..|
          < 0000aef9
          ---
          > 0000aef0 07 81 6b fd bf f2 2a 01 00 5b 42 46 41 5d 20 53 |..k...*..[BFA] S|
          > 0000af00 63 61 6e 6e 69 6e 67 20 62 75 69 6c 64 20 66 6f |canning build fo|
          > 0000af10 72 20 6b 6e 6f 77 6e 20 63 61 75 73 65 73 2e 2e |r known causes..|
          > 0000af20 2e 0a 2e 2e 0a 5b 42 46 41 5d 20 44 6f 6e 65 2e |.....[BFA] Done.|
          > 0000af30 20 32 73 0a | 2s.|
          > 0000af34


          This then breaks the displaying of the logs in the sense that every GET results in one core continuously loaded in endless loop.

              "Handling GET /job/sync-images-rhel-7v0/24/console : RequestHandlerThread[#10]" daemon prio=10 tid=0x00007f55c400b000 nid=0x3b4e runnable [0x00007f56077f8000]
                 java.lang.Thread.State: RUNNABLE
                      at org.kohsuke.stapler.framework.io.LargeText$GzipAwareSession.skip(LargeText.java:437)
                      at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:211)
                      at hudson.console.AnnotatedLargeText.writeHtmlTo(AnnotatedLargeText.java:156)
                      at hudson.model.Run.writeLogTo(Run.java:1351)
                      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                      at java.lang.reflect.Method.invoke(Method.java:606)
                      at org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258)
                      at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
                      at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
                      at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
                      at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
                      at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
                      at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
                      at org.apache.commons.jelly.parser.EscapingExpression.evaluate(EscapingExpression.java:24)
                      at org.apache.commons.jelly.impl.ExpressionScript.run(ExpressionScript.java:66)
                      at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
                      at org.apache.commons.jelly.tags.core.WhitespaceTag.doTag(WhitespaceTag.java:48)
                      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
                      at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
                      at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41)
                      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
                      at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
                      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
                      at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
                      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
                      at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
                      at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
                      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
                      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
                      at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
                      at org.kohsuke.stapler.jelly.CompressTag.doTag(CompressTag.java:44)
                      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                      at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
                      at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
                      at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
                      at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95)
                      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
                      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
                      at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:390)
                      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
                      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
                      at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
                      at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
                      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
                      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
                      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:631)
                      at org.kohsuke.stapler.Stapler.service(Stapler.java:225)
                      at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
                      at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
                      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
                      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
                      at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
                      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
                      at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
                      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
                      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
                      at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
                      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                      at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
                      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                      at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
                      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                      at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
                      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                      at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
                      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                      at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174)
                      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                      at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:79)
                      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                      at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
                      at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
                      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                      at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
                      at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
                      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
                      at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
                      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
                      at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
                      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
                      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
                      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
                      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
                      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
                      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
                      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
                      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
                      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
                      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
                      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
                      at org.eclipse.jetty.server.Server.handle(Server.java:370)
                      at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
                      at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
                      at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
                      at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
                      at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
                      at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
                      at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
                      at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
                      at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                      at java.lang.Thread.run(Thread.java:745)
          New: Jenkins has ability to diplay gunzip gzipped logs on the fly (https://issues.jenkins-ci.org/browse/JENKINS-2551) though doesn't natively compress the logs. The BFA doesn't expect that the logs can be compressed and appends a plaintext to the end log:

          # diff <(hexdump -C /var/lib/jenkins/jobs/sync-images-rhel-7v0/builds/25/log.gz.previous) <(hexdump -C /var/lib/jenkins/jobs/sync-images-rhel-7v0/builds/25/log.gz)
          2800,2801c2800,2805
          < 0000aef0 07 81 6b fd bf f2 2a 01 00 |..k...*..|
          < 0000aef9
          ---
          > 0000aef0 07 81 6b fd bf f2 2a 01 00 5b 42 46 41 5d 20 53 |..k...*..[BFA] S|
          > 0000af00 63 61 6e 6e 69 6e 67 20 62 75 69 6c 64 20 66 6f |canning build fo|
          > 0000af10 72 20 6b 6e 6f 77 6e 20 63 61 75 73 65 73 2e 2e |r known causes..|
          > 0000af20 2e 0a 2e 2e 0a 5b 42 46 41 5d 20 44 6f 6e 65 2e |.....[BFA] Done.|
          > 0000af30 20 32 73 0a | 2s.|
          > 0000af34


          This then breaks the displaying of the logs in the sense that every GET results in one core continuously loaded in endless loop.

              "Handling GET /job/sync-images-rhel-7v0/24/console : RequestHandlerThread[#10]" daemon prio=10 tid=0x00007f55c400b000 nid=0x3b4e runnable [0x00007f56077f8000]
                 java.lang.Thread.State: RUNNABLE
                      at org.kohsuke.stapler.framework.io.LargeText$GzipAwareSession.skip(LargeText.java:437)
                      at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:211)
                      at hudson.console.AnnotatedLargeText.writeHtmlTo(AnnotatedLargeText.java:156)
                      at hudson.model.Run.writeLogTo(Run.java:1351)
                      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                      at java.lang.reflect.Method.invoke(Method.java:606)
                      at org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258)
                      at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
                      at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
                      at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
                      at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
                      at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
                      at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
                      at org.apache.commons.jelly.parser.EscapingExpression.evaluate(EscapingExpression.java:24)
                      at org.apache.commons.jelly.impl.ExpressionScript.run(ExpressionScript.java:66)
                      at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
                      at org.apache.commons.jelly.tags.core.WhitespaceTag.doTag(WhitespaceTag.java:48)
                      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
                      at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
                      at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41)
                      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
                      at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
                      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
                      at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
                      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
                      at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
                      at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
                      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
                      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
                      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                      at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
                      at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
                      at org.kohsuke.stapler.jelly.CompressTag.doTag(CompressTag.java:44)
                      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                      at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
                      at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
                      at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
                      at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95)
                      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
                      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
                      at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:390)
                      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
                      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
                      at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
                      at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
                      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
                      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
                      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:631)
                      at org.kohsuke.stapler.Stapler.service(Stapler.java:225)
                      at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
                      at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
                      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
                      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
                      at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
                      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
                      at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
                      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
                      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
                      at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
                      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                      at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
                      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                      at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
                      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                      at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
                      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                      at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
                      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                      at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174)
                      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                      at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:79)
                      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                      at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
                      at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
                      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                      at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
                      at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
                      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
                      at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
                      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
                      at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
                      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
                      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
                      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
                      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
                      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
                      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
                      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
                      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
                      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
                      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
                      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
                      at org.eclipse.jetty.server.Server.handle(Server.java:370)
                      at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
                      at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
                      at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
                      at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
                      at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
                      at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
                      at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
                      at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
                      at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                      at java.lang.Thread.run(Thread.java:745)

          Daniel Beck added a comment -

          Could you explain how you ended up with a gzipped build log? Or in general, how this issue can be reproduced?


          From a superficial BFA source code reading, it seems to do everything right.

          While the log file could be gzipped, it's safe to assume TaskListener.getLogger() (a PrintStream) is handling this transparently.

          Daniel Beck added a comment - Could you explain how you ended up with a gzipped build log? Or in general, how this issue can be reproduced? From a superficial BFA source code reading, it seems to do everything right. While the log file could be gzipped, it's safe to assume TaskListener.getLogger() (a PrintStream) is handling this transparently.

          Jaroslav Henner added a comment - - edited

          We are just using gzip on the log files. As written in the https://issues.jenkins-ci.org/browse/JENKINS-2551. Something like
          # gzip /var/lib/jenkins/jobs/sync-images-rhel-7v0/builds/25/log

          Jaroslav Henner added a comment - - edited We are just using gzip on the log files. As written in the https://issues.jenkins-ci.org/browse/JENKINS-2551 . Something like # gzip /var/lib/jenkins/jobs/sync-images-rhel-7v0/builds/25/log

          Daniel Beck added a comment -

          Assuming to be an issue with BFA opening log file non-gzipped until proven otherwise.

          Daniel Beck added a comment - Assuming to be an issue with BFA opening log file non-gzipped until proven otherwise.
          Daniel Beck made changes -
          Component/s Original: core [ 15593 ]

          Daniel, I don't really know how this "feature" it is implemented, but I smell it is problem on both sides

          1. BFA should not append plaintext to compressed stream
          2. Jenkins should not hang when displaying gzipped log with trailing garbage at the end.

          Jaroslav Henner added a comment - Daniel, I don't really know how this "feature" it is implemented, but I smell it is problem on both sides BFA should not append plaintext to compressed stream Jenkins should not hang when displaying gzipped log with trailing garbage at the end.

          Daniel Beck added a comment - - edited

          Makes sense. Probably needs to throw on 0 returned from gz.skip(int) unless the parameter is also 0.

          However this seems to be a separate issue (robustness in the face of invalid input) and should be filed separately, as the solutions are independent (i.e. fixing BFA does not seem to depend on core fixes or something like that).

          Daniel Beck added a comment - - edited Makes sense. Probably needs to throw on 0 returned from gz.skip(int) unless the parameter is also 0. However this seems to be a separate issue (robustness in the face of invalid input) and should be filed separately, as the solutions are independent (i.e. fixing BFA does not seem to depend on core fixes or something like that).

            t_westling Tomas Westling
            jhenner Jaroslav Henner
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: