Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56446

IOException in FileLogStorage maybeFlush

    XMLWordPrintable

Details

    Description

      Some jobs get stuck in a state where they're unable to continue to write data to the console log.  The system log contains multiple entries of the following form until the job ends:

       

      failed to flush /net/users/jenkins/configuration/jobs/XXX/builds/5/log
      java.io.IOException: Stream Closed
      at java.io.FileOutputStream.writeBytes(Native Method)
      at java.io.FileOutputStream.write(FileOutputStream.java:326)
      at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream$FlushControlledOutputStream.write(DelayBufferedOutputStream.java:134)
      at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
      at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
      at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
      at org.jenkinsci.plugins.workflow.log.FileLogStorage.maybeFlush(FileLogStorage.java:190)
      at org.jenkinsci.plugins.workflow.log.FileLogStorage.overallLog(FileLogStorage.java:198)
      at org.jenkinsci.plugins.workflow.job.WorkflowRun.getLogText(WorkflowRun.java:1018)
      at org.jenkinsci.plugins.workflow.job.WorkflowRun.getLogInputStream(WorkflowRun.java:1026)
      at org.jenkinsci.plugins.gwt.GenericWebhookEnvironmentContributor.notLogged(GenericWebhookEnvironmentContributor.java:54)
      at org.jenkinsci.plugins.gwt.GenericWebhookEnvironmentContributor.buildEnvironmentFor(GenericWebhookEnvironmentContributor.java:30)
      at hudson.model.Run.getEnvironment(Run.java:2373)
      at org.jenkinsci.plugins.workflow.job.WorkflowRun.getEnvironment(WorkflowRun.java:468)
      at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getEnvironment(EnvActionImpl.java:86)
      at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getEnvironment(EnvActionImpl.java:67)
      at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:72)
      at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:305)
      at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:268)
      at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:176)
      at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
      at sun.reflect.GeneratedMethodAccessor387.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
      at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
      at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
      at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
      at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
      at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
      at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
      at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157)
      at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
      at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:155)
      at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
      at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159)
      at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
      at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
      at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
      at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
      at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      at sun.reflect.GeneratedMethodAccessor384.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:103)
      at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      at sun.reflect.GeneratedMethodAccessor384.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:60)
      at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
      at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      at sun.reflect.GeneratedMethodAccessor384.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
      at com.cloudbees.groovy.cps.Next.step(Next.java:83)
      at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
      at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
      at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
      at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
      at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
      at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
      at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
      at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:121)
      at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
      at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:182)
      at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
      at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
      at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
      at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
      at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
      at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)

       

      When enabling DelayBufferedOutputStream logging, we also see this in the log:

      null
      java.io.IOException: Stream Closed
      at java.io.FileOutputStream.writeBytes(Native Method)
      at java.io.FileOutputStream.write(FileOutputStream.java:326)
      at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream$FlushControlledOutputStream.write(DelayBufferedOutputStream.java:134)
      at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
      at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
      at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.flushBuffer(DelayBufferedOutputStream.java:82)
      at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.flushAndReschedule(DelayBufferedOutputStream.java:91)
      at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream$Flush.run(DelayBufferedOutputStream.java:114)
      at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)

      Attachments

        Issue Links

          Activity

            mrdevops mel nal added a comment -

            Hi All,

            Any update on this? rshade Have you tried any work-around? 

            mrdevops mel nal added a comment - Hi All, Any update on this? rshade  Have you tried any work-around? 
            materigoprecitec Mate Rigo added a comment - - edited

            I have had the same issue described here.

            Now after digging around I have found, that the root cause was indeed this issue here: JENKINS-62999

            I am using the pipeline tee command in many of our parallel build steps, and it does have a heavy output load as described in JENKINS-62999

             

            My log does look like this:

            2022-01-31 13:19:38.241+0000 [id=210]   WARNING o.j.r.u.AnonymousClassWarnings#warn: Attempt to (de-)serialize anonymous class org.jenkinsci.plugins.pipeline.utility.steps.fs.TeeStep$1 in file:/home/mrigo/.jenkins/plugins/pipeline-utility-steps/WEB-INF/lib/pipeline-utility-steps.jar; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
            2022-01-31 13:20:00.194+0000 [id=84]    WARNING h.i.i.InstallUncaughtExceptionHandler#handleException: Caught unhandled exception with ID d7135add-6c31-4f76-8331-e8d609127f0b
            2022-01-31 13:20:14.451+0000 [id=259]   INFO    o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 5 sec
            2022-01-31 13:20:19.451+0000 [id=259]   INFO    o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 10 sec
            2022-01-31 13:20:24.452+0000 [id=259]   INFO    o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 15 sec
            2022-01-31 13:20:29.452+0000 [id=259]   INFO    o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 20 sec
            2022-01-31 13:20:34.453+0000 [id=259]   INFO    o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 25 sec
            2022-01-31 13:20:39.453+0000 [id=259]   INFO    o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 30 sec
            2022-01-31 13:20:44.453+0000 [id=259]   INFO    o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 35 sec
            2022-01-31 13:20:49.454+0000 [id=259]   INFO    o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 40 sec
            2022-01-31 13:20:54.454+0000 [id=259]   INFO    o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 45 sec
            2022-01-31 13:20:59.454+0000 [id=259]   INFO    o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 50 sec
            2022-01-31 13:21:04.455+0000 [id=259]   INFO    o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 55 sec
            2022-01-31 13:21:08.882+0000 [id=22]    WARNING o.j.p.w.log.FileLogStorage#maybeFlush: failed to flush /home/mrigo/.jenkins/jobs/project-Build-pipe/builds/1526/log
            java.io.IOException: Stream Closed
                    at java.io.FileOutputStream.writeBytes(Native Method)
                    at java.io.FileOutputStream.write(FileOutputStream.java:326)
                    at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream$FlushControlledOutputStream.write(DelayBufferedOutputStream.java:127)
                    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
                    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
                    at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
                    at org.jenkinsci.plugins.workflow.log.FileLogStorage.maybeFlush(FileLogStorage.java:193)
                    at org.jenkinsci.plugins.workflow.log.FileLogStorage.overallLog(FileLogStorage.java:202)
                    at org.jenkinsci.plugins.workflow.job.WorkflowRun.getLogText(WorkflowRun.java:1065)
                    at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
                    at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:393)
                    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:405)
                    at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:220)
                    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
                    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                    at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:457)
                    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                    at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:281)
                    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
                    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:694)
                    at org.kohsuke.stapler.Stapler.service(Stapler.java:240)
                    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
                    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
                    at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
                    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:156)
                    at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)
                    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:153)
                    at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:80)
                    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:153)
                    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
                    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
                    at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
                    at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:121)
                    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:153)
                    at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
                    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:153)
                    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:159)
                    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
                    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
                    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:153)
                    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
                    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
                    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:92)
                    at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
                    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
                    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:53)
                    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
                    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:121)
                    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
                    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
                    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105)
                    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
                    at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
                    at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:92)
                    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
                    at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:218)
                    at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:212)
                    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
                    at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
                    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
                    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
                    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
                    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:62)
                    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
                    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:109)
                    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
                    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
                    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
                    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:51)
                    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
                    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
                    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:85)
                    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
                    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
                    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
                    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
                    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
                    at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:39)
                    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
                    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
                    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
                    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
                    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
                    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
                    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
                    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
                    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
                    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
                    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
                    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
                    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
                    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
                    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
                    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
                    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
                    at org.eclipse.jetty.server.Server.handle(Server.java:516)
                    at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
                    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
                    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
                    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
                    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
                    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
                    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
                    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
                    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
                    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
                    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
                    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
                    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
                    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
                    at java.lang.Thread.run(Thread.java:748)
            

            The reason I was using the pipeline tee was not to mask my make's error output.

            tee ('build.log'){
                sh 'make -j $(nproc --all)'
            }
            

            After modifying to this the make's return value still gets passed to the pipe, hence the pipe fails if make fails:

            sh """#!/bin/bash
               set -o pipefail
               make -j \$(nproc --all) 2>&1 | tee build.log
               """
            

            Maybe some of you out there also do use tee in pipelines.

            Only thing I am wondering is how to do now tee in the windows builds that we do have, as the pipeline tee was actually quiet handy.

             

            materigoprecitec Mate Rigo added a comment - - edited I have had the same issue described here. Now after digging around I have found, that the root cause was indeed this issue here:  JENKINS-62999 I am using the pipeline tee command in many of our parallel build steps, and it does have a heavy output load as described in  JENKINS-62999   My log does look like this: 2022-01-31 13:19:38.241+0000 [id=210] WARNING o.j.r.u.AnonymousClassWarnings#warn: Attempt to (de-)serialize anonymous class org.jenkinsci.plugins.pipeline.utility.steps.fs.TeeStep$1 in file:/home/mrigo/.jenkins/plugins/pipeline-utility-steps/WEB-INF/lib/pipeline-utility-steps.jar; see: https: //jenkins.io/redirect/serialization-of-anonymous-classes/ 2022-01-31 13:20:00.194+0000 [id=84] WARNING h.i.i.InstallUncaughtExceptionHandler#handleException: Caught unhandled exception with ID d7135add-6c31-4f76-8331-e8d609127f0b 2022-01-31 13:20:14.451+0000 [id=259] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 5 sec 2022-01-31 13:20:19.451+0000 [id=259] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 10 sec 2022-01-31 13:20:24.452+0000 [id=259] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 15 sec 2022-01-31 13:20:29.452+0000 [id=259] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 20 sec 2022-01-31 13:20:34.453+0000 [id=259] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 25 sec 2022-01-31 13:20:39.453+0000 [id=259] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 30 sec 2022-01-31 13:20:44.453+0000 [id=259] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 35 sec 2022-01-31 13:20:49.454+0000 [id=259] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 40 sec 2022-01-31 13:20:54.454+0000 [id=259] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 45 sec 2022-01-31 13:20:59.454+0000 [id=259] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 50 sec 2022-01-31 13:21:04.455+0000 [id=259] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#22]: checking /home/jenkins/jenkins/workspace/project-Build-pipe_3/project on linux unresponsive for 55 sec 2022-01-31 13:21:08.882+0000 [id=22] WARNING o.j.p.w.log.FileLogStorage#maybeFlush: failed to flush /home/mrigo/.jenkins/jobs/project-Build-pipe/builds/1526/log java.io.IOException: Stream Closed at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:326) at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream$FlushControlledOutputStream.write(DelayBufferedOutputStream.java:127) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at java.io.FilterOutputStream.flush(FilterOutputStream.java:140) at org.jenkinsci.plugins.workflow.log.FileLogStorage.maybeFlush(FileLogStorage.java:193) at org.jenkinsci.plugins.workflow.log.FileLogStorage.overallLog(FileLogStorage.java:202) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getLogText(WorkflowRun.java:1065) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:393) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:405) at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:220) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898) at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:457) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:281) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:694) at org.kohsuke.stapler.Stapler.service(Stapler.java:240) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:156) at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:153) at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:80) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:153) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:121) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:153) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:153) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:159) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:153) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:92) at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:53) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:121) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:92) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:218) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:212) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:62) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:109) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:51) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:85) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:39) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.lang. Thread .run( Thread .java:748) The reason I was using the pipeline tee was not to mask my make's error output. tee ( 'build.log' ){ sh 'make -j $(nproc --all)' } After modifying to this the make's return value still gets passed to the pipe, hence the pipe fails if make fails: sh """#!/bin/bash set -o pipefail make -j \$(nproc --all) 2>&1 | tee build.log """ Maybe some of you out there also do use tee in pipelines. Only thing I am wondering is how to do now tee in the windows builds that we do have, as the pipeline tee was actually quiet handy.  

            I don’t use the tee step anywhere, so it’s not exclusive to that.

            haw777 Christopher Head added a comment - I don’t use the tee step anywhere, so it’s not exclusive to that.
            vishakumar Vishal kumar added a comment -

            Any updates on this issue please ? What might be causing this ? what plugin combination might be causing this ?

            vishakumar Vishal kumar added a comment - Any updates on this issue please ? What might be causing this ? what plugin combination might be causing this ?
            materigoprecitec Mate Rigo added a comment -

            The only feedback I can give is, that ever since I have removed all the jenkins pipeline tee commands, this issue has never reappeared.

            I know I was told that it is not totally related to this, but yet this is some input that might interest folks who follow this thread.

             

            Cheers!

            materigoprecitec Mate Rigo added a comment - The only feedback I can give is, that ever since I have removed all the jenkins pipeline tee commands, this issue has never reappeared. I know I was told that it is not totally related to this, but yet this is some input that might interest folks who follow this thread.   Cheers!

            People

              Unassigned Unassigned
              rshade Robert Shade
              Votes:
              9 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated: