Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56446

IOException in FileLogStorage maybeFlush

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Some jobs get stuck in a state where they're unable to continue to write data to the console log.  The system log contains multiple entries of the following form until the job ends:

       

      failed to flush /net/users/jenkins/configuration/jobs/XXX/builds/5/log
      java.io.IOException: Stream Closed
      at java.io.FileOutputStream.writeBytes(Native Method)
      at java.io.FileOutputStream.write(FileOutputStream.java:326)
      at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream$FlushControlledOutputStream.write(DelayBufferedOutputStream.java:134)
      at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
      at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
      at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
      at org.jenkinsci.plugins.workflow.log.FileLogStorage.maybeFlush(FileLogStorage.java:190)
      at org.jenkinsci.plugins.workflow.log.FileLogStorage.overallLog(FileLogStorage.java:198)
      at org.jenkinsci.plugins.workflow.job.WorkflowRun.getLogText(WorkflowRun.java:1018)
      at org.jenkinsci.plugins.workflow.job.WorkflowRun.getLogInputStream(WorkflowRun.java:1026)
      at org.jenkinsci.plugins.gwt.GenericWebhookEnvironmentContributor.notLogged(GenericWebhookEnvironmentContributor.java:54)
      at org.jenkinsci.plugins.gwt.GenericWebhookEnvironmentContributor.buildEnvironmentFor(GenericWebhookEnvironmentContributor.java:30)
      at hudson.model.Run.getEnvironment(Run.java:2373)
      at org.jenkinsci.plugins.workflow.job.WorkflowRun.getEnvironment(WorkflowRun.java:468)
      at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getEnvironment(EnvActionImpl.java:86)
      at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getEnvironment(EnvActionImpl.java:67)
      at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:72)
      at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:305)
      at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:268)
      at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:176)
      at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
      at sun.reflect.GeneratedMethodAccessor387.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
      at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
      at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
      at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
      at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
      at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
      at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
      at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157)
      at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
      at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:155)
      at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
      at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159)
      at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
      at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
      at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
      at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
      at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      at sun.reflect.GeneratedMethodAccessor384.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:103)
      at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      at sun.reflect.GeneratedMethodAccessor384.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:60)
      at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
      at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      at sun.reflect.GeneratedMethodAccessor384.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
      at com.cloudbees.groovy.cps.Next.step(Next.java:83)
      at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
      at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
      at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
      at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
      at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
      at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
      at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
      at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:121)
      at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
      at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:182)
      at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
      at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
      at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
      at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
      at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
      at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)

       

      When enabling DelayBufferedOutputStream logging, we also see this in the log:

      null
      java.io.IOException: Stream Closed
      at java.io.FileOutputStream.writeBytes(Native Method)
      at java.io.FileOutputStream.write(FileOutputStream.java:326)
      at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream$FlushControlledOutputStream.write(DelayBufferedOutputStream.java:134)
      at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
      at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
      at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.flushBuffer(DelayBufferedOutputStream.java:82)
      at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.flushAndReschedule(DelayBufferedOutputStream.java:91)
      at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream$Flush.run(DelayBufferedOutputStream.java:114)
      at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)

        Attachments

          Issue Links

            Activity

            Hide
            haw777 Christopher Head added a comment -

            I see this problem. I’m not using logstash. I’m not using NFS; storage is a local btrfs filesystem. After the problem shows up, restarting Jenkins seems to clear it up, in the sense that jobs keep running (as they usually do across a restart) but the missing output arrives and the console view starts showing live output again.

            Show
            haw777 Christopher Head added a comment - I see this problem. I’m not using logstash. I’m not using NFS; storage is a local btrfs filesystem. After the problem shows up, restarting Jenkins seems to clear it up, in the sense that jobs keep running (as they usually do across a restart) but the missing output arrives and the console view starts showing live output again.
            Hide
            owenmehegan Owen Mehegan added a comment -

            Christopher Head thanks for your input. I wonder if the btrfs filesystem could be a contributor here. We've wondered if the root cause of this is a transient IO performance issue where writes get slowed down or blocked enough to then screw up or overload a buffer in Jenkins. The original stack trace in this bug has a reference to the Generic Webhook plugin in addition to the expected references to the Pipeline plugins. In another case I have seen references to the Splunk plugin, and in that case the issue was greatly reduced (though not eliminated) by downgrading the version of that. Can you (and anyone else with this issue) share the exact stack trace you are seeing?

            Show
            owenmehegan Owen Mehegan added a comment - Christopher Head thanks for your input. I wonder if the btrfs filesystem could be a contributor here. We've wondered if the root cause of this is a transient IO performance issue where writes get slowed down or blocked enough to then screw up or overload a buffer in Jenkins. The original stack trace in this bug has a reference to the Generic Webhook plugin in addition to the expected references to the Pipeline plugins. In another case I have seen references to the Splunk plugin, and in that case the issue was greatly reduced (though not eliminated) by downgrading the version of that. Can you (and anyone else with this issue) share the exact stack trace you are seeing?
            Hide
            haw777 Christopher Head added a comment -

            I can definitely confirm that, when the problem happened, the machine in question was in a state where it suffered slower than normal I/O performance due to a hardware issue. Here’s my stack trace.

            2021-11-30 04:11:42.956+0000 [id=58892]	WARNING	o.j.p.w.log.FileLogStorage#maybeFlush: failed to flush /var/lib/jenkins/jobs/<REDACTED>/branches/<REDACTED>/builds/2/log
            java.io.IOException: Stream Closed
            	at java.base/java.io.FileOutputStream.writeBytes(Native Method)
            	at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
            	at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream$FlushControlledOutputStream.write(DelayBufferedOutputStream.java:125)
            	at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
            	at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
            	at java.base/java.io.FilterOutputStream.flush(FilterOutputStream.java:153)
            	at org.jenkinsci.plugins.workflow.log.FileLogStorage.maybeFlush(FileLogStorage.java:190)
            	at org.jenkinsci.plugins.workflow.log.FileLogStorage.overallLog(FileLogStorage.java:198)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getLogText(WorkflowRun.java:1065)
            	at jdk.internal.reflect.GeneratedMethodAccessor1058.invoke(Unknown Source)
            	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
            	at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
            	at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
            	at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
            	at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
            	at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
            	at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
            	at org.apache.commons.jexl.parser.ASTGTNode.value(ASTGTNode.java:62)
            	at org.apache.commons.jexl.parser.ASTExpression.value(ASTExpression.java:54)
            	at org.apache.commons.jexl.parser.ASTExpressionExpression.value(ASTExpressionExpression.java:56)
            	at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
            	at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:76)
            	at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61)
            	at org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsBoolean(ExpressionSupport.java:71)
            	at org.apache.commons.jelly.tags.core.WhenTag.doTag(WhenTag.java:44)
            	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
            	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
            	at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
            	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
            	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            	at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
            	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
            	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
            	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100)
            	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
            	at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
            	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
            	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
            	at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
            	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
            	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            	at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
            	at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:171)
            	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
            	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            	at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
            	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
            	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
            	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100)
            	at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
            	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100)
            	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100)
            	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100)
            	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            	at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
            	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
            	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
            	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
            	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
            	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            	at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
            	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:64)
            	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:54)
            	at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:56)
            	at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:43)
            	at org.kohsuke.stapler.Facet.handleIndexRequest(Facet.java:284)
            	at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:100)
            	at org.kohsuke.stapler.IndexViewDispatcher.dispatch(IndexViewDispatcher.java:32)
            	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
            	at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:457)
            	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
            	at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:281)
            	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
            	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
            	at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:281)
            	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
            	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:694)
            	at org.kohsuke.stapler.Stapler.service(Stapler.java:240)
            	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
            	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
            	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:156)
            	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:153)
            	at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:153)
            	at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:80)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:153)
            	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:153)
            	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:153)
            	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:159)
            	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
            	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
            	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:159)
            	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
            	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:92)
            	at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
            	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:53)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
            	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:121)
            	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
            	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
            	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:133)
            	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:92)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
            	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:218)
            	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:212)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
            	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
            	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
            	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
            	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:62)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
            	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:109)
            	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
            	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
            	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
            	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:51)
            	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
            	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
            	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:85)
            	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
            	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
            	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
            	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
            	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
            	at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:35)
            	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
            	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
            	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
            	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
            	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
            	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
            	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
            	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
            	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
            	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
            	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
            	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
            	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
            	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
            	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
            	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
            	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
            	at org.eclipse.jetty.server.Server.handle(Server.java:516)
            	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
            	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
            	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
            	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
            	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
            	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
            	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
            	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
            	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
            	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
            	at java.base/java.lang.Thread.run(Thread.java:829)
            
            Show
            haw777 Christopher Head added a comment - I can definitely confirm that, when the problem happened, the machine in question was in a state where it suffered slower than normal I/O performance due to a hardware issue. Here’s my stack trace. 2021-11-30 04:11:42.956+0000 [id=58892] WARNING o.j.p.w.log.FileLogStorage#maybeFlush: failed to flush /var/lib/jenkins/jobs/<REDACTED>/branches/<REDACTED>/builds/2/log java.io.IOException: Stream Closed at java.base/java.io.FileOutputStream.writeBytes(Native Method) at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354) at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream$FlushControlledOutputStream.write(DelayBufferedOutputStream.java:125) at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) at java.base/java.io.FilterOutputStream.flush(FilterOutputStream.java:153) at org.jenkinsci.plugins.workflow.log.FileLogStorage.maybeFlush(FileLogStorage.java:190) at org.jenkinsci.plugins.workflow.log.FileLogStorage.overallLog(FileLogStorage.java:198) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getLogText(WorkflowRun.java:1065) at jdk.internal.reflect.GeneratedMethodAccessor1058.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125) at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314) at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185) at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75) at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83) at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57) at org.apache.commons.jexl.parser.ASTGTNode.value(ASTGTNode.java:62) at org.apache.commons.jexl.parser.ASTExpression.value(ASTExpression.java:54) at org.apache.commons.jexl.parser.ASTExpressionExpression.value(ASTExpressionExpression.java:56) at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80) at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:76) at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61) at org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsBoolean(ExpressionSupport.java:71) at org.apache.commons.jelly.tags.core.WhenTag.doTag(WhenTag.java:44) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:171) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:64) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:54) at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:56) at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:43) at org.kohsuke.stapler.Facet.handleIndexRequest(Facet.java:284) at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:100) at org.kohsuke.stapler.IndexViewDispatcher.dispatch(IndexViewDispatcher.java:32) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898) at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:457) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:281) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:281) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:694) at org.kohsuke.stapler.Stapler.service(Stapler.java:240) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:156) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:153) at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:153) at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:80) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:153) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:153) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:153) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:159) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:159) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:92) at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:53) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:121) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:133) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:92) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:218) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:212) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:62) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:109) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:51) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:85) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:35) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.base/java.lang.Thread.run(Thread.java:829)
            Hide
            owenmehegan Owen Mehegan added a comment -

            Christopher Head thanks for that. In your case I only see references to Pipeline (workflow) and the SSE Gateway plugin, which I think is just coming through via Blue Ocean. I continue to suspect that this issue is caused by a storage performance problem. Whether that issue is triggered by excessive IO from Jenkins (maybe the case with the Splunk and Logstash plugins doing some heavy reads or something) or just unrelated infrastructure problems may be irrelevant. I don't know whether the Jenkins code here can be made more robust. Needs input from a Pipeline maintainer.

            Show
            owenmehegan Owen Mehegan added a comment - Christopher Head thanks for that. In your case I only see references to Pipeline (workflow) and the SSE Gateway plugin, which I think is just coming through via Blue Ocean. I continue to suspect that this issue is caused by a storage performance problem. Whether that issue is triggered by excessive IO from Jenkins (maybe the case with the Splunk and Logstash plugins doing some heavy reads or something) or just unrelated infrastructure problems may be irrelevant. I don't know whether the Jenkins code here can be made more robust. Needs input from a Pipeline maintainer.
            Hide
            haw777 Christopher Head added a comment -

            I’m not sure how steps actually run under the hood, but it seems to me, if the step writes to a pipe which Jenkins is reading, then if the disk is unable to keep up with writing log data to the job directory then it should stop reading the pipe, which will apply backpressure, eventually slowing down the process producing the output in the first place. On the other hand, if the step writes to a file which Jenkins is tailing, then if the disk is unable to keep up with writing log data to the job directory then it should just read the output file less frequently (i.e. if Jenkins’s buffer is full, then stop reading from the step output file until it drains); this might result in the log data in the Job directory getting significantly behind the process’s live output, but that seems like a much less bad failure mode than completely blowing up and not working at all, which is what we get right now—and eventually, presumably, the running process will end, and thus stop producing output, allowing Jenkins to catch up before starting the next step.

            But this is all hypothesizing by someone who really doesn’t know how Jenkins works.

            Show
            haw777 Christopher Head added a comment - I’m not sure how steps actually run under the hood, but it seems to me, if the step writes to a pipe which Jenkins is reading, then if the disk is unable to keep up with writing log data to the job directory then it should stop reading the pipe, which will apply backpressure, eventually slowing down the process producing the output in the first place. On the other hand, if the step writes to a file which Jenkins is tailing, then if the disk is unable to keep up with writing log data to the job directory then it should just read the output file less frequently (i.e. if Jenkins’s buffer is full, then stop reading from the step output file until it drains); this might result in the log data in the Job directory getting significantly behind the process’s live output, but that seems like a much less bad failure mode than completely blowing up and not working at all, which is what we get right now—and eventually, presumably, the running process will end, and thus stop producing output, allowing Jenkins to catch up before starting the next step. But this is all hypothesizing by someone who really doesn’t know how Jenkins works.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              rshade Robert Shade
              Votes:
              8 Vote for this issue
              Watchers:
              14 Start watching this issue

                Dates

                Created:
                Updated: