Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63778

Windows agents drop connection on large unstash

    XMLWordPrintable

Details

    Description

      Similar to https://issues.jenkins-ci.org/browse/JENKINS-61314 (I'll link it)

      I'm finding my EC2 Agents sometimes lose their connection during a large unstash (don't ask why. I'm working on that).

      At that point I the Agent remains offline. I can see on the Agent side it's trying to ping the server every 5 minutes without success.

       

      I'll add some logs in the comments

      Attachments

        Issue Links

          Activity

            geoff Geoff Dunn added a comment -

            Example failure from job console
            [Pipeline] unstash[Pipeline] }[Pipeline] // timestamps[Pipeline] cleanWsEC2 (AWS Dev Account) - Windows Test Executor (i-056d044cb9d06f9f9) was marked offline: Connection was broken: java.io.EOFException
            at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2842)
            at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3337)
            at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:925)
            at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:368)
            at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
            at hudson.remoting.Command.readFrom(Command.java:142)
            at hudson.remoting.Command.readFrom(Command.java:128)
            at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
            Caused: java.io.IOException: Unexpected termination of the channel
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)[Pipeline] }[Pipeline] // stage[Pipeline] }[Pipeline] // node[Pipeline] End of Pipeline[Checks API] No suitable checks publisher found.
            Also: hudson.remoting.ProxyException: hudson.model.Computer$TerminationRequest: Termination requested at Fri Sep 25 16:28:40 AEST 2020 by Thread[Ping thread for channel hudson.remoting.Channel@68806821:EC2 (AWS Dev Account) - Windows Test Executor (i-056d044cb9d06f9f9),5,main] [id=7700]
            at hudson.model.Computer.recordTermination(Computer.java:229)
            at hudson.model.Computer.disconnect(Computer.java:495)
            at hudson.slaves.SlaveComputer.disconnect(SlaveComputer.java:759)
            at hudson.slaves.ChannelPinger$1.onDead(ChannelPinger.java:198)
            at hudson.remoting.PingThread.run(PingThread.java:101)
            hudson.remoting.ProxyException: java.io.IOException: Unable to create live FilePath for EC2 (AWS Dev Account) - Windows Test Executor (i-056d044cb9d06f9f9)
            at org.jenkinsci.plugins.workflow.support.steps.FilePathDynamicContext.get(FilePathDynamicContext.java:64)
            at org.jenkinsci.plugins.workflow.support.steps.FilePathDynamicContext.get(FilePathDynamicContext.java:47)
            at org.jenkinsci.plugins.workflow.steps.DynamicContext$Typed.get(DynamicContext.java:94)
            at org.jenkinsci.plugins.workflow.cps.ContextVariableSet.get(ContextVariableSet.java:138)
            at org.jenkinsci.plugins.workflow.cps.CpsThread.getContextVariable(CpsThread.java:135)
            at org.jenkinsci.plugins.workflow.cps.CpsStepContext.doGet(CpsStepContext.java:297)
            at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:67)
            at org.jenkinsci.plugins.workflow.steps.StepDescriptor.checkContextAvailability(StepDescriptor.java:264)
            at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:263)
            Caused: hudson.remoting.ProxyException: org.codehaus.groovy.runtime.InvokerInvocationException: java.io.IOException: Unable to create live FilePath for EC2 (AWS Dev Account) - Windows Test Executor (i-056d044cb9d06f9f9)
            at org.jenkinsci.plugins.workflow.cps.CpsStepContext.replay(CpsStepContext.java:496)
            at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:317)
            at org.jenkinsci.plugins.workflow.cps.DSL.invokeDescribable(DSL.java:417)
            at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:182)
            at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
            at jdk.internal.reflect.GeneratedMethodAccessor518.invoke(Unknown Source)
            at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.base/java.lang.reflect.Method.invoke(Method.java:566)
            at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
            at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
            at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
            at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
            at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
            at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
            at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
            at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
            at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
            at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157)
            at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142)
            at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
            at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
            at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
            Caused: hudson.remoting.ProxyException: java.lang.IllegalArgumentException: Failed to prepare cleanWs step
            at org.jenkinsci.plugins.workflow.cps.DSL.invokeDescribable(DSL.java:419)
            at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:182)
            at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
            at jdk.internal.reflect.GeneratedMethodAccessor518.invoke(Unknown Source)
            at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.base/java.lang.reflect.Method.invoke(Method.java:566)
            at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
            at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
            at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
            at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
            at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
            at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
            at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
            at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
            at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
            at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157)
            at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142)
            at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
            at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
            at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
            at WorkflowScript.withWorkspace(WorkflowScript:551)
            at WorkflowScript.withWorkspaceFromStash(WorkflowScript:565)
            at WorkflowScript.run(WorkflowScript:342)
            at __cps.transform__(Native Method)
            at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
            at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
            at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixName(FunctionCallBlock.java:78)
            at jdk.internal.reflect.GeneratedMethodAccessor510.invoke(Unknown Source)
            at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.base/java.lang.reflect.Method.invoke(Method.java:566)
            at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
            at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
            at com.cloudbees.groovy.cps.Next.step(Next.java:83)
            at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
            at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
            at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
            at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
            at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
            at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
            at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
            at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
            at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
            at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
            at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
            at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
            at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
            at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
            at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:136)
            at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
            at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
            at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
            at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
            at java.base/java.lang.Thread.run(Thread.java:834)
            Finished: FAILURE

            geoff Geoff Dunn added a comment - Example failure from job console [Pipeline] unstash [Pipeline] } [Pipeline] // timestamps [Pipeline] cleanWsEC2 (AWS Dev Account) - Windows Test Executor (i-056d044cb9d06f9f9) was marked offline: Connection was broken: java.io.EOFException at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2842) at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3337) at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:925) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:368) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) Caused: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77) [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline [Checks API] No suitable checks publisher found. Also: hudson.remoting.ProxyException: hudson.model.Computer$TerminationRequest: Termination requested at Fri Sep 25 16:28:40 AEST 2020 by Thread [Ping thread for channel hudson.remoting.Channel@68806821:EC2 (AWS Dev Account) - Windows Test Executor (i-056d044cb9d06f9f9),5,main] [id=7700] at hudson.model.Computer.recordTermination(Computer.java:229) at hudson.model.Computer.disconnect(Computer.java:495) at hudson.slaves.SlaveComputer.disconnect(SlaveComputer.java:759) at hudson.slaves.ChannelPinger$1.onDead(ChannelPinger.java:198) at hudson.remoting.PingThread.run(PingThread.java:101) hudson.remoting.ProxyException: java.io.IOException: Unable to create live FilePath for EC2 (AWS Dev Account) - Windows Test Executor (i-056d044cb9d06f9f9) at org.jenkinsci.plugins.workflow.support.steps.FilePathDynamicContext.get(FilePathDynamicContext.java:64) at org.jenkinsci.plugins.workflow.support.steps.FilePathDynamicContext.get(FilePathDynamicContext.java:47) at org.jenkinsci.plugins.workflow.steps.DynamicContext$Typed.get(DynamicContext.java:94) at org.jenkinsci.plugins.workflow.cps.ContextVariableSet.get(ContextVariableSet.java:138) at org.jenkinsci.plugins.workflow.cps.CpsThread.getContextVariable(CpsThread.java:135) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.doGet(CpsStepContext.java:297) at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:67) at org.jenkinsci.plugins.workflow.steps.StepDescriptor.checkContextAvailability(StepDescriptor.java:264) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:263) Caused: hudson.remoting.ProxyException: org.codehaus.groovy.runtime.InvokerInvocationException: java.io.IOException: Unable to create live FilePath for EC2 (AWS Dev Account) - Windows Test Executor (i-056d044cb9d06f9f9) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.replay(CpsStepContext.java:496) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:317) at org.jenkinsci.plugins.workflow.cps.DSL.invokeDescribable(DSL.java:417) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:182) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) at jdk.internal.reflect.GeneratedMethodAccessor518.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163) at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165) at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17) Caused: hudson.remoting.ProxyException: java.lang.IllegalArgumentException: Failed to prepare cleanWs step at org.jenkinsci.plugins.workflow.cps.DSL.invokeDescribable(DSL.java:419) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:182) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) at jdk.internal.reflect.GeneratedMethodAccessor518.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163) at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165) at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17) at WorkflowScript.withWorkspace(WorkflowScript:551) at WorkflowScript.withWorkspaceFromStash(WorkflowScript:565) at WorkflowScript.run(WorkflowScript:342) at __ cps.transform __(Native Method) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixName(FunctionCallBlock.java:78) at jdk.internal.reflect.GeneratedMethodAccessor510.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) at com.cloudbees.groovy.cps.Next.step(Next.java:83) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:136) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Finished: FAILURE
            geoff Geoff Dunn added a comment -

            Agent log (from the server)

            EC2 (AWS Dev Account) - Windows Test Executor (i-056d044cb9d06f9f9) booted at 1601008317000EC2 (AWS Dev Account) - Windows Test Executor (i-056d044cb9d06f9f9) booted at 1601008317000Connecting to (10.161.10.40) with WinRM as AdministratorWaiting for WinRM to come up. Sleeping 10s.Waiting for WinRM to come up. Sleeping 10s.Waiting for WinRM to come up. Sleeping 10s.WinRM service responded. Waiting for WinRM service to stabilize on EC2 (AWS Dev Account) - Windows Test Executor (i-056d044cb9d06f9f9)WinRM should now be ok on EC2 (AWS Dev Account) - Windows Test Executor (i-056d044cb9d06f9f9)Connected with WinRM.Creating tmp directory if it does not existremoting.jar sent remotely. Bootstrapping itLaunching via WinRM:java -Xmx2048m -jar C:\Windows\Temp\remoting.jar -workDir c:\jenkins<===[JENKINS REMOTING CAPACITY]===>Remoting version: 4.5This is a Windows agentERROR: Failed to monitor for Free Swap Spacejava.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:320) at hudson.remoting.Request$1.get(Request.java:239) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:64) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:78) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:306)ERROR: Failed to monitor for Free Temp Spacejava.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:320) at hudson.remoting.Request$1.get(Request.java:239) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:64) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:78) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:306)ERROR: Failed to monitor for Free Disk Spacejava.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:320) at hudson.remoting.Request$1.get(Request.java:239) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:64) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:78) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:306)Agent successfully connected and onlineERROR: Connection terminatedhudson.remoting.FastPipedInputStream$ClosedBy: The pipe was closed at... at hudson.remoting.FastPipedOutputStream.error(FastPipedOutputStream.java:101) at hudson.remoting.FastPipedOutputStream.close(FastPipedOutputStream.java:90) at hudson.plugins.ec2.util.Closeables.closeQuietly(Closeables.java:23) at hudson.plugins.ec2.win.winrm.WindowsProcess$2.run(WindowsProcess.java:146)Caused: java.io.IOException: Pipe is already closed at hudson.remoting.FastPipedOutputStream.write(FastPipedOutputStream.java:156) at hudson.remoting.FastPipedOutputStream.write(FastPipedOutputStream.java:140) at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89) at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85) at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54) at java.base/java.io.OutputStream.write(OutputStream.java:122) at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46) at hudson.remoting.Channel.send(Channel.java:766) at hudson.remoting.Channel.close(Channel.java:1488) at hudson.remoting.Channel.close(Channel.java:1455) at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:874) at hudson.slaves.SlaveComputer.access$100(SlaveComputer.java:110) at hudson.slaves.SlaveComputer$2.run(SlaveComputer.java:765) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)ERROR: Connection terminatedjava.io.EOFException at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2842) at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3337) at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:925) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:368) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)Caused: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)

            geoff Geoff Dunn added a comment - Agent log (from the server) EC2 (AWS Dev Account) - Windows Test Executor (i-056d044cb9d06f9f9) booted at 1601008317000EC2 (AWS Dev Account) - Windows Test Executor (i-056d044cb9d06f9f9) booted at 1601008317000Connecting to (10.161.10.40) with WinRM as AdministratorWaiting for WinRM to come up. Sleeping 10s.Waiting for WinRM to come up. Sleeping 10s.Waiting for WinRM to come up. Sleeping 10s.WinRM service responded. Waiting for WinRM service to stabilize on EC2 (AWS Dev Account) - Windows Test Executor (i-056d044cb9d06f9f9)WinRM should now be ok on EC2 (AWS Dev Account) - Windows Test Executor (i-056d044cb9d06f9f9)Connected with WinRM.Creating tmp directory if it does not existremoting.jar sent remotely. Bootstrapping itLaunching via WinRM:java -Xmx2048m -jar C:\Windows\Temp\remoting.jar -workDir c:\jenkins<=== [JENKINS REMOTING CAPACITY] ===>Remoting version: 4.5This is a Windows agentERROR: Failed to monitor for Free Swap Spacejava.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:320) at hudson.remoting.Request$1.get(Request.java:239) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:64) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:78) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:306)ERROR: Failed to monitor for Free Temp Spacejava.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:320) at hudson.remoting.Request$1.get(Request.java:239) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:64) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:78) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:306)ERROR: Failed to monitor for Free Disk Spacejava.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:320) at hudson.remoting.Request$1.get(Request.java:239) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:64) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:78) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:306)Agent successfully connected and onlineERROR: Connection terminatedhudson.remoting.FastPipedInputStream$ClosedBy: The pipe was closed at... at hudson.remoting.FastPipedOutputStream.error(FastPipedOutputStream.java:101) at hudson.remoting.FastPipedOutputStream.close(FastPipedOutputStream.java:90) at hudson.plugins.ec2.util.Closeables.closeQuietly(Closeables.java:23) at hudson.plugins.ec2.win.winrm.WindowsProcess$2.run(WindowsProcess.java:146)Caused: java.io.IOException: Pipe is already closed at hudson.remoting.FastPipedOutputStream.write(FastPipedOutputStream.java:156) at hudson.remoting.FastPipedOutputStream.write(FastPipedOutputStream.java:140) at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89) at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85) at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54) at java.base/java.io.OutputStream.write(OutputStream.java:122) at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46) at hudson.remoting.Channel.send(Channel.java:766) at hudson.remoting.Channel.close(Channel.java:1488) at hudson.remoting.Channel.close(Channel.java:1455) at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:874) at hudson.slaves.SlaveComputer.access$100(SlaveComputer.java:110) at hudson.slaves.SlaveComputer$2.run(SlaveComputer.java:765) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)ERROR: Connection terminatedjava.io.EOFException at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2842) at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3337) at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:925) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:368) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)Caused: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
            geoff Geoff Dunn added a comment -

            Log on the Agent side

            {{Sep 24, 2020 12:04:11 PM hudson.slaves.ChannelPinger$1 onDead
            INFO: Ping failed. Terminating the channel channel.
            java.util.concurrent.TimeoutException: Ping started at 1600912811820 hasn't completed by 1600913051827
            at hudson.remoting.PingThread.ping(PingThread.java:134)
            at hudson.remoting.PingThread.run(PingThread.java:90)Sep 24, 2020 12:09:11 PM hudson.slaves.ChannelPinger$1 onDead
            INFO: Ping failed. Terminating the channel channel.
            java.util.concurrent.TimeoutException: Ping started at 1600913111821 hasn't completed by 1600913351832
            at hudson.remoting.PingThread.ping(PingThread.java:134)
            at hudson.remoting.PingThread.run(PingThread.java:90)Sep 24, 2020 12:14:11 PM hudson.slaves.ChannelPinger$1 onDead
            INFO: Ping failed. Terminating the channel channel.
            java.util.concurrent.TimeoutException: Ping started at 1600913411822 hasn't completed by 1600913651827
            at hudson.remoting.PingThread.ping(PingThread.java:134)
            at hudson.remoting.PingThread.run(PingThread.java:90)Sep 24, 2020 12:19:11 PM hudson.slaves.ChannelPinger$1 onDead
            INFO: Ping failed. Terminating the channel channel.
            java.util.concurrent.TimeoutException: Ping started at 1600913711822 hasn't completed by 1600913951829
            at hudson.remoting.PingThread.ping(PingThread.java:134)
            at hudson.remoting.PingThread.run(PingThread.java:90)Sep 24, 2020 12:24:11 PM hudson.slaves.ChannelPinger$1 onDead
            INFO: Ping failed. Terminating the channel channel.
            java.util.concurrent.TimeoutException: Ping started at 1600914011822 hasn't completed by 1600914251828
            at hudson.remoting.PingThread.ping(PingThread.java:134)
            at hudson.remoting.PingThread.run(PingThread.java:90)Sep 24, 2020 12:29:11 PM hudson.slaves.ChannelPinger$1 onDead
            INFO: Ping failed. Terminating the channel channel.
            java.util.concurrent.TimeoutException: Ping started at 1600914311823 hasn't completed by 1600914551837
            at hudson.remoting.PingThread.ping(PingThread.java:134)
            at hudson.remoting.PingThread.run(PingThread.java:90)Sep 24, 2020 12:34:11 PM hudson.slaves.ChannelPinger$1 onDead
            INFO: Ping failed. Terminating the channel channel.
            java.util.concurrent.TimeoutException: Ping started at 1600914611824 hasn't completed by 1600914851830
            at hudson.remoting.PingThread.ping(PingThread.java:134)
            at hudson.remoting.PingThread.run(PingThread.java:90)Sep 24, 2020 12:39:11 PM hudson.slaves.ChannelPinger$1 onDead
            INFO: Ping failed. Terminating the channel channel.
            java.util.concurrent.TimeoutException: Ping started at 1600914911824 hasn't completed by 1600915151831
            at hudson.remoting.PingThread.ping(PingThread.java:134)
            at hudson.remoting.PingThread.run(PingThread.java:90)Sep 24, 2020 12:44:11 PM hudson.slaves.ChannelPinger$1 onDead
            INFO: Ping failed. Terminating the channel channel.
            java.util.concurrent.TimeoutException: Ping started at 1600915211824 hasn't completed by 1600915451831
            at hudson.remoting.PingThread.ping(PingThread.java:134)
            at hudson.remoting.PingThread.run(PingThread.java:90)Sep 24, 2020 12:49:11 PM hudson.slaves.ChannelPinger$1 onDead
            INFO: Ping failed. Terminating the channel channel.
            java.util.concurrent.TimeoutException: Ping started at 1600915511824 hasn't completed by 1600915751832
            at hudson.remoting.PingThread.ping(PingThread.java:134)
            at hudson.remoting.PingThread.run(PingThread.java:90)

            geoff Geoff Dunn added a comment - Log on the Agent side {{ Sep 24, 2020 12:04:11 PM hudson.slaves.ChannelPinger$1 onDead INFO: Ping failed. Terminating the channel channel. java.util.concurrent.TimeoutException: Ping started at 1600912811820 hasn't completed by 1600913051827 at hudson.remoting.PingThread.ping(PingThread.java:134) at hudson.remoting.PingThread.run(PingThread.java:90) Sep 24, 2020 12:09:11 PM hudson.slaves.ChannelPinger$1 onDead INFO: Ping failed. Terminating the channel channel. java.util.concurrent.TimeoutException: Ping started at 1600913111821 hasn't completed by 1600913351832 at hudson.remoting.PingThread.ping(PingThread.java:134) at hudson.remoting.PingThread.run(PingThread.java:90) Sep 24, 2020 12:14:11 PM hudson.slaves.ChannelPinger$1 onDead INFO: Ping failed. Terminating the channel channel. java.util.concurrent.TimeoutException: Ping started at 1600913411822 hasn't completed by 1600913651827 at hudson.remoting.PingThread.ping(PingThread.java:134) at hudson.remoting.PingThread.run(PingThread.java:90) Sep 24, 2020 12:19:11 PM hudson.slaves.ChannelPinger$1 onDead INFO: Ping failed. Terminating the channel channel. java.util.concurrent.TimeoutException: Ping started at 1600913711822 hasn't completed by 1600913951829 at hudson.remoting.PingThread.ping(PingThread.java:134) at hudson.remoting.PingThread.run(PingThread.java:90) Sep 24, 2020 12:24:11 PM hudson.slaves.ChannelPinger$1 onDead INFO: Ping failed. Terminating the channel channel. java.util.concurrent.TimeoutException: Ping started at 1600914011822 hasn't completed by 1600914251828 at hudson.remoting.PingThread.ping(PingThread.java:134) at hudson.remoting.PingThread.run(PingThread.java:90) Sep 24, 2020 12:29:11 PM hudson.slaves.ChannelPinger$1 onDead INFO: Ping failed. Terminating the channel channel. java.util.concurrent.TimeoutException: Ping started at 1600914311823 hasn't completed by 1600914551837 at hudson.remoting.PingThread.ping(PingThread.java:134) at hudson.remoting.PingThread.run(PingThread.java:90) Sep 24, 2020 12:34:11 PM hudson.slaves.ChannelPinger$1 onDead INFO: Ping failed. Terminating the channel channel. java.util.concurrent.TimeoutException: Ping started at 1600914611824 hasn't completed by 1600914851830 at hudson.remoting.PingThread.ping(PingThread.java:134) at hudson.remoting.PingThread.run(PingThread.java:90) Sep 24, 2020 12:39:11 PM hudson.slaves.ChannelPinger$1 onDead INFO: Ping failed. Terminating the channel channel. java.util.concurrent.TimeoutException: Ping started at 1600914911824 hasn't completed by 1600915151831 at hudson.remoting.PingThread.ping(PingThread.java:134) at hudson.remoting.PingThread.run(PingThread.java:90) Sep 24, 2020 12:44:11 PM hudson.slaves.ChannelPinger$1 onDead INFO: Ping failed. Terminating the channel channel. java.util.concurrent.TimeoutException: Ping started at 1600915211824 hasn't completed by 1600915451831 at hudson.remoting.PingThread.ping(PingThread.java:134) at hudson.remoting.PingThread.run(PingThread.java:90) Sep 24, 2020 12:49:11 PM hudson.slaves.ChannelPinger$1 onDead INFO: Ping failed. Terminating the channel channel. java.util.concurrent.TimeoutException: Ping started at 1600915511824 hasn't completed by 1600915751832 at hudson.remoting.PingThread.ping(PingThread.java:134) at hudson.remoting.PingThread.run(PingThread.java:90)
            sodul Stephane Odul added a comment -

            We are experiencing the same issue very frequently when unstashing large number of files. We use the Artifact Manager on S3 plugin and we run our workload through kubernetes

            sodul Stephane Odul added a comment - We are experiencing the same issue very frequently when unstashing large number of files. We use the  Artifact Manager on S3 plugin  and we run our workload through kubernetes
            adubey alok dubey added a comment -

            We are also experiencing this issue since the recent Jenkins upgrade. Any quick help would be appreciated, thanks!

            adubey alok dubey added a comment - We are also experiencing this issue since the recent Jenkins upgrade. Any quick help would be appreciated, thanks!

            People

              thoulen FABRIZIO MANFREDI
              geoff Geoff Dunn
              Votes:
              6 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: