Reproduced in jenkins ver 1.560

      Apr 21, 2014 10:46:43 AM hudson.remoting.Channel$2 handle
      SEVERE: Failed to execute command Pipe.EOF(1895) (channel SRV6026)
      java.lang.NullPointerException
              at hudson.remoting.ProxyWriter$EOF.execute(ProxyWriter.java:177)
              at hudson.remoting.Channel$2.handle(Channel.java:461)
              at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
      
      Apr 21, 2014 10:46:43 AM hudson.remoting.Channel$2 handle
      SEVERE: This command is created here
      Command Pipe.EOF(1895) created at
              at hudson.remoting.Command.<init>(Command.java:67)
              at hudson.remoting.Command.<init>(Command.java:50)
              at hudson.remoting.ProxyWriter$EOF.<init>(ProxyWriter.java:169)
              at hudson.remoting.ProxyWriter.close(ProxyWriter.java:124)
              at hudson.remoting.ProxyWriter.finalize(ProxyWriter.java:132)
              at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
              at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:101)
              at java.lang.ref.Finalizer.access$100(Finalizer.java:32)
              at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:190)
      

      Issue appears when jenkins try to archive artifacts when job runs on slave. Some of artifacts has 150+ Mb size.
      After abort job in console I have:

      ERROR: Failed to parse POMs
      java.io.IOException: java.io.IOException: Failed to extract /var/lib/jenkins/workspace/Platform-0/assemblies/karaf/transfer of 3 files
      	at hudson.FilePath.readFromTar(FilePath.java:2095)
      	at hudson.FilePath.copyRecursiveTo(FilePath.java:2007)
      	at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:57)
      	at hudson.maven.MavenBuild$ProxyImpl.performArchiving(MavenBuild.java:474)
      	at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:841)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:519)
      	at hudson.model.Run.execute(Run.java:1703)
      	at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:525)
      	at hudson.model.ResourceController.execute(ResourceController.java:88)
      	at hudson.model.Executor.run(Executor.java:231)
      Caused by: java.io.IOException
      	at hudson.remoting.FastPipedInputStream.read(FastPipedInputStream.java:177)
      	at hudson.util.HeadBufferingStream.read(HeadBufferingStream.java:61)
      	at com.jcraft.jzlib.InflaterInputStream.fill(InflaterInputStream.java:175)
      	at com.jcraft.jzlib.InflaterInputStream.read(InflaterInputStream.java:106)
      	at org.apache.tools.tar.TarBuffer.readBlock(TarBuffer.java:257)
      	at org.apache.tools.tar.TarBuffer.readRecord(TarBuffer.java:223)
      	at hudson.org.apache.tools.tar.TarInputStream.read(TarInputStream.java:345)
      	at java.io.FilterInputStream.read(FilterInputStream.java:107)
      	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792)
      	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
      	at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
      	at hudson.util.IOUtils.copy(IOUtils.java:40)
      	at hudson.FilePath.readFromTar(FilePath.java:2085)
      	... 9 more
      
      	at hudson.FilePath.copyRecursiveTo(FilePath.java:2014)
      	at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:57)
      	at hudson.maven.MavenBuild$ProxyImpl.performArchiving(MavenBuild.java:474)
      	at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:841)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:519)
      	at hudson.model.Run.execute(Run.java:1703)
      	at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:525)
      	at hudson.model.ResourceController.execute(ResourceController.java:88)
      	at hudson.model.Executor.run(Executor.java:231)
      Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
      	at hudson.remoting.Request$1.get(Request.java:278)
      	at hudson.remoting.Request$1.get(Request.java:210)
      	at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
      	at hudson.FilePath.copyRecursiveTo(FilePath.java:2010)
      	... 8 more
      Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
      	at hudson.remoting.Request.abort(Request.java:299)
      	at hudson.remoting.Channel.terminate(Channel.java:801)
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:76)
      Caused by: java.io.IOException: Unexpected reader termination
      	... 1 more
      Caused by: java.lang.OutOfMemoryError: Java heap space
      	at java.util.Arrays.copyOf(Arrays.java:2271)
      	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
      	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
      	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
      	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87)
      	at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46)
      	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88)
      	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33)
      	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
      Looks like the node went offline during the build. Check the slave log for the details.FATAL: /var/lib/jenkins/logs/slaves/SERVER6026/slave.log (No such file or directory)
      java.io.FileNotFoundException: /var/lib/jenkins/logs/slaves/SERVER6026/slave.log (No such file or directory)
      	at java.io.RandomAccessFile.open(Native Method)
      	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
      	at org.kohsuke.stapler.framework.io.LargeText$FileSession.<init>(LargeText.java:397)
      	at org.kohsuke.stapler.framework.io.LargeText$2.open(LargeText.java:120)
      	at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:210)
      	at hudson.console.AnnotatedLargeText.writeHtmlTo(AnnotatedLargeText.java:156)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:532)
      	at hudson.model.Run.execute(Run.java:1703)
      	at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:525)
      	at hudson.model.ResourceController.execute(ResourceController.java:88)
      	at hudson.model.Executor.run(Executor.java:231)
      

      Jenkins slave runs with: -XX:MaxPermSize=256m -Xmx1024m
      Think that 1024 is really enough for transfer. Slave has only one executor, so no other jobs on slave are running.

          [JENKINS-22734] OutOfMemory on file transfer in 1.560

          jsirex added a comment -

          Probably this issue is related to JENKINS-20769

          jsirex added a comment - Probably this issue is related to JENKINS-20769

          Tomas Kopal added a comment - - edited

          I do not think its related, at least not directly. We have observed very similar issue. But it exhibits in 1.560 only, after downgrading to 1.559 everything works fine again (no change in plugins). If I understand correctly JENKINS-20769, it was fixed in 1.554, but this really looks like regression in 1.560.

          Apr 23, 2014 4:34:45 PM hudson.slaves.SlaveComputer tryReconnect
          	at hudson.FilePath.act(FilePath.java:893)
          	at hudson.FilePath.act(FilePath.java:909)
          	at hudson.FilePath.archive(FilePath.java:423)
          	at hudson.FilePath.tar(FilePath.java:2047)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:740)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:740)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:710)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:710)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:659)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:659)
          	at hudson.model.Build$BuildExecution.post2(Build.java:182)
          	at hudson.model.Build$BuildExecution.post2(Build.java:182)
          	at hudson.model.Executor.run(Executor.java:231)
          	at hudson.model.Executor.run(Executor.java:231)
          	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          	at hudson.model.ResourceController.execute(ResourceController.java:88)
          	at hudson.model.ResourceController.execute(ResourceController.java:88)
          	at hudson.model.Run.execute(Run.java:1728)
          	at hudson.model.Run.execute(Run.java:1728)
          	at hudson.plugins.cloneworkspace.CloneWorkspacePublisher.perform(CloneWorkspacePublisher.java:178)
          	at hudson.plugins.cloneworkspace.CloneWorkspacePublisher.snapshot(CloneWorkspacePublisher.java:231)
          	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
          	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
          	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
          	at hudson.remoting.Channel.call(Channel.java:738)
          	at hudson.remoting.Channel.terminate(Channel.java:801)
          	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33)
          	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33)
          	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33)
          	at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46)
          	at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46)
          	at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46)
          	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88)
          	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88)
          	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88)
          	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87)
          	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87)
          	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87)
          	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
          	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
          	at hudson.remoting.Request.abort(Request.java:299)
          	at hudson.remoting.Request.call(Request.java:174)
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:76)
          	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
          	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
          	at hudson.tasks.Mailer.perform(Mailer.java:137)
          	at hudson.tasks.MailSender.createFailureMail(MailSender.java:278)
          	at hudson.tasks.MailSender.execute(MailSender.java:101)
          	at hudson.tasks.MailSender.getMail(MailSender.java:153)
          	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
          	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
          	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
          	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
          	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
          	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
          	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
          	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
          	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
          	at java.util.Arrays.copyOf(Arrays.java:2271)
          	at java.util.Arrays.copyOf(Arrays.java:2271)
          	at java.util.Arrays.copyOf(Arrays.java:2271)
          Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
          Caused by: java.io.IOException: Unexpected reader termination
          Caused by: java.lang.OutOfMemoryError: Java heap space
          Exception in thread "Channel reader thread: BM4-Ubuntu_10.04_64" java.lang.OutOfMemoryError: Java heap space
          hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
          INFO: Attempting to reconnect BM4-Ubuntu_10.04_64
          java.lang.NullPointerException
          java.lang.OutOfMemoryError: Java heap space
          SEVERE: Unexpected error in channel BM4-Ubuntu_10.04_64
          WARNING: java.io.IOException: Broken pipe
          WARNING: java.io.IOException: Broken pipe
          WARNING: Publisher hudson.plugins.cloneworkspace.CloneWorkspacePublisher aborted due to exception
          WARNING: Publisher hudson.tasks.Mailer aborted due to exception
          

          Tomas Kopal added a comment - - edited I do not think its related, at least not directly. We have observed very similar issue. But it exhibits in 1.560 only, after downgrading to 1.559 everything works fine again (no change in plugins). If I understand correctly JENKINS-20769 , it was fixed in 1.554, but this really looks like regression in 1.560. Apr 23, 2014 4:34:45 PM hudson.slaves.SlaveComputer tryReconnect at hudson.FilePath.act(FilePath.java:893) at hudson.FilePath.act(FilePath.java:909) at hudson.FilePath.archive(FilePath.java:423) at hudson.FilePath.tar(FilePath.java:2047) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:740) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:740) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:710) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:710) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:659) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:659) at hudson.model.Build$BuildExecution.post2(Build.java:182) at hudson.model.Build$BuildExecution.post2(Build.java:182) at hudson.model.Executor.run(Executor.java:231) at hudson.model.Executor.run(Executor.java:231) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Run.execute(Run.java:1728) at hudson.model.Run.execute(Run.java:1728) at hudson.plugins.cloneworkspace.CloneWorkspacePublisher.perform(CloneWorkspacePublisher.java:178) at hudson.plugins.cloneworkspace.CloneWorkspacePublisher.snapshot(CloneWorkspacePublisher.java:231) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.Channel.call(Channel.java:738) at hudson.remoting.Channel.terminate(Channel.java:801) at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33) at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33) at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33) at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46) at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46) at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46) at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88) at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88) at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87) at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34) at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41) at hudson.remoting.Request.abort(Request.java:299) at hudson.remoting.Request.call(Request.java:174) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:76) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.tasks.Mailer.perform(Mailer.java:137) at hudson.tasks.MailSender.createFailureMail(MailSender.java:278) at hudson.tasks.MailSender.execute(MailSender.java:101) at hudson.tasks.MailSender.getMail(MailSender.java:153) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) at java.util.Arrays.copyOf(Arrays.java:2271) at java.util.Arrays.copyOf(Arrays.java:2271) at java.util.Arrays.copyOf(Arrays.java:2271) Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination Caused by: java.io.IOException: Unexpected reader termination Caused by: java.lang.OutOfMemoryError: Java heap space Exception in thread "Channel reader thread: BM4-Ubuntu_10.04_64" java.lang.OutOfMemoryError: Java heap space hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination INFO: Attempting to reconnect BM4-Ubuntu_10.04_64 java.lang.NullPointerException java.lang.OutOfMemoryError: Java heap space SEVERE: Unexpected error in channel BM4-Ubuntu_10.04_64 WARNING: java.io.IOException: Broken pipe WARNING: java.io.IOException: Broken pipe WARNING: Publisher hudson.plugins.cloneworkspace.CloneWorkspacePublisher aborted due to exception WARNING: Publisher hudson.tasks.Mailer aborted due to exception

          moa_arosii added a comment -

          We see the same problem when large files are copied to the slave in v. 1.560:

          It is also similar to this recently reported problem: http://jenkins-ci.361315.n4.nabble.com/Problems-with-memory-usage-on-1-560-td4699645.html

          It looks like the problem lies on the slave in this part:

          Caused by: java.lang.OutOfMemoryError: Java heap space
          	at java.util.Arrays.copyOf(Arrays.java:2271)
          	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
          	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
          	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
          	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87)}}
          

          If you look on the 'blame' for ChannelBuilder it looks like the usage of FlightRecorderInputStream was introduced in MakeTransport in line 251 20 days ago in commit "Merge branch 'master' into nio" (bdd486dbe6584c607bed0e4324abbd6eeddbb39e).

          It might be a good guess that this "recording' of content causes the problem when large amount of files are transferred. In our case it is close to 100 mb.

          moa_arosii added a comment - We see the same problem when large files are copied to the slave in v. 1.560: It is also similar to this recently reported problem: http://jenkins-ci.361315.n4.nabble.com/Problems-with-memory-usage-on-1-560-td4699645.html It looks like the problem lies on the slave in this part: Caused by: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2271) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87)}} If you look on the 'blame' for ChannelBuilder it looks like the usage of FlightRecorderInputStream was introduced in MakeTransport in line 251 20 days ago in commit "Merge branch 'master' into nio" ( bdd486dbe6584c607bed0e4324abbd6eeddbb39e ). It might be a good guess that this "recording' of content causes the problem when large amount of files are transferred. In our case it is close to 100 mb.

          Shane Turner added a comment -

          Someone should update the "Affects Version" field to indicate that this is an issue with 1.560.

          Shane Turner added a comment - Someone should update the "Affects Version" field to indicate that this is an issue with 1.560.

          Peter Van Biesen added a comment - - edited

          1.559 does not have the problem, 1.560 does. To reproduce, create a job with a shell build step and following code :

          dd if=/dev/urandom of=target.out bs=1M count=50
          for a in 1 2 3 4
          do
          cat target.out >> bigtarget.out
          done
          

          and in post-build actions add a "Archive the artifacts", file : bigtarget.out

          For every build on the same slave, the archived file will increase by 200 Mb.
          Effects :

          • the slave will disconnect
          • the master will not be able to reconnect to the slave UNLESS you choose "configure" on the node, click "save" and then relaunch the slave connection
          • the master will not stop in a normal fashion, only kill -9 will do the trick
          • the slave.jar will keep running on the client and needs to be killed too ( otherwise multiple will keep running )

          Peter Van Biesen added a comment - - edited 1.559 does not have the problem, 1.560 does. To reproduce, create a job with a shell build step and following code : dd if =/dev/urandom of=target.out bs=1M count=50 for a in 1 2 3 4 do cat target.out >> bigtarget.out done and in post-build actions add a "Archive the artifacts", file : bigtarget.out For every build on the same slave, the archived file will increase by 200 Mb. Effects : the slave will disconnect the master will not be able to reconnect to the slave UNLESS you choose "configure" on the node, click "save" and then relaunch the slave connection the master will not stop in a normal fashion, only kill -9 will do the trick the slave.jar will keep running on the client and needs to be killed too ( otherwise multiple will keep running )

          This issue actually makes jenkins 1.560 unusable I think. Shouldn't be a blocker ?

          Peter Van Biesen added a comment - This issue actually makes jenkins 1.560 unusable I think. Shouldn't be a blocker ?

          I tried Peter's script with 1.561, and I've run it 3 times. So far run 1 and 2 ran fine. I had to kill run 3, and the slave. Also, the mail-watcher-plugin alerted me to the slave disconnecting and reconnecting... So I've downgraded back to 1.559 again.

          Larry Shatzer, Jr. added a comment - I tried Peter's script with 1.561, and I've run it 3 times. So far run 1 and 2 ran fine. I had to kill run 3, and the slave. Also, the mail-watcher-plugin alerted me to the slave disconnecting and reconnecting... So I've downgraded back to 1.559 again.

          On my side since version 1.560 I am having a memory leak on Jenkins. On the master, looking at the size of the java process, it is 2.6G ( and increasing... 3.3G after 30 minutes ). On 1.559, the size is stable around 1.6G, 1.8G.

          Eric Frangulian added a comment - On my side since version 1.560 I am having a memory leak on Jenkins. On the master, looking at the size of the java process, it is 2.6G ( and increasing... 3.3G after 30 minutes ). On 1.559, the size is stable around 1.6G, 1.8G.

          Experienced slave disconnecting after builds. Found outofmemory in slave.log.
          Tried 1.561 same problem still persists.
          Rolled back to 1.559

          Corneil du Plessis added a comment - Experienced slave disconnecting after builds. Found outofmemory in slave.log. Tried 1.561 same problem still persists. Rolled back to 1.559

          Jesse Glick added a comment -

          Jesse Glick added a comment - https://github.com/jenkinsci/remoting/commit/c0e42ffe12c6bfb3f1e52257589f10dd896817e6 may have fixed this.

          Jesse Glick added a comment -

          Remoting 2.38 likely introduced the problem.

          Jesse Glick added a comment - Remoting 2.38 likely introduced the problem.

          Dell Green added a comment -

          Same problem here, as soon as I upgraded from 1.559 to 1.561 jobs that run daily and have been running for years started failing with out of memory errors on some slaves.

          FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
          hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
          	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
          	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
          	at hudson.remoting.Request.call(Request.java:174)
          	at hudson.remoting.Channel.call(Channel.java:739)
          	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:168)
          	at com.sun.proxy.$Proxy51.join(Unknown Source)
          	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:951)
          	at hudson.Launcher$ProcStarter.join(Launcher.java:362)
          	at hudson.tasks.Ant.perform(Ant.java:217)
          	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:745)
          	at hudson.model.Build$BuildExecution.build(Build.java:198)
          	at hudson.model.Build$BuildExecution.doRun(Build.java:159)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:518)
          	at hudson.model.Run.execute(Run.java:1709)
          	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          	at hudson.model.ResourceController.execute(ResourceController.java:88)
          	at hudson.model.Executor.run(Executor.java:231)
          Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
          	at hudson.remoting.Request.abort(Request.java:299)
          	at hudson.remoting.Channel.terminate(Channel.java:802)
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:76)
          Caused by: java.io.IOException: Unexpected reader termination
          	... 1 more
          Caused by: java.lang.OutOfMemoryError: Java heap space
          	at java.util.Arrays.copyOf(Arrays.java:2786)
          	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
          	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87)
          	at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46)
          	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88)
          	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33)
          	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
          
          
          
          

          Dell Green added a comment - Same problem here, as soon as I upgraded from 1.559 to 1.561 jobs that run daily and have been running for years started failing with out of memory errors on some slaves. FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41) at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34) at hudson.remoting.Request.call(Request.java:174) at hudson.remoting.Channel.call(Channel.java:739) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:168) at com.sun.proxy.$Proxy51.join(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:951) at hudson.Launcher$ProcStarter.join(Launcher.java:362) at hudson.tasks.Ant.perform(Ant.java:217) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:745) at hudson.model.Build$BuildExecution.build(Build.java:198) at hudson.model.Build$BuildExecution.doRun(Build.java:159) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:518) at hudson.model.Run.execute(Run.java:1709) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:231) Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination at hudson.remoting.Request.abort(Request.java:299) at hudson.remoting.Channel.terminate(Channel.java:802) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:76) Caused by: java.io.IOException: Unexpected reader termination ... 1 more Caused by: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2786) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87) at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46) at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88) at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

          Code changed in jenkins
          User: Jesse Glick
          Path:
          changelog.html
          pom.xml
          http://jenkins-ci.org/commit/jenkins/aebb876cc82d405022dd3317ece199017d3b7aa8
          Log:
          [FIXED JENKINS-22734] Integrating remoting 2.41 with attempted OOME fix.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html pom.xml http://jenkins-ci.org/commit/jenkins/aebb876cc82d405022dd3317ece199017d3b7aa8 Log: [FIXED JENKINS-22734] Integrating remoting 2.41 with attempted OOME fix.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #3351
          [FIXED JENKINS-22734] Integrating remoting 2.41 with attempted OOME fix. (Revision aebb876cc82d405022dd3317ece199017d3b7aa8)

          Result = SUCCESS
          Jesse Glick : aebb876cc82d405022dd3317ece199017d3b7aa8
          Files :

          • pom.xml
          • changelog.html

          dogfood added a comment - Integrated in jenkins_main_trunk #3351 [FIXED JENKINS-22734] Integrating remoting 2.41 with attempted OOME fix. (Revision aebb876cc82d405022dd3317ece199017d3b7aa8) Result = SUCCESS Jesse Glick : aebb876cc82d405022dd3317ece199017d3b7aa8 Files : pom.xml changelog.html

          1.562 doesn't have this fix yet ...

          Peter Van Biesen added a comment - 1.562 doesn't have this fix yet ...

          Marc Cloud added a comment -

          I've installed Jenkins ver. 1.563-SNAPSHOT - works for our installation.

          Got it from

           wget   "https://ci.jenkins-ci.org/view/Jenkins%20core/job/jenkins_rc_branch/lastSuccessfulBuild/artifact/war/target/jenkins.war"
          

          Marc Cloud added a comment - I've installed Jenkins ver. 1.563-SNAPSHOT - works for our installation. Got it from wget "https://ci.jenkins-ci.org/view/Jenkins%20core/job/jenkins_rc_branch/lastSuccessfulBuild/artifact/war/target/jenkins.war"

          Daniel Beck added a comment -

          This change is scheduled for 1.563 (due to how weird the weekly release process is set up, it takes at least 2-3 releases even for a serious bug to get fixed).

          Daniel Beck added a comment - This change is scheduled for 1.563 (due to how weird the weekly release process is set up, it takes at least 2-3 releases even for a serious bug to get fixed).

          Hi Marc, what other changes did you see after using this SNAPSHOT version? Is there any major change in job configuration? Or something else that I should be aware of before using this version?

          Thanks

          Sridevi Soundarraju added a comment - Hi Marc, what other changes did you see after using this SNAPSHOT version? Is there any major change in job configuration? Or something else that I should be aware of before using this version? Thanks

          Marc Cloud added a comment -

          No abnormalities so far, Sridevi.

          Marc Cloud added a comment - No abnormalities so far, Sridevi.

            kohsuke Kohsuke Kawaguchi
            jsirex jsirex
            Votes:
            21 Vote for this issue
            Watchers:
            42 Start watching this issue

              Created:
              Updated:
              Resolved: