Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22816

SocketException on copy artifact after upgrad

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Blocker
    • Resolution: Duplicate
    • copyartifact-plugin
    • None
    • Windows Server 2008
      Jenkins ver. 1.561
      Copy Artifact Plugin 1.30

    Description

      Recently after upgrading both Jenkins and several plugins we very often get failed matrix builds with following exception. This totaly blocks our CI activities.
      Let me know if you need more info.

      FATAL: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
      hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
      	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
      	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
      	at hudson.remoting.Request.call(Request.java:174)
      	at hudson.remoting.Channel.call(Channel.java:739)
      	at hudson.FilePath.act(FilePath.java:909)
      	at hudson.FilePath.act(FilePath.java:893)
      	at hudson.FilePath.touch(FilePath.java:1355)
      	at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:90)
      	at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyAll(FingerprintingCopyMethod.java:68)
      	at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:368)
      	at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:306)
      	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:745)
      	at hudson.model.Build$BuildExecution.build(Build.java:198)
      	at hudson.model.Build$BuildExecution.doRun(Build.java:159)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:518)
      	at hudson.model.Run.execute(Run.java:1709)
      	at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
      	at hudson.model.ResourceController.execute(ResourceController.java:88)
      	at hudson.model.Executor.run(Executor.java:231)
      Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
      	at hudson.remoting.Request.abort(Request.java:299)
      	at hudson.remoting.Channel.terminate(Channel.java:802)
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
      Caused by: java.net.SocketException: Connection reset
      	at java.net.SocketInputStream.read(Unknown Source)
      	at java.io.FilterInputStream.read(Unknown Source)
      	at java.io.BufferedInputStream.fill(Unknown Source)
      	at java.io.BufferedInputStream.read(Unknown Source)
      	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
      	at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:67)
      	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:93)
      	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33)
      	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
      

      Attachments

        Issue Links

          Activity

            danielbeck Daniel Beck added a comment -

            Copy Artifact has a transfer mode that basically copies all files in one step, so chances are it accumulates the data of all files into the buffer.

            It's suspicious that FlightRecorderInputStream is in the stack trace.

            My suggestion is upgrade to 1.563 ASAP (maybe even the current RC if urgent), or downgrade to 1.559, and see whether the issue is gone. If so, this is a duplicate.

            danielbeck Daniel Beck added a comment - Copy Artifact has a transfer mode that basically copies all files in one step, so chances are it accumulates the data of all files into the buffer. It's suspicious that FlightRecorderInputStream is in the stack trace. My suggestion is upgrade to 1.563 ASAP (maybe even the current RC if urgent), or downgrade to 1.559, and see whether the issue is gone. If so, this is a duplicate.
            ikedam ikedam added a comment -

            > * Hmmm... good question, anyway I can determine that post upgrade? I've upgraded to 1.29 of copy artifact and then almost immediately to 1.30 since it appeared then so I don't have "downgrade to X.YY" shown with the original version that I've started
            > * had issues with some other plugin and one led to another
            > ** unfortunately I went the wrong path

            As there are cases you cannot downgrading plugins easily (e.g. the configuration fields are changed), you would better determine you need a new version BEFORE upgrading.
            Anyway, this problem seems caused by Jenkins core as Daniel points, you should switch your Jenkins core to the LTS versions. It might also resolve problems with other plugins.

            > maybe copy artifact plugin could be extended with some retry procedure in case of file copy issues?

            That feature could not help you as the slave crashes and the build cannot continue anymore.
            JobRequeue-Plugin might help you. (As I haven't tried that plugin, I recommend you test that plugin before introducing to the production environment.)

            > seems that slave node is dying (not 100% sure if related and what happens first - slave down or copy problem), here's the entry from log file

            This log file is the one stored in the master, isn't it?
            (8mha:AAAAWB+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT... is an annotation appended by Jenkins)
            Please check the log files in slave nodes. They are placed in JENKINS_HOME you specified in node configuration pages.

            ikedam ikedam added a comment - > * Hmmm... good question, anyway I can determine that post upgrade? I've upgraded to 1.29 of copy artifact and then almost immediately to 1.30 since it appeared then so I don't have "downgrade to X.YY" shown with the original version that I've started > * had issues with some other plugin and one led to another > ** unfortunately I went the wrong path As there are cases you cannot downgrading plugins easily (e.g. the configuration fields are changed), you would better determine you need a new version BEFORE upgrading. Anyway, this problem seems caused by Jenkins core as Daniel points, you should switch your Jenkins core to the LTS versions. It might also resolve problems with other plugins. > maybe copy artifact plugin could be extended with some retry procedure in case of file copy issues? That feature could not help you as the slave crashes and the build cannot continue anymore. JobRequeue-Plugin might help you. (As I haven't tried that plugin, I recommend you test that plugin before introducing to the production environment.) > seems that slave node is dying (not 100% sure if related and what happens first - slave down or copy problem), here's the entry from log file This log file is the one stored in the master, isn't it? (8mha:AAAAWB+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT... is an annotation appended by Jenkins) Please check the log files in slave nodes. They are placed in JENKINS_HOME you specified in node configuration pages.

            Thanks for explaining all this. You seem to be right about that OutOfMemory issue

            SEVERE: Unexpected error in channel channel
            java.lang.OutOfMemoryError: Requested array size exceeds VM limit
            	at java.util.Arrays.copyOf(Unknown Source)
            	at java.io.ByteArrayOutputStream.grow(Unknown Source)
            	at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source)
            	at java.io.ByteArrayOutputStream.write(Unknown Source)
            	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87)
            	at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46)
            	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88)
            	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33)
            	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
            	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            
            Exception in thread "Channel reader thread: channel" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
            	at java.util.Arrays.copyOf(Unknown Source)
            	at java.io.ByteArrayOutputStream.grow(Unknown Source)
            	at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source)
            	at java.io.ByteArrayOutputStream.write(Unknown Source)
            	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87)
            	at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46)
            	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88)
            	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33)
            	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
            	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            channel stopped
            channel started
            
            swiniak Andrzej Pasterczyk added a comment - Thanks for explaining all this. You seem to be right about that OutOfMemory issue SEVERE: Unexpected error in channel channel java.lang.OutOfMemoryError: Requested array size exceeds VM limit at java.util.Arrays.copyOf(Unknown Source) at java.io.ByteArrayOutputStream.grow(Unknown Source) at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source) at java.io.ByteArrayOutputStream.write(Unknown Source) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87) at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46) at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88) at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) Exception in thread "Channel reader thread: channel" java.lang.OutOfMemoryError: Requested array size exceeds VM limit at java.util.Arrays.copyOf(Unknown Source) at java.io.ByteArrayOutputStream.grow(Unknown Source) at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source) at java.io.ByteArrayOutputStream.write(Unknown Source) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87) at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46) at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88) at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) channel stopped channel started
            danielbeck Daniel Beck added a comment -

            General agreement seems to be that this duplicates JENKINS-22734.

            If in 1.563 onward JENKINS-22734 is confirmed fix and this still occurs, feel free to reopen (with new stack traces etc.).

            danielbeck Daniel Beck added a comment - General agreement seems to be that this duplicates JENKINS-22734 . If in 1.563 onward JENKINS-22734 is confirmed fix and this still occurs, feel free to reopen (with new stack traces etc.).

            Fix in 1.563 confirmed. Thanks

            swiniak Andrzej Pasterczyk added a comment - Fix in 1.563 confirmed. Thanks

            People

              Unassigned Unassigned
              swiniak Andrzej Pasterczyk
              Votes:
              2 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: