Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-24895

An existing connection was forcibly closed by the remote host

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • core, remoting
    • Master: Widows Server 2008 R2, Jenkins 1.565.1
      Slave: See this issue on Win 7/ Windows Server 2008R2 slave.

      We have a testing which will run for several hours. We meet intermittent failure like below which terminated the testing and the job just failed.

      12:42:10 FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: An existing connection was forcibly closed by the remote host
      12:42:10 hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: An existing connection was forcibly closed by the remote host
      12:42:10 at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
      12:42:10 at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
      12:42:10 at hudson.remoting.Request.call(Request.java:174)
      12:42:10 at hudson.remoting.Channel.call(Channel.java:739)
      12:42:10 at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:168)
      12:42:10 at com.sun.proxy.$Proxy61.join(Unknown Source)
      12:42:10 at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:956)
      12:42:10 at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137)
      12:42:10 at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)
      12:42:10 at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
      12:42:10 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
      12:42:10 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:756)
      12:42:10 at hudson.model.Build$BuildExecution.build(Build.java:198)
      12:42:10 at hudson.model.Build$BuildExecution.doRun(Build.java:159)
      12:42:10 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
      12:42:10 at hudson.model.Run.execute(Run.java:1706)
      12:42:10 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      12:42:10 at hudson.model.ResourceController.execute(ResourceController.java:88)
      12:42:10 at hudson.model.Executor.run(Executor.java:232)
      12:42:10 Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: An existing connection was forcibly closed by the remote host
      12:42:10 at hudson.remoting.Request.abort(Request.java:299)
      12:42:10 at hudson.remoting.Channel.terminate(Channel.java:802)
      12:42:10 at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
      12:42:10 Caused by: java.io.IOException: An existing connection was forcibly closed by the remote host
      12:42:10 at sun.nio.ch.SocketDispatcher.read0(Native Method)
      12:42:10 at sun.nio.ch.SocketDispatcher.read(Unknown Source)
      12:42:10 at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
      12:42:10 at sun.nio.ch.IOUtil.read(Unknown Source)
      12:42:10 at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
      12:42:10 at hudson.remoting.SocketChannelStream$1.read(SocketChannelStream.java:33)
      12:42:10 at sun.nio.ch.ChannelInputStream.read(Unknown Source)
      12:42:10 at sun.nio.ch.ChannelInputStream.read(Unknown Source)
      12:42:10 at sun.nio.ch.ChannelInputStream.read(Unknown Source)
      12:42:10 at java.io.InputStream.read(Unknown Source)
      12:42:10 at sun.nio.ch.ChannelInputStream.read(Unknown Source)
      12:42:10 at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
      12:42:10 at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
      12:42:10 at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
      12:42:10 at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
      12:42:10 at java.io.ObjectInputStream.readObject0(Unknown Source)
      12:42:10 at java.io.ObjectInputStream.readObject(Unknown Source)
      12:42:10 at hudson.remoting.Command.readFrom(Command.java:92)
      12:42:10 at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:70)
      12:42:10 at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

          [JENKINS-24895] An existing connection was forcibly closed by the remote host

          sharon xia created issue -
          Daniel Beck made changes -
          Component/s New: remoting [ 15489 ]
          Labels Original: connection New: connection remoting

          Daniel Beck added a comment -

          Given this is an exception originating from Windows networking, did you try Google?

          E.g. http://blogs.msdn.com/b/sql_protocols/archive/2008/04/08/understanding-connection-forcibly-closed-by-remote-host-errors-caused-by-toe-chimney.aspx explains how to test for broken hardware.

          http://social.msdn.microsoft.com/Forums/en-US/497175cf-a548-476e-ad19-feb938873032/exception-an-existing-connection-was-forcibly-closed-by-the-remote-host?forum=netfxnetcom indicates timeouts can cause this, which may be consistent with your bad line across the world (JENKINS-23868).

          Daniel Beck added a comment - Given this is an exception originating from Windows networking, did you try Google? E.g. http://blogs.msdn.com/b/sql_protocols/archive/2008/04/08/understanding-connection-forcibly-closed-by-remote-host-errors-caused-by-toe-chimney.aspx explains how to test for broken hardware. http://social.msdn.microsoft.com/Forums/en-US/497175cf-a548-476e-ad19-feb938873032/exception-an-existing-connection-was-forcibly-closed-by-the-remote-host?forum=netfxnetcom indicates timeouts can cause this, which may be consistent with your bad line across the world ( JENKINS-23868 ).

          Henning Seljenes added a comment -

          I also ran into this and it seems Symantec Endpoint Protection was killing the socket when virus definitions were updated. Would be nice if the Node just reconnected instead of crash.

          Henning Seljenes added a comment - I also ran into this and it seems Symantec Endpoint Protection was killing the socket when virus definitions were updated. Would be nice if the Node just reconnected instead of crash.

          Andy Waterson added a comment -

          Same problem here. Unfortunately, I have no control over when SEP updates are done. The end result is randomly disconnecting jobs make the build/test process unreliable.

          Andy Waterson added a comment - Same problem here. Unfortunately, I have no control over when SEP updates are done. The end result is randomly disconnecting jobs make the build/test process unreliable.

          Daniel Beck added a comment -

          Is anyone experiencing this problem without running a tool that forces disconnecting established network connections? Sharon?

          Daniel Beck added a comment - Is anyone experiencing this problem without running a tool that forces disconnecting established network connections? Sharon?

          G. Ancona added a comment - - edited

          Same problem without Symantec. I even created a "ping batch" to verify any eventual network default but the connection falls even if the network is up and running.
          Vers. 1.577

          G. Ancona added a comment - - edited Same problem without Symantec. I even created a "ping batch" to verify any eventual network default but the connection falls even if the network is up and running. Vers. 1.577

          Daniel Beck added a comment -

          Ping doesn't have a persistent connection. Something like telnet to a service that doesn't force connections closed due to inactivity may be better.

          Daniel Beck added a comment - Ping doesn't have a persistent connection. Something like telnet to a service that doesn't force connections closed due to inactivity may be better.

          t h added a comment -

          We noticed that the same issue occurs on a slave that's running as a service.
          same job works OK as slave-agent.
          Still, this is a very urgent issue for us.

          t h added a comment - We noticed that the same issue occurs on a slave that's running as a service. same job works OK as slave-agent. Still, this is a very urgent issue for us.

          Andy Waterson added a comment -

          FYI, in my case, it was discovered that updating to the latest SEP release available to us (12.1.4100.xxxx) from 11.something much improved the issue.

          Andy Waterson added a comment - FYI, in my case, it was discovered that updating to the latest SEP release available to us (12.1.4100.xxxx) from 11.something much improved the issue.

            Unassigned Unassigned
            sharon_xia sharon xia
            Votes:
            8 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated: