I am sure it is not a network issue as I tried pinging the network from system start-up till the problem occurs.

Why are you so sure about this? Pinging does not keep a connection open.

Hi Daniel,

I will rephrase - I am sure that pinging the master from the slave is fine as I initially suspect some kind of network failure.
Do you have some suggestion that can help me debug or resolve my problem?

SQL server that I have installed on this slave, has some TCP settings. By default it listens to 1433. Could that be some kind of a problem? Any ideas? SQL killing some connection?

The post here: http://www.redmountainsw.com/wordpress/864/tcp-provider-error-0-an-existing-connection-was-forcibly-closed-by-the-remote-host/ provides a lot of information, but my error is not on SQL side and I don’t want to turn connection pooling off…

On my slave where I see this error in the server management console I see this:

TCP/IP failed to establish an outgoing connection because the selected local endpoint was recently used to connect to the same remote endpoint. This error typically occurs when outgoing connections are opened and closed at a high rate, causing all available local ports to be used and forcing TCP/IP to reuse a local port for an outgoing connection. To minimize the risk of data corruption, the TCP/IP standard requires a minimum time period to elapse between successive connections from a given local endpoint to a given remote endpoint.

I guess Jenkins is abusing the connection somehow Any ideas how to resolve it?

We may have a similar problem. We start a windows vm and connect it via jnlp to the master, then it starts some work and after some time, usually 3-5h the slave gets disconnected unintentionally:

java.io.IOException: Connection aborted: org.jenkinsci.remoting.nio.NioChannelHub$MonoNioTransport@7aa84678[name=QF-WinXP]
	at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.abort(NioChannelHub.java:211)
	at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:631)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:197)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
	at org.jenkinsci.remoting.nio.FifoBuffer$Pointer.receive(FifoBuffer.java:136)
	at org.jenkinsci.remoting.nio.FifoBuffer.receive(FifoBuffer.java:306)
	at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:564)

We have only one windows machine and this machine shows this behavior. All the linux slaves are connected constantly without any problems.

UPDATE: It turned out my issue was related to the recent switch to https (so probably not related to this one).
After providing Java trustStore with RootCA and Intermediate CA - issue resolved.

====

Same here - although reason might be different. Slave started on WIndows 7, on an AD account, as a service.
Drops almost immediately.

java.io.IOException: Connection aborted: org.jenkinsci.remoting.nio.NioChannelHub$MonoNioTransport@240c0b4e[name=ST_windows_slave]
	at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.abort(NioChannelHub.java:208)
	at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:628)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:197)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
	at org.jenkinsci.remoting.nio.FifoBuffer$Pointer.receive(FifoBuffer.java:136)
	at org.jenkinsci.remoting.nio.FifoBuffer.receive(FifoBuffer.java:306)
	at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:561)
	... 6 more

Found some more in slave's log. My master instance is using https and although the certificate
is installed in Windows and browsers recognize it, slave complains about it - as below.
Any ideas would be welcome.

Exception in thread "main" java.io.IOException: Failed to validate a server certificate. If you are using a self-signed certificate, you can use the -noCertificateCheck option to bypass this check.
	at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:327)
	at hudson.remoting.Launcher.run(Launcher.java:219)
	at hudson.remoting.Launcher.main(Launcher.java:192)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alerts.getSSLException(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source)
	at sun.security.ssl.Handshaker.fatalSE(Unknown Source)
	at sun.security.ssl.Handshaker.fatalSE(Unknown Source)
	at sun.security.ssl.ClientHandshaker.serverCertificate(Unknown Source)
	at sun.security.ssl.ClientHandshaker.processMessage(Unknown Source)
	at sun.security.ssl.Handshaker.processLoop(Unknown Source)
	at sun.security.ssl.Handshaker.process_record(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
	at sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(Unknown Source)
	at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:269)
	... 2 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.validator.PKIXValidator.doBuild(Unknown Source)
	at sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
	at sun.security.validator.Validator.validate(Unknown Source)
	at sun.security.ssl.X509TrustManagerImpl.validate(Unknown Source)
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
	... 14 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source)
	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)
	at java.security.cert.CertPathBuilder.build(Unknown Source)
	... 20 more

I have no actual solution for this problem but after doing many different experiments we managed to configure our windows slaves (win server 2012 R2 standart) with a scheduled task (Jenkins Slave.xml attached here) and so far they are quite stable. A prerequisite is to have the test user automatically logged at OS boot.

I have started to have the same issues with the same configuration. I am using the LTS version of Jenkins 1.609.1

Still an issue in Jenkins ver. 1.641

I have the same issue:
used: jenkins 2.09 (bitbnami stack)
master: win 7 64 bit, java 8.91 32 bit
slave: win 2008R2 64 bit, java 8.91 32 bit, 64 bit

I had the same issue, also using the Windows' task scheduler to attach the slave.

For me, the issue seems to be due to the "Stop task if it runs longer than x time" option being activated and set to 3 days by default.
Windows kills the task after 3 days, stopping the connection between Jenkins and the slave.
This option is set in the properties of the task, inside Windows' task scheduler.

Hope this can help someone, I was blocked by this problem for a very long time myself.

Hello,
We are also getting this exception with Jenkins ver. 2.38 inconsistently, has anyone found any working solution for this issue? Found https://issues.jenkins-ci.org/browse/JENKINS-27251 where i see a comment saying "It was likely SocketTimeoutException on the slave side in this case (fixed in 2.60 & released in 2.9)." We see "SocketTimeoutException" on slave side. So, does upgrading to 2.9 will solve this issue?

Thanks in advance!

Hi,

my working solution is the following:

I use windows for master and for the slave as well.

On the slave I have an administrator user with autologin, when the user logged in it waits 1 minute, then starts jenkins slave with java webstart with the jnlp file:
javaws.lnk c:\CI\slave-agent.jnlp
(javaws.lnk is created to be on path)

I use java 1.7.1 because the javaws needs 1.7, but 1.7.89 has a security enhancement, and it does not allow to start the javaws automatically, it needs manual confirmation. (I did not check the versions between 1.7.1 and 1.7.89) 1.7.1 allows the starting.

This way the connection is stable.

BR,
George

We're also seeing much the same thing - the Jenkins server is saying that the (JNLP) slave dropped the connection.

We've been looking into this and we've found that (at least in our system) the problem is coming from the Slave's Windows OS itself.  On the slave, we are seeing the following error logged:

java.io.IOException: An established connection was aborted by the software in your host machine.
    at sun.nio.ch.SocketDispatcher.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:55)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:235)
    at sun.nio.ch.IOUtil.read(IOUtil.java:209)
    at sun.nio.ch.SockerChannelImpl.read(SocketChannelImpl.java:409)
    at hudson.remoting.SocketChannelStream$1.read(SocketChannelStream.java:35)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:77)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:121)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:115)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
    at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:86)
    at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72)
    at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103)
    at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)

Some searching later, we identified that "An established connection was aborted by the software in your host machine" is the Windows socket error WSACONNABORTED (10053).

Much investigation later, we've found that various Windows services running on the Slave's OS are logging (in the Windows event log) that they're restarting at the exact same time, and (more interestingly!) we've also seen that the DHCP lease was renewed at the exact time that the slave reported the connection had died.

• Note: Windows handles TCP connections differently to other operating systems.  If Windows decides that the physical network layer has gone down (however briefly), it actively kills (with a WSACONNABORTED) all TCP connections that were being routed over that network, thus turning a transient outage (that normal TCP retransmissions would handle so that the user doesn't even see the problem) into an application-level outage (as the TCP connection closes, forcing the application to deal with it, usually by reporting that the connection has failed and it's "game over").  This is why a brief network outage that should cause no operational impact will result in a flurry of service restarts as they all try to handle the connection losses.  Windows' (mis?)handling of this scenario has been like this for so long that I doubt Microsoft would be willing to change it now.

So I think that something, somewhere deep within Windows, is making Windows believe that it has lost the network layer.  Problem is, I don't (yet) know what's doing it - all I see is a lot of symptoms of it doing it, not the root cause.

"ipconfig /release && ipconfig /renew" will cause this (even if you immediately get the same IP address back), as will unplugging/replugging your Cat5 cable or disconnecting/reconnecting your WiFi connection, or power-saving on your NIC, or reconfiguring your NIC and thus forcing a reload of the driver, or...

We've yet to find the root cause in our setup, but investigations are ongoing.

Still seems like a generic infrastructure issue symptom, which may be caused by Remoting defects in some cases. Investigation from pjdarton is definitely correct, but I'd guess the most of reporters/voters here see quite another issue. Maybe it should be reported as a separate ticket since it may be worked around somehow.

I do not anticipate to have any time for Remoting maintenance anytime soon, so I am going to unassign this issue.

FYI we got to the bottom of why our Windows slaves were disconnecting - it would appear that the Windows DHCP Client is incompatible with the Windows Time Service.

Our slaves were VMs created within OpenStack, and what we were seeing was a failure to renew the DHCP lease correctly. When OpenStack detects that the guest OS has failed to renew the DHCP lease on time, it (briefly) drops the network link in order to prompt a lease renewal. However this causes Windows to panic and kill all TCP connections (due to the way Windows mishandles network layers).
It seems that the DHCP client is not calculating the renewal time in a manner that's independent of the system's idea of "real time", and so it all goes wrong when the date/time gets changed (by the Windows Time Service), triggering OpenStack to bounce the physical layer, which Windows cascades in to an application-layer network outage killing the TCP connection that the slave relies on.

We "fixed" this by forcing our slaves to:

1. run "w32tm /resync" until they'd got the time synchronized,
2. turn off the Windows Time Service entirely,
3. ipconfig /release /renew to update the DHCP lease time
4. start the Jenkins JNLP slave process

This ensured that Windows would not update its clock while the slave's TCP connection was live, meaning that we weren't affected by the DHCP client's inability to keep the network alive after clock changes.
Since doing that we've not had any further problems of this nature (and we're quite pleased with that!)

Note: I've also seen Windows 10 report unpredictable (and incorrect) DHCP lease renewal times on other (non-OpenStack) machines - it lies.

After i did mass refactoring on my INFRA i do not see this type of errors anymore, except when the network in the lab goes high wire.