My only Jenkins slave dies every day almost at the same time giving below error :
Jul 29, 2016 3:02:42 AM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1469775521795 hasn't completed by 1469775762018
at hudson.remoting.PingThread.ping(PingThread.java:126)
at hudson.remoting.PingThread.run(PingThread.java:85)
Jul 29, 2016 3:02:46 AM hudson.remoting.SynchronousCommandTransport$ReaderThread run
SEVERE: I/O error in channel channel
java.net.SocketException: Socket closed
at java.net.SocketInputStream.read(SocketInputStream.java:203)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
Jul 29, 2016 3:02:47 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Jul 29, 2016 3:03:00 AM jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$2$1 onReconnect
INFO: Restarting agent via jenkins.slaves.restarter.UnixSlaveRestarter@e374354
Jul 29, 2016 3:03:39 AM hudson.remoting.jnlp.Main createEngine
INFO: Trying protocol: JNLP2-connect
Jul 29, 2016 3:03:41 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Server didn't accept the handshake: ICI-Internal_1 is already connected to this master. Rejecting this connection.
AT the same time, Jenkins Server (master) exhibits following logs:
Jul 29, 2016 3:02:37 AM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel ICI-Internal_1.
java.util.concurrent.TimeoutException: Ping started at 1469775517149 hasn't completed by 1469775757149
at hudson.remoting.PingThread.ping(PingThread.java:126)
at hudson.remoting.PingThread.run(PingThread.java:85)
Jul 29, 2016 3:03:40 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted connection #2 from /12.170.11.58:46412
Jul 29, 2016 3:03:41 AM org.jenkinsci.remoting.engine.JnlpServerHandshake error
WARNING: TCP agent connection handler #2 with /12.170.11.58:46412 is aborted: ICI-Internal_1 is already connected to this master. Rejecting this connection.
Jul 29, 2016 3:03:41 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted connection #3 from /12.170.11.58:46413
Jul 29, 2016 3:03:41 AM org.jenkinsci.remoting.engine.JnlpServerHandshake error
WARNING: TCP agent connection handler #3 with /12.170.11.58:46413 is aborted: ICI-Internal_1 is already connected to this master. Rejecting this connection.
I only understood that Client/Server both are not able to connect to each other due to some reason and failed. However, it works for rest of the day when it is started manually after above error. There is no firewall/selinux running on any of the server.
-Amit