-
Bug
-
Resolution: Unresolved
-
Major
-
We have a Linux master with 77 windows 2012R2 slaves and a few linux node's.
On this moment we have Jenkins 2.9. In the near future we will update Jenkins to the latest version and split up the environment into DEV, ACC and PRD. So than we have three masters with each approx. 25 windows 2012R2 nodes (slaves)
Under Manage Jenkins --> Globaal security we have enabled security and "TCP port for JNLP agents" is Fixed with a port number.We have a Linux master with 77 windows 2012R2 slaves and a few linux node's. On this moment we have Jenkins 2.9. In the near future we will update Jenkins to the latest version and split up the environment into DEV, ACC and PRD. So than we have three masters with each approx. 25 windows 2012R2 nodes (slaves) Under Manage Jenkins --> Globaal security we have enabled security and "TCP port for JNLP agents" is Fixed with a port number.
The main job is started, which is starting a some other (sub)jobs. Mostly at the same sub-job, after the first four jobs has been executed without any problem, a job is started. When that job is started or if this job runs one or three minutes we get the issue "Agent went offline". When it runs like say 4 min. than it runs to the end without any issue! That job runs for 1,5 hour or even more!
After it went wrong with "Agent went offliner" or with "Channel already closed" we start the main job again and than it works always without a issue!!!!????? WHY?????
Analyzing this:
- Windows event viewer (see screenshot), I can see that the Jenkinsslave-C_Jenkins is "normally" stopped. But by whom? The Job is not doing this or at least so far as I can see the Job is not doing this and should not do this. See my screenshot. In this screenshot it's a few seconds later the service is stopped. But I have other circumstances and then it was in the exact same second!
- Jenkins logging "Jenkins-slave.err". There I see issue's with JNLP connection but on this moment I don't know what to do about it! What do I have to change or were can I find documentation.
INFO: Trying protocol: JNLP3-connect
Oct 21, 2019 7:01:43 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Server didn't accept the handshake: Unknown protocol:Protocol:JNLP3-connect
Oct 21, 2019 7:01:43 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to <ip adress>:<fixed port adress>
Oct 21, 2019 7:01:43 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP2-connect
Oct 21, 2019 7:01:43 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Oct 21, 2019 7:12:21 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Oct 21, 2019 7:12:31 PM jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$2$1 onReconnect
INFO: Restarting agent via jenkins.slaves.restarter.WinswSlaveRestarter@48eae634
Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main createEngine
INFO: Setting up slave: nodename
Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among http://<ip adres>/jenkins/
Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to <ip adress>:<fixed port adress>
Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP3-connect
Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Server didn't accept the handshake: Unknown protocol:Protocol:JNLP3-connect
Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to <ip adress>:<fixed port adress>
Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP2-connect
Oct 21, 2019 7:12:34 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main createEngine
INFO: Setting up slave: nodename
Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among http://<ip adress>jenkins/
Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to <ip adress>:<fixed port adress>
Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP3-connect
Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Server didn't accept the handshake: Unknown protocol:Protocol:JNLP3-connect
Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to <ip adress>:<fixed port adress>
Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP2-connect
Oct 21, 2019 7:13:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Failed to load native POSIX impl; falling back on Java impl. Unsupported OS.
Oct 21, 2019 10:51:41 PM hudson.plugins.tmpcleaner.TmpCleanTask call
INFO: end TmpCleanTask
Oct 22, 2019 4:51:41 AM hudson.plugins.tmpcleaner.TmpCleanTask call
INFO: end TmpCleanTask
Oct 22, 2019 10:51:41 AM hudson.plugins.tmpcleaner.TmpCleanTask call
INFO: end TmpCleanTask
- The console log Agent went offline at the same time the JNLP is TERMINATED in Jenkins-slave.err file!:
19:12:10 2019-10-21
19:12:10.2303|INFO|Deploy.Program|File upload successfully done: xxxx.json
19:12:10 2019-10-21 19:12:10.2303|INFO|Deploy.Program|File upload successfully done: xxxx.json
19:12:10 2019-10-21 19:12:10.3153|INFO|Deploy.Program|File upload successfully done: yyy.json
19:12:21 Agent went offline during the build19:12:21 ERROR: Connection was broken: java.io.IOException: Connection aborted: org.jenkinsci.remoting.nio.NioChannelHub$MonoNioTransport@2838043[name=<node_name>]
19:12:21 at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.abort(NioChannelHub.java:208)
19:12:21 at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:629)19:12:21 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
19:12:21 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)19:12:21 at java.util.concurrent.FutureTask.run(FutureTask.java:266)19:12:21 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
19:12:21 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
19:12:21 at java.lang.Thread.run(Thread.java:745)
19:12:21 Caused by: java.io.IOException: Connection timed out19:12:21 at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
19:12:21 at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
19:12:21 at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
19:12:21 at sun.nio.ch.IOUtil.read(IOUtil.java:197)19:12:21 at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
19:12:21 at org.jenkinsci.remoting.nio.FifoBuffer$Pointer.receive(FifoBuffer.java:137)
19:12:21 at org.jenkinsci.remoting.nio.FifoBuffer.receive(FifoBuffer.java:310)
19:12:21 at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:561)
19:12:21 ... 6 more19:12:21
- Screenshot configuring the node
I hope that someone can help me out this issue. When go DuckDuckGo than I find out I'm not the only one who have this issue but I can't find a solution.
Regards,
Nico