Failed connections between JNLP agent and master on Kubernetes

This issue is archived. You can view it, but you can't modify it. Learn more

XMLWordPrintable

      I've set up Jenkins to run on a k8s cluster.

      I'm using the kubernetes plugin to create agent in the cluster.

      I have a discovery service available atĀ jenkins-discovery.jenkins.svc.cluster.local:50000.

      I've set up the tunnel property toĀ jenkins-discovery.jenkins.svc.cluster.local:50000

      When I start a job, the agent seems to be able to connect but then fails for some unknown reasons.

      Ā 

      The agent logs:

      INFO: Locating server among [http://jenkins-internal.jenkins.svc.cluster.local:8080/]
Feb 12, 2018 4:08:37 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Feb 12, 2018 4:08:37 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
Agent address: jenkins-discovery.jenkins.svc.cluster.local
Agent port: 50000
Identity: a1:e6:e0:d2:ad:30:2e:11:47:4f:0f:95:41:d3:d0:a4
Feb 12, 2018 4:08:37 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Feb 12, 2018 4:08:37 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-discovery.jenkins.svc.cluster.local:50000
Feb 12, 2018 4:08:37 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Feb 12, 2018 4:08:37 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: a1:e6:e0:d2:ad:30:2e:11:47:4f:0f:95:41:d3:d0:a4
Feb 12, 2018 4:08:39 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Feb 12, 2018 4:08:41 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated

      It repeats this 4 times and then I seeĀ 

      WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@71b21b06
hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@4d5ba314:JNLP4-connect connection to jenkins-discovery.jenkins.svc.cluster.local/100.71.204.63:50000": channel is already closed
at hudson.remoting.Channel.send(Channel.java:672)
at hudson.remoting.Request$2.run(Request.java:371)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine$1$1.run(Engine.java:94)
at java.lang.Thread.run(Thread.java:748)
Caused by: hudson.remoting.Channel$OrderlyShutdown: Command close created at
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1226)
at hudson.remoting.Channel$1.handle(Channel.java:561)
at hudson.remoting.AbstractByteBufferCommandTransport.processCommand(AbstractByteBufferCommandTransport.java:202)
at hudson.remoting.AbstractByteBufferCommandTransport.receive(AbstractByteBufferCommandTransport.java:188)
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onRead(ChannelApplicationLayer.java:186)
at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecv(ApplicationLayer.java:207)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:369)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669)
at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$2200(BIONetworkLayer.java:48)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:283)
... 4 more
Caused by: Command close created at
at hudson.remoting.Command.<init>(Command.java:62)
at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1220)
at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1218)
at hudson.remoting.Channel.close(Channel.java:1391)
at hudson.remoting.Channel.close(Channel.java:1358)
at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:745)
at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:98)
at hudson.slaves.SlaveComputer$2.onClosed(SlaveComputer.java:533)
at hudson.remoting.Channel.terminate(Channel.java:1014)
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208)
at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800)
at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:313)
at hudson.remoting.Channel.close(Channel.java:1405)
at hudson.remoting.Channel.close(Channel.java:1358)
at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:603)
at jenkins.slaves.DefaultJnlpSlaveReceiver.afterChannel(DefaultJnlpSlaveReceiver.java:168)
at org.jenkinsci.remoting.engine.JnlpConnectionState$4.invoke(JnlpConnectionState.java:421)
at org.jenkinsci.remoting.engine.JnlpConnectionState.fire(JnlpConnectionState.java:312)
at org.jenkinsci.remoting.engine.JnlpConnectionState.fireAfterChannel(JnlpConnectionState.java:418)
at org.jenkinsci.remoting.engine.JnlpProtocol4Handler$Handler$1.run(JnlpProtocol4Handler.java:334)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more

      In the master I see

      INFO: Created Pod: slave-with-docker-g2k77 in namespace jenkins
Feb 12, 2018 4:08:36 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for Pod to be scheduled (0/100): slave-with-docker-g2k77
Feb 12, 2018 4:08:37 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #482 from /100.96.10.76:41026
Feb 12, 2018 4:08:38 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #483 from /100.96.10.76:41036
Feb 12, 2018 4:08:41 PM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
WARNING: Computer.threadPoolForRemoting [#553] for slave-with-docker-g2k77 terminated
java.nio.channels.ClosedChannelException
	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208)
	at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800)
	at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:313)
	at hudson.remoting.Channel.close(Channel.java:1405)
	at hudson.remoting.Channel.close(Channel.java:1358)
	at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:603)
	at jenkins.slaves.DefaultJnlpSlaveReceiver.afterChannel(DefaultJnlpSlaveReceiver.java:168)
	at org.jenkinsci.remoting.engine.JnlpConnectionState$4.invoke(JnlpConnectionState.java:421)
	at org.jenkinsci.remoting.engine.JnlpConnectionState.fire(JnlpConnectionState.java:312)
	at org.jenkinsci.remoting.engine.JnlpConnectionState.fireAfterChannel(JnlpConnectionState.java:418)
	at org.jenkinsci.remoting.engine.JnlpProtocol4Handler$Handler$1.run(JnlpProtocol4Handler.java:334)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

      I can see no meaningful errors in the logs. It really confuses me that it says the connection is successful but then it fails

            Assignee:
            Cesar Tron-Lozai
            Reporter:
            Cesar Tron-Lozai
            Archiver:
            Jenkins Service Account

              Created:
              Updated:
              Resolved:
              Archived: