[JENKINS-49511] Connection fails between agent and master - Jenkins Jira

Type: Bug
Resolution: Not A Defect
Priority: Major
Component/s: kubernetes-plugin
Labels:
None
Environment:
Jenkins: 2.105
Kubernetes-plugin: 1.2

Similar Issues:
Powered by SuggestiMate

Show

I've set up Jenkins to run on a k8s cluster.

I'm using the kubernetes plugin to create agent in the cluster.

I have a discovery service available at jenkins-discovery.jenkins.svc.cluster.local:50000.

I've set up the tunnel property to jenkins-discovery.jenkins.svc.cluster.local:50000

When I start a job, the agent seems to be able to connect but then fails for some unknown reasons.

The agent logs:

INFO: Locating server among [http://jenkins-internal.jenkins.svc.cluster.local:8080/]
Feb 12, 2018 4:08:37 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Feb 12, 2018 4:08:37 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
Agent address: jenkins-discovery.jenkins.svc.cluster.local
Agent port: 50000
Identity: a1:e6:e0:d2:ad:30:2e:11:47:4f:0f:95:41:d3:d0:a4
Feb 12, 2018 4:08:37 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Feb 12, 2018 4:08:37 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-discovery.jenkins.svc.cluster.local:50000
Feb 12, 2018 4:08:37 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Feb 12, 2018 4:08:37 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: a1:e6:e0:d2:ad:30:2e:11:47:4f:0f:95:41:d3:d0:a4
Feb 12, 2018 4:08:39 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Feb 12, 2018 4:08:41 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated

It repeats this 4 times and then I see

WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@71b21b06
hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@4d5ba314:JNLP4-connect connection to jenkins-discovery.jenkins.svc.cluster.local/100.71.204.63:50000": channel is already closed
at hudson.remoting.Channel.send(Channel.java:672)
at hudson.remoting.Request$2.run(Request.java:371)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine$1$1.run(Engine.java:94)
at java.lang.Thread.run(Thread.java:748)
Caused by: hudson.remoting.Channel$OrderlyShutdown: Command close created at
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1226)
at hudson.remoting.Channel$1.handle(Channel.java:561)
at hudson.remoting.AbstractByteBufferCommandTransport.processCommand(AbstractByteBufferCommandTransport.java:202)
at hudson.remoting.AbstractByteBufferCommandTransport.receive(AbstractByteBufferCommandTransport.java:188)
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onRead(ChannelApplicationLayer.java:186)
at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecv(ApplicationLayer.java:207)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:369)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669)
at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$2200(BIONetworkLayer.java:48)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:283)
... 4 more
Caused by: Command close created at
at hudson.remoting.Command.<init>(Command.java:62)
at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1220)
at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1218)
at hudson.remoting.Channel.close(Channel.java:1391)
at hudson.remoting.Channel.close(Channel.java:1358)
at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:745)
at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:98)
at hudson.slaves.SlaveComputer$2.onClosed(SlaveComputer.java:533)
at hudson.remoting.Channel.terminate(Channel.java:1014)
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208)
at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800)
at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:313)
at hudson.remoting.Channel.close(Channel.java:1405)
at hudson.remoting.Channel.close(Channel.java:1358)
at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:603)
at jenkins.slaves.DefaultJnlpSlaveReceiver.afterChannel(DefaultJnlpSlaveReceiver.java:168)
at org.jenkinsci.remoting.engine.JnlpConnectionState$4.invoke(JnlpConnectionState.java:421)
at org.jenkinsci.remoting.engine.JnlpConnectionState.fire(JnlpConnectionState.java:312)
at org.jenkinsci.remoting.engine.JnlpConnectionState.fireAfterChannel(JnlpConnectionState.java:418)
at org.jenkinsci.remoting.engine.JnlpProtocol4Handler$Handler$1.run(JnlpProtocol4Handler.java:334)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more

In the master I see

INFO: Created Pod: slave-with-docker-g2k77 in namespace jenkins
Feb 12, 2018 4:08:36 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for Pod to be scheduled (0/100): slave-with-docker-g2k77
Feb 12, 2018 4:08:37 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #482 from /100.96.10.76:41026
Feb 12, 2018 4:08:38 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #483 from /100.96.10.76:41036
Feb 12, 2018 4:08:41 PM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
WARNING: Computer.threadPoolForRemoting [#553] for slave-with-docker-g2k77 terminated
java.nio.channels.ClosedChannelException
	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208)
	at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800)
	at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:313)
	at hudson.remoting.Channel.close(Channel.java:1405)
	at hudson.remoting.Channel.close(Channel.java:1358)
	at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:603)
	at jenkins.slaves.DefaultJnlpSlaveReceiver.afterChannel(DefaultJnlpSlaveReceiver.java:168)
	at org.jenkinsci.remoting.engine.JnlpConnectionState$4.invoke(JnlpConnectionState.java:421)
	at org.jenkinsci.remoting.engine.JnlpConnectionState.fire(JnlpConnectionState.java:312)
	at org.jenkinsci.remoting.engine.JnlpConnectionState.fireAfterChannel(JnlpConnectionState.java:418)
	at org.jenkinsci.remoting.engine.JnlpProtocol4Handler$Handler$1.run(JnlpProtocol4Handler.java:334)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

I can see no meaningful errors in the logs. It really confuses me that it says the connection is successful but then it fails

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

logs-from-jnlp-in-slave-with-docker-g2k77.txt
35 kB
2018-02-12 16:39
logs-from-master-in-jenkins-67b96c6474-6m6wt.txt
441 kB
2018-02-12 16:39

is blocking

JENKINS-49510 Failed connections between JNLP agent and master on Kubernetes

Closed

is related to

JENKINS-40847 JNLP slave pod gets created even when a different slave is configured

Closed

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates