[JENKINS-45910] Template namespace is ignored when deleting pod

Type: Bug
Resolution: Fixed
Priority: Major
Component/s: kubernetes-plugin, remoting
Labels:
None
Environment:
Jenkins 2.60.2
Kubernetes plugin 0.12
Kubernetes 1.7.2

Similar Issues:
Powered by SuggestiMate

Show

We are dynamically creating and destroying build slaves and they have now started to exit in a Error state.

Although the build job completes successfully it seems that the slave tries to reconnect.

Log from slave:

Aug 01, 2017 1:37:46 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Aug 01, 2017 1:37:56 PM jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$2$1 onReconnect
INFO: Restarting agent via jenkins.slaves.restarter.UnixSlaveRestarter@863f8a3
Aug 01, 2017 1:37:57 PM hudson.remoting.jnlp.Main createEngine
INFO: Setting up slave: jenkins-build-slave-qvdpn
Aug 01, 2017 1:37:57 PM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://jenkins-service]
Aug 01, 2017 1:37:58 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, JNLP-connect, Ping, JNLP2-connect]
Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: jenkins-service
  Agent port:    50000
  Identity:      51:ac:fd:f7:69:94:53:62:f9:ac:43:6b:72:ff:e7:c2
Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-service:50000
Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 51:ac:fd:f7:69:94:53:62:f9:ac:43:6b:72:ff:e7:c2
Aug 01, 2017 1:37:58 PM org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv
INFO: [JNLP4-connect connection to jenkins-service/10.251.248.121:50000] Local headers refused by remote: Unknown client name: jenkins-build-slave-qvdpn
Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: jenkins-build-slave-qvdpn
	at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
	at hudson.remoting.Engine.innerRun(Engine.java:385)
	at hudson.remoting.Engine.run(Engine.java:287)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: jenkins-build-slave-qvdpn
	at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.newAbortCause(ConnectionHeadersFilterLayer.java:375)
	at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecvClosed(ConnectionHeadersFilterLayer.java:432)
	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172)
	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
	at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
	at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48)
	at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at hudson.remoting.Engine$1$1.run(Engine.java:94)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: java.nio.channels.ClosedChannelException
		... 7 more

Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-service:50000
Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Server reports protocol JNLP4-plaintext not supported, skipping
Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP3-connect is not enabled, skipping
Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP2-connect
Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP2-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at hudson.remoting.Engine.innerRun(Engine.java:385)
	at hudson.remoting.Engine.run(Engine.java:287)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
	at org.jenkinsci.remoting.engine.JnlpProtocol2Handler.sendHandshake(JnlpProtocol2Handler.java:134)
	at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:162)
	at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:158)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at hudson.remoting.Engine$1$1.run(Engine.java:94)
	at java.lang.Thread.run(Thread.java:748)

Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-service:50000
Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP-connect
Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at hudson.remoting.Engine.innerRun(Engine.java:385)
	at hudson.remoting.Engine.run(Engine.java:287)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
	at org.jenkinsci.remoting.engine.JnlpProtocol1Handler.sendHandshake(JnlpProtocol1Handler.java:121)
	at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:162)
	at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:158)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at hudson.remoting.Engine$1$1.run(Engine.java:94)
	at java.lang.Thread.run(Thread.java:748)

Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: The server rejected the connection: None of the protocols were accepted
java.lang.Exception: The server rejected the connection: None of the protocols were accepted
	at hudson.remoting.Engine.onConnectionRejected(Engine.java:484)
	at hudson.remoting.Engine.innerRun(Engine.java:448)
	at hudson.remoting.Engine.run(Engine.java:287)

Log from master:

Aug 01, 2017 3:37:46 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate

Terminating Kubernetes instance for slave jenkins-build-slave-qvdpn

Aug 01, 2017 3:37:46 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate

Terminated Kubernetes instance for slave jenkins-build-slave-qvdpn

Aug 01, 2017 3:37:46 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate

Disconnected computer jenkins-build-slave-qvdpn

Aug 01, 2017 3:37:46 PM WARNING jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed

Computer.threadPoolForRemoting [#324] for jenkins-build-slave-qvdpn terminated
java.nio.channels.ClosedChannelException
	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208)
	at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800)
	at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:311)
	at hudson.remoting.Channel.close(Channel.java:1295)
	at hudson.remoting.Channel.close(Channel.java:1263)
	at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:708)
	at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:96)
	at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:626)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Aug 01, 2017 3:37:46 PM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish

api-gateway #269 completed: SUCCESS

Aug 01, 2017 3:37:58 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run

Accepted JNLP4-connect connection #145 from 10.248.2.34/10.248.2.34:34644

Aug 01, 2017 3:37:58 PM INFO org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv

[JNLP4-connect connection from 10.248.2.34/10.248.2.34:34644] Refusing headers from remote: Unknown client name: jenkins-build-slave-qvdpn

Aug 01, 2017 3:37:58 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run

Accepted JNLP2-connect connection #146 from /10.248.2.34:34646

Aug 01, 2017 3:37:58 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run

Accepted JNLP-connect connection #147 from /10.248.2.34:34648

From kubectl describe

State:		Terminated
      Reason:		Error
      Exit Code:	255

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

kubernetes-plugin.log
17 kB
2017-08-10 11:13
okhttp3-log-recorder.png
25 kB
2017-08-07 13:42
first-slave-describe
4 kB
2017-08-07 13:41
first-slave-log
7 kB
2017-08-07 13:41
kubernetes-plugin.log
7 kB
2017-08-07 13:41
second-slave-describe
4 kB
2017-08-07 13:41
second-slave-log
8 kB
2017-08-07 13:41

is duplicated by

JENKINS-46363 Slave pod using global pod definition end up with error state

Closed

JENKINS-46504 Kubernetes plugin requires ClusterRoles

Closed

Viktor Gunnarson created issue - 2017-08-01 14:15

Viktor Gunnarson made changes - 2017-08-02 09:20

Component/s

New: remoting [ 15489 ]

Alon Lavi made changes - 2017-08-02 12:57

Rank

New: Ranked higher

Juha Tiensyrjä added a comment - 2017-08-04 10:57

We seem to be exhibiting the same problem. I can reproduce it with this simple Pipeline script:

node('jnlp') {
    sh("echo 'foo'")
}

Jenkins master and Kubernetes plugin versions are the same as in the report, but the cluster is running 1.5.6.

Juha Tiensyrjä added a comment - 2017-08-04 10:57 We seem to be exhibiting the same problem. I can reproduce it with this simple Pipeline script: node( 'jnlp' ) { sh( "echo 'foo' " ) } Jenkins master and Kubernetes plugin versions are the same as in the report, but the cluster is running 1.5.6.

Tyler Sanders added a comment - 2017-08-04 13:39

Same problem here in my slave logs. The slave spins up and executes the job fine, but then goes into an "Error" state when finished.

Jenkins 2.67
Kubernetes plugin 0.12
Kubernetes 1.7.2

Tyler Sanders added a comment - 2017-08-04 13:39 Same problem here in my slave logs. The slave spins up and executes the job fine, but then goes into an "Error" state when finished. Jenkins 2.67 Kubernetes plugin 0.12 Kubernetes 1.7.2

Vladislav Trayanov added a comment - 2017-08-07 10:47

Same problem here.

Jenkins 2.60.2
Kubernetes plugin 0.12
Kubernetes 1.7.2

Vladislav Trayanov added a comment - 2017-08-07 10:47 Same problem here. Jenkins 2.60.2 Kubernetes plugin 0.12 Kubernetes 1.7.2

Carlos Sanchez added a comment - 2017-08-07 11:06

The log says

Aug 01, 2017 3:37:46 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Terminated Kubernetes instance for slave jenkins-build-slave-qvdpn

which should mean the pod is deleted

Would need the debug logs for plugin and okhttp3 as described in https://github.com/jenkinsci/kubernetes-plugin#debugging and the pod events from describe, to check whether the pod is not being terminated or if it takes a while to terminate and in that meantime it tries to reconnect

Carlos Sanchez added a comment - 2017-08-07 11:06 The log says Aug 01, 2017 3:37:46 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate Terminated Kubernetes instance for slave jenkins-build-slave-qvdpn which should mean the pod is deleted Would need the debug logs for plugin and okhttp3 as described in https://github.com/jenkinsci/kubernetes-plugin#debugging and the pod events from describe, to check whether the pod is not being terminated or if it takes a while to terminate and in that meantime it tries to reconnect

Viktor Gunnarson made changes - 2017-08-07 13:26

Attachment

New: okhttp3-log-recorder.png [ 39175 ]

Viktor Gunnarson made changes - 2017-08-07 13:26

Attachment		New: first-slave-describe [ 39176 ]
Attachment		New: first-slave-log [ 39177 ]
Attachment		New: kubernetes-plugin.log [ 39178 ]
Attachment		New: second-slave-describe [ 39179 ]
Attachment		New: second-slave-log [ 39180 ]

Viktor Gunnarson made changes - 2017-08-07 13:37

Comment

[ After debugging a little bit more it seems as the kubernetes-plugin actually creates two slave pods that fails.

First it creates a pod that fails fast (see attachment [^first-slave-log] and [^first-slave-describe]) and then tries to create a new pod. This second pod successfully completes the job but then tries to reconnect to the master and then fails (see attachment [^second-slave-log] and [^second-slave-describe]).

The job started at 2017-08-07 13:01:10 UTC and finished at 2017-08-07 13:04:28 UTC.

The second slave logs:
{code}
Aug 07, 2017 1:04:29 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
{code}
but then logs:
{code}
Aug 07, 2017 1:04:39 PM jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$2$1 onReconnect
INFO: Restarting agent via jenkins.slaves.restarter.UnixSlaveRestarter@43361483
{code}

I have attached the [^kubernetes-plugin.log] but I was unable to get any output from the okhttp log recorder, please advise (screenshot of [^okhttp3-log-recorder.png]). ]

Assignee:: Carlos Sanchez

Reporter:: Viktor Gunnarson

Votes:: 10 Vote for this issue

Watchers:: 14 Start watching this issue

Created:: 2017-08-01 14:15

Updated:: 2017-08-28 17:51

Resolved:: 2017-08-14 08:23

Jenkins

Details

Description

Log from slave:

Log from master:

From kubectl describe

Attachments

Attachments

Issue Links

Activity

Collapse comment: Juha Tiensyrjä added a comment - 2017-08-04 10:57

Expand comment: Juha Tiensyrjä added a comment - 2017-08-04 10:57

Collapse comment: Tyler Sanders added a comment - 2017-08-04 13:39

Expand comment: Tyler Sanders added a comment - 2017-08-04 13:39

Collapse comment: Vladislav Trayanov added a comment - 2017-08-07 10:47

Expand comment: Vladislav Trayanov added a comment - 2017-08-07 10:47

Collapse comment: Carlos Sanchez added a comment - 2017-08-07 11:06

Expand comment: Carlos Sanchez added a comment - 2017-08-07 11:06

People

Dates