Juha Tiensyrjä added a comment - 2017-08-04 10:57 We seem to be exhibiting the same problem. I can reproduce it with this simple Pipeline script: node( 'jnlp' ) { sh( "echo 'foo' " ) } Jenkins master and Kubernetes plugin versions are the same as in the report, but the cluster is running 1.5.6.

Tyler Sanders added a comment - 2017-08-04 13:39 Same problem here in my slave logs.  The slave spins up and executes the job fine, but then goes into an "Error" state when finished. Jenkins 2.67 Kubernetes plugin 0.12  Kubernetes 1.7.2

Vladislav Trayanov added a comment - 2017-08-07 10:47 Same problem here.  Jenkins 2.60.2  Kubernetes plugin 0.12  Kubernetes 1.7.2

Carlos Sanchez added a comment - 2017-08-07 11:06 The log says Aug 01, 2017 3:37:46 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate Terminated Kubernetes instance for slave jenkins-build-slave-qvdpn which should mean the pod is deleted Would need the debug logs for plugin and okhttp3 as described in https://github.com/jenkinsci/kubernetes-plugin#debugging and the pod events from describe, to check whether the pod is not being terminated or if it takes a while to terminate and in that meantime it tries to reconnect

Carlos Sanchez added a comment - 2017-08-07 13:39 Are pods deleted by jenkins or are they leftover ? The only theory I have is that in your cluster the pod deletion takes longer and before the pod is actually deleted it still has time to reconnect to jenkins

hugo rut added a comment - 2017-08-07 13:40 Same problem here, slave executes the job, then goes into an "Error" state. Jenkins 2.67 Kubernetes plugin 0.12  Kubernetes 1.6.7

Viktor Gunnarson added a comment - 2017-08-07 13:42 After debugging a little bit more it seems as the kubernetes-plugin actually creates two slave pods that fails. First it creates a pod that fails fast (see attachment first-slave-log and first-slave-describe ) and then tries to create a new pod. This second pod successfully completes the job but then tries to reconnect to the master and then fails (see attachment second-slave-log and second-slave-describe ). The job started at 2017-08-07 13:01:10 UTC and finished at 2017-08-07 13:04:28 UTC. The second slave logs: Aug 07, 2017 1:04:29 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated but then logs: Aug 07, 2017 1:04:39 PM jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$2$1 onReconnect INFO: Restarting agent via jenkins.slaves.restarter.UnixSlaveRestarter@43361483 I have attached the kubernetes-plugin.log but I was unable to get any output from the okhttp log recorder, please advise (screenshot of okhttp3-log-recorder.png ).

Viktor Gunnarson added a comment - 2017-08-07 13:43 - edited The pods are never deleted, they are left in state "Terminated" and reason "Error". Our work around has been a small script that deletes any jenkins-slave pod in error state every 30 seconds.

Matt Doran added a comment - 2017-08-09 03:11 We have just started seeing this too.  I'm sorry that I'm not sure whether this aligns with an update to the plugin. We see the same sort of logs, and then the pod dying with 255 and hanging around. This hasn't been a problem recently. We have some slave images that don't see this error problem.  We see similar reconnection errors, but they shutdown properly.   I'm not sure what else I can do to diagnose. This is a high-impact bug.

Urmas Talimaa added a comment - 2017-08-09 05:37 We also started seeing this problem after upgrade to 0.12, the issue is not present in 0.11 (we have downgraded for the time being).

Zalan Meggyesi added a comment - 2017-08-09 14:31 I ran into the same issue on my cluster. For the time being, I've remained on 0.12 and configured the recommended log recorders, see if I can pick up anything there. What is strange is that for me, the issue only seems to affect pods that have service containers attached to them. I have several pod templates, and the ones that are only the jnlp container finish properly and are destroyed, but the ones that have two support containers running services remain errored out, cluttering the cluster.

jota camou added a comment - 2017-08-09 20:49 Same problem with the following environment: Jenkins: 2.73 Kubernetes Plugin versions: 0.11 and 0.12 Kubernetes: 1.7.3 I'm running two different Jenkins instances in the same cluster, only one seems to have the problem.

Carlos Sanchez added a comment - 2017-08-10 11:01 What you'd need to check on the master logs is if you are getting a Terminated Kubernetes instance for slave... line or any error.

Zalan Meggyesi added a comment - 2017-08-10 11:15 I am, yes. I'm attaching the relevant piece of the log (pod name redacted). Disconnecting the pod takes place immediately, then the pod terminates, then waits around to be re-scheduled before getting the exception from the master. kubernetes-plugin.log

Carlos Sanchez added a comment - 2017-08-10 11:17 "then the pod terminates, then waits around" is the pod terminating in kubernetes? because if it is how can it wait around?

Zalan Meggyesi added a comment - 2017-08-10 11:21 No, as far as I can tell, when it receives the termination signal in these circumstances, it remains as "Running" in kubectl's output, before switching to "Error" after the exception. The logs, however, still state that the pod was terminated (line 4). If you can put out an hpi with enhanced logging, I could install it in my cluster to tell you more.

Carlos Sanchez added a comment - 2017-08-10 11:25 ok, so it is the same issue as above, the plugin is sending the "delete" to the pod, it seems to succeed (no errors from api), but the pod continues running. okhttp3 logs would help as described in https://github.com/jenkinsci/kubernetes-plugin#debugging although there are some cases when they are not populated, may need a restart of jenkins. Would be good to see what events show up in kubectl describe pod/jenkins-slave... at the bottom

Viktor Gunnarson added a comment - 2017-08-10 12:02 Carlos Sanchez Here are the events from a `kubectl describe pod`: Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 1m 1m 1 default -scheduler Normal Scheduled Successfully assigned jenkins-build-slave-pd2tx to gke- default -gci-pool-0e10fc26-j9nc 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "workspace-volume" 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "volume-1" 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "volume-0" 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume " default -token-j3cga" 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc spec.containers{jnlp} Normal Pulling pulling image "gcr.io/cloud-solutions-images/jenkins-k8s-slave" 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc spec.containers{jnlp} Normal Pulled Successfully pulled image "gcr.io/cloud-solutions-images/jenkins-k8s-slave" 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc spec.containers{jnlp} Normal Created Created container 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc spec.containers{jnlp} Normal Started Started container Here are logs from okhttp3: Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log --> GET https: //kubernetes. default /api/v1/pods?labelSelector=jenkins%3Dslave http/1.1 Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VueC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW3tajNjZ2EiLCJrdWJlcm5ldGVzLmlvL3NldnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjdiZmY0Njc5LWIxODctMTFlNS1iZTM0LTQyMDEwYWYwMDAxNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.JdJtcAa1nalzsqQnqiTlK0FmE_6OtjSrrjqeEb56Bq0TmRMwnq_Av9UOhGy6yPXF0OsJjbBOaQdFXLOKJ3K7ZBIJVm-oH5yvIu36B1kEdl4QlG82tP6XwooadmEKWSn0qczfHS3yz3ozV0yGEMJZ6CeYA32y5QAcpMwqpfuTGY5FKXpArCdgEaDT4D-XYSYyTrpZViXBkHHHbk54i7xzEj9jEFikV5S7IuwvPV1RpgQz7nQhGnBLCFQ27MAri80OGxpZkrUvihZBQN6VyKvp2D943rl8iaDGqFXJuXjEC04H_o1-a0hhM2nJlZn3mRXSXlm_xKUDB1riljZqvMlubg Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Host: kubernetes. default Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Connection: Keep-Alive Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Accept-Encoding: gzip Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log User-Agent: okhttp/3.7.0 Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log --> END GET Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log <-- 200 OK https: //kubernetes. default /api/v1/pods?labelSelector=jenkins%3Dslave (8ms) Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Date: Thu, 10 Aug 2017 11:52:04 GMT Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Content-Length: 116 Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log { "kind" : "PodList" , "apiVersion" : "v1" , "metadata" :{ "selfLink" : "/api/v1/pods" , "resourceVersion" : "41458438" }, "items" :[]} Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log <-- END HTTP (116- byte body) Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log --> GET https: //kubernetes. default /api/v1/pods?labelSelector=jenkins%3Dslave,jenkins/jenkins-build-slave%3Dtrue http/1.1 Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VueC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW3tajNjZ2EiLCJrdWJlcm5ldGVzLmlvL3NldnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjdiZmY0Njc5LWIxODctMTFlNS1iZTM0LTQyMDEwYWYwMDAxNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.JdJtcAa1nalzsqQnqiTlK0FmE_6OtjSrrjqeEb56Bq0TmRMwnq_Av9UOhGy6yPXF0OsJjbBOaQdFXLOKJ3K7ZBIJVm-oH5yvIu36B1kEdl4QlG82tP6XwooadmEKWSn0qczfHS3yz3ozV0yGEMJZ6CeYA32y5QAcpMwqpfuTGY5FKXpArCdgEaDT4D-XYSYyTrpZViXBkHHHbk54i7xzEj9jEFikV5S7IuwvPV1RpgQz7nQhGnBLCFQ27MAri80OGxpZkrUvihZBQN6VyKvp2D943rl8iaDGqFXJuXjEC04H_o1-a0hhM2nJlZn3mRXSXlm_xKUDB1riljZqvMlubg Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Host: kubernetes. default Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Connection: Keep-Alive Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Accept-Encoding: gzip Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log User-Agent: okhttp/3.7.0 Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log --> END GET Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log <-- 200 OK https: //kubernetes. default /api/v1/pods?labelSelector=jenkins%3Dslave,jenkins/jenkins-build-slave%3Dtrue (8ms) Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Date: Thu, 10 Aug 2017 11:52:04 GMT Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Content-Length: 116 Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log { "kind" : "PodList" , "apiVersion" : "v1" , "metadata" :{ "selfLink" : "/api/v1/pods" , "resourceVersion" : "41458438" }, "items" :[]} Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log <-- END HTTP (116- byte body) Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log --> POST https: //kubernetes. default /api/v1/namespaces/ default /pods http/1.1 Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json; charset=utf-8 Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Content-Length: 1475 Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VueC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW3tajNjZ2EiLCJrdWJlcm5ldGVzLmlvL3NldnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjdiZmY0Njc5LWIxODctMTFlNS1iZTM0LTQyMDEwYWYwMDAxNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.JdJtcAa1nalzsqQnqiTlK0FmE_6OtjSrrjqeEb56Bq0TmRMwnq_Av9UOhGy6yPXF0OsJjbBOaQdFXLOKJ3K7ZBIJVm-oH5yvIu36B1kEdl4QlG82tP6XwooadmEKWSn0qczfHS3yz3ozV0yGEMJZ6CeYA32y5QAcpMwqpfuTGY5FKXpArCdgEaDT4D-XYSYyTrpZViXBkHHHbk54i7xzEj9jEFikV5S7IuwvPV1RpgQz7nQhGnBLCFQ27MAri80OGxpZkrUvihZBQN6VyKvp2D943rl8iaDGqFXJuXjEC04H_o1-a0hhM2nJlZn3mRXSXlm_xKUDB1riljZqvMlubg Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Host: kubernetes. default Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Connection: Keep-Alive Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Accept-Encoding: gzip Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log User-Agent: okhttp/3.7.0 Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log { "apiVersion" : "v1" , "kind" : "Pod" , "metadata" :{ "annotations" :{}, "finalizers" :[], "labels" :{ "jenkins" : "slave" , "jenkins/jenkins-build-slave" : " true " }, "name" : "jenkins-build-slave-pd2tx" , "ownerReferences" :[]}, "spec" :{ "containers" :[{ "args" :[], "command" :[], "env" :[{ "name" : "JENKINS_LOCATION_URL" , "value" : "https: //jenkins/" },{ "name" : "JENKINS_SECRET" , "value" : "3d8b55a21d7d84f4ad54f95f24a730b858e23edfa09d2bbcbaefd85a42df15c3" },{ "name" : "JENKINS_JNLP_URL" , "value" : "http://jenkins-service/computer/jenkins-build-slave-pd2tx/slave-agent.jnlp" },{ "name" : "JENKINS_TUNNEL" , "value" : "jenkins-service:50000" },{ "name" : "JENKINS_NAME" , "value" : "jenkins-build-slave-pd2tx" },{ "name" : "JENKINS_URL" , "value" : "http://jenkins-service" },{ "name" : "HOME" , "value" : "/home/jenkins" }], "image" : "gcr.io/cloud-solutions-images/jenkins-k8s-slave" , "imagePullPolicy" : "Always" , "name" : "jnlp" , "ports" :[], "resources" :{ "limits" :{}, "requests" :{ "memory" : "2600Mi" }}, "securityContext" :{ "privileged" : false }, "tty" : false , "volumeMounts" :[{ "mountPath" : "/home/jenkins/.m2" , "name" : "volume-0" , "readOnly" : false },{ "mountPath" : "/ var /run/docker.sock" , "name" : "volume-1" , "readOnly" : false },{ "mountPath" : "/home/jenkins" , "name" : "workspace-volume" , "readOnly" : false }], "workingDir" : "/home/jenkins" }], "imagePullSecrets" :[], "nodeSelector" :{}, "restartPolicy" : "Never" , "volumes" :[{ "hostPath" :{ "path" : "/ var /tmp" }, "name" : "volume-0" },{ "hostPath" :{ "path" : "/ var /run/docker.sock" }, "name" : "volume-1" },{ "emptyDir" :{ "medium" : ""}," name ":" workspace-volume"}]}} Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log --> END POST (1475- byte body) Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log <-- 201 Created https: //kubernetes. default /api/v1/namespaces/ default /pods (7ms) Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Date: Thu, 10 Aug 2017 11:52:04 GMT Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Transfer-Encoding: chunked Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log { "kind" : "Pod" , "apiVersion" : "v1" , "metadata" :{ "name" : "jenkins-build-slave-pd2tx" , "namespace" : " default " , "selfLink" : "/api/v1/namespaces/ default /pods/jenkins-build-slave-pd2tx" , "uid" : "53c5e2e5-7dc2-11e7-b3bc-42010af0001e" , "resourceVersion" : "41458439" , "creationTimestamp" : "2017-08-10T11:52:04Z" , "labels" :{ "jenkins" : "slave" , "jenkins/jenkins-build-slave" : " true " }, "annotations" :{ "kubernetes.io/limit-ranger" : "LimitRanger plugin set: cpu request for container jnlp" }}, "spec" :{ "volumes" :[{ "name" : "volume-0" , "hostPath" :{ "path" : "/ var /tmp" }},{ "name" : "volume-1" , "hostPath" :{ "path" : "/ var /run/docker.sock" }},{ "name" : "workspace-volume" , "emptyDir" :{ "sizeLimit" : "0" }},{ "name" : " default -token-j3cga" , "secret" :{ "secretName" : " default -token-j3cga" , "defaultMode" :420}}], "containers" :[{ "name" : "jnlp" , "image" : "gcr.io/cloud-solutions-images/jenkins-k8s-slave" , "workingDir" : "/home/jenkins" , "env" :[{ "name" : "JENKINS_LOCATION_URL" , "value" : "https: //jenkins/" },{ "name" : "JENKINS_SECRET" , "value" : "3d8b55a21d7d84f4ad54f95f24a730b858e23edfa09d2bbcbaefd85a42df15c3" },{ "name" : "JENKINS_JNLP_URL" , "value" : "http://jenkins-service/computer/jenkins-build-slave-pd2tx/slave-agent.jnlp" },{ "name" : "JENKINS_TUNNEL" , "value" : "jenkins-service:50000" },{ "name" : "JENKINS_NAME" , "value" : "jenkins-build-slave-pd2tx" },{ "name" : "JENKINS_URL" , "value" : "http://jenkins-service" },{ "name" : "HOME" , "value" : "/home/jenkins" }], "resources" :{ "requests" :{ "cpu" : "100m" , "memory" : "2600Mi" }}, "volumeMounts" :[{ "name" : "volume-0" , "mountPath" : "/home/jenkins/.m2" },{ "name" : "volume-1" , "mountPath" : "/ var /run/docker.sock" },{ "name" : "workspace-volume" , "mountPath" : "/home/jenkins" },{ "name" : " default -token-j3cga" , "readOnly" : true , "mountPath" : "/ var /run/secrets/kubernetes.io/serviceaccount" }], "terminationMessagePath" : "/dev/termination-log" , "terminationMessagePolicy" : "File" , "imagePullPolicy" : "Always" , "securityContext" :{ "privileged" : false }}], "restartPolicy" : "Never" , "terminationGracePeriodSeconds" :30, "dnsPolicy" : "ClusterFirst" , "serviceAccountName" : " default " , "serviceAccount" : " default " , "securityContext" :{}, "schedulerName" : " default -scheduler" , "tolerations" :[{ "key" : "node.alpha.kubernetes.io/notReady" , " operator " : "Exists" , "effect" : "NoExecute" , "tolerationSeconds" :300},{ "key" : "node.alpha.kubernetes.io/unreachable" , " operator " : "Exists" , "effect" : "NoExecute" , "tolerationSeconds" :300}]}, "status" :{ "phase" : "Pending" , "qosClass" : "Burstable" }} Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log <-- END HTTP (2352- byte body) Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log --> GET https: //kubernetes. default /api/v1/namespaces/ default /pods/jenkins-build-slave-pd2tx http/1.1 Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VueC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW3tajNjZ2EiLCJrdWJlcm5ldGVzLmlvL3NldnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjdiZmY0Njc5LWIxODctMTFlNS1iZTM0LTQyMDEwYWYwMDAxNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.JdJtcAa1nalzsqQnqiTlK0FmE_6OtjSrrjqeEb56Bq0TmRMwnq_Av9UOhGy6yPXF0OsJjbBOaQdFXLOKJ3K7ZBIJVm-oH5yvIu36B1kEdl4QlG82tP6XwooadmEKWSn0qczfHS3yz3ozV0yGEMJZ6CeYA32y5QAcpMwqpfuTGY5FKXpArCdgEaDT4D-XYSYyTrpZViXBkHHHbk54i7xzEj9jEFikV5S7IuwvPV1RpgQz7nQhGnBLCFQ27MAri80OGxpZkrUvihZBQN6VyKvp2D943rl8iaDGqFXJuXjEC04H_o1-a0hhM2nJlZn3mRXSXlm_xKUDB1riljZqvMlubg Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Host: kubernetes. default Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Connection: Keep-Alive Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Accept-Encoding: gzip Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log User-Agent: okhttp/3.7.0 Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log --> END GET Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log <-- 200 OK https: //kubernetes. default /api/v1/namespaces/ default /pods/jenkins-build-slave-pd2tx (4ms) Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Date: Thu, 10 Aug 2017 11:52:10 GMT Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Transfer-Encoding: chunked Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log { "kind" : "Pod" , "apiVersion" : "v1" , "metadata" :{ "name" : "jenkins-build-slave-pd2tx" , "namespace" : " default " , "selfLink" : "/api/v1/namespaces/ default /pods/jenkins-build-slave-pd2tx" , "uid" : "53c5e2e5-7dc2-11e7-b3bc-42010af0001e" , "resourceVersion" : "41458444" , "creationTimestamp" : "2017-08-10T11:52:04Z" , "labels" :{ "jenkins" : "slave" , "jenkins/jenkins-build-slave" : " true " }, "annotations" :{ "kubernetes.io/limit-ranger" : "LimitRanger plugin set: cpu request for container jnlp" }}, "spec" :{ "volumes" :[{ "name" : "volume-0" , "hostPath" :{ "path" : "/ var /tmp" }},{ "name" : "volume-1" , "hostPath" :{ "path" : "/ var /run/docker.sock" }},{ "name" : "workspace-volume" , "emptyDir" :{ "sizeLimit" : "0" }},{ "name" : " default -token-j3cga" , "secret" :{ "secretName" : " default -token-j3cga" , "defaultMode" :420}}], "containers" :[{ "name" : "jnlp" , "image" : "gcr.io/cloud-solutions-images/jenkins-k8s-slave" , "workingDir" : "/home/jenkins" , "env" :[{ "name" : "JENKINS_LOCATION_URL" , "value" : "https: //jenkins/" },{ "name" : "JENKINS_SECRET" , "value" : "3d8b55a21d7d84f4ad54f95f24a730b858e23edfa09d2bbcbaefd85a42df15c3" },{ "name" : "JENKINS_JNLP_URL" , "value" : "http://jenkins-service/computer/jenkins-build-slave-pd2tx/slave-agent.jnlp" },{ "name" : "JENKINS_TUNNEL" , "value" : "jenkins-service:50000" },{ "name" : "JENKINS_NAME" , "value" : "jenkins-build-slave-pd2tx" },{ "name" : "JENKINS_URL" , "value" : "http://jenkins-service" },{ "name" : "HOME" , "value" : "/home/jenkins" }], "resources" :{ "requests" :{ "cpu" : "100m" , "memory" : "2600Mi" }}, "volumeMounts" :[{ "name" : "volume-0" , "mountPath" : "/home/jenkins/.m2" },{ "name" : "volume-1" , "mountPath" : "/ var /run/docker.sock" },{ "name" : "workspace-volume" , "mountPath" : "/home/jenkins" },{ "name" : " default -token-j3cga" , "readOnly" : true , "mountPath" : "/ var /run/secrets/kubernetes.io/serviceaccount" }], "terminationMessagePath" : "/dev/termination-log" , "terminationMessagePolicy" : "File" , "imagePullPolicy" : "Always" , "securityContext" :{ "privileged" : false }}], "restartPolicy" : "Never" , "terminationGracePeriodSeconds" :30, "dnsPolicy" : "ClusterFirst" , "serviceAccountName" : " default " , "serviceAccount" : " default " , "nodeName" : "gke- default -gci-pool-0e10fc26-j9nc" , "securityContext" :{}, "schedulerName" : " default -scheduler" , "tolerations" :[{ "key" : "node.alpha.kubernetes.io/notReady" , " operator " : "Exists" , "effect" : "NoExecute" , "tolerationSeconds" :300},{ "key" : "node.alpha.kubernetes.io/unreachable" , " operator " : "Exists" , "effect" : "NoExecute" , "tolerationSeconds" :300}]}, "status" :{ "phase" : "Running" , "conditions" :[{ "type" : "Initialized" , "status" : "True" , "lastProbeTime" : null , "lastTransitionTime" : "2017-08-10T11:52:04Z" },{ "type" : "Ready" , "status" : "True" , "lastProbeTime" : null , "lastTransitionTime" : "2017-08-10T11:52:05Z" },{ "type" : "PodScheduled" , "status" : "True" , "lastProbeTime" : null , "lastTransitionTime" : "2017-08-10T11:52:04Z" }], "hostIP" : "10.240.0.5" , "podIP" : "10.248.2.117" , "startTime" : "2017-08-10T11:52:04Z" , "containerStatuses" :[{ "name" : "jnlp" , "state" :{ "running" :{ "startedAt" : "2017-08-10T11:52:05Z" }}, "lastState" :{}, "ready" : true , "restartCount" :0, "image" : "gcr.io/cloud-solutions-images/jenkins-k8s-slave:latest" , "imageID" : "docker://sha256:a3f51422197c75f12c673f351c3da04a79d1c1508b8f5899eeb5005156f0c719" , "containerID" : "docker://9f6cf64a243b105abf35da0f1e66a03f6b625d02de35d030318fdfeb3c9ddb50" }], "qosClass" : "Burstable" }} Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log <-- END HTTP (3203- byte body) Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log --> DELETE https: //kubernetes. default /api/v1/pods/jenkins-build-slave-pd2tx http/1.1 Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json; charset=utf-8 Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Content-Length: 67 Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VueC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW3tajNjZ2EiLCJrdWJlcm5ldGVzLmlvL3NldnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjdiZmY0Njc5LWIxODctMTFlNS1iZTM0LTQyMDEwYWYwMDAxNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.JdJtcAa1nalzsqQnqiTlK0FmE_6OtjSrrjqeEb56Bq0TmRMwnq_Av9UOhGy6yPXF0OsJjbBOaQdFXLOKJ3K7ZBIJVm-oH5yvIu36B1kEdl4QlG82tP6XwooadmEKWSn0qczfHS3yz3ozV0yGEMJZ6CeYA32y5QAcpMwqpfuTGY5FKXpArCdgEaDT4D-XYSYyTrpZViXBkHHHbk54i7xzEj9jEFikV5S7IuwvPV1RpgQz7nQhGnBLCFQ27MAri80OGxpZkrUvihZBQN6VyKvp2D943rl8iaDGqFXJuXjEC04H_o1-a0hhM2nJlZn3mRXSXlm_xKUDB1riljZqvMlubg Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Host: kubernetes. default Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Connection: Keep-Alive Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Accept-Encoding: gzip Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log User-Agent: okhttp/3.7.0 Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log { "apiVersion" : "v1" , "kind" : "DeleteOptions" , "orphanDependents" : false } Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log --> END DELETE (67- byte body) Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log <-- 404 Not Found https: //kubernetes. default /api/v1/pods/jenkins-build-slave-pd2tx (3ms) Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Date: Thu, 10 Aug 2017 11:53:34 GMT Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Content-Length: 174 Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log { "kind" : "Status" , "apiVersion" : "v1" , "metadata" :{}, "status" : "Failure" , "message" : "the server could not find the requested resource" , "reason" : "NotFound" , "details" :{}, "code" :404} Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log <-- END HTTP (174- byte body)

Juha Tiensyrjä added a comment - 2017-08-10 12:11 Looks like this is missing the namespace? DELETE https://kubernetes.default/api/v1/pods/jenkins-build-slave-pd2tx http/1.1  

Carlos Sanchez added a comment - 2017-08-10 13:36 do you have a namespace in the cloud definition or in the template ? have you changed the cloud definition namespace ? it would require a restart until https://github.com/jenkinsci/kubernetes-plugin/pull/189 is released

Juha Tiensyrjä added a comment - 2017-08-10 13:40 I have the 'default' namespace in cloud definition. Tried adding the namespace to also the 'jnlp' pod, and it looks like now it's deleted from Kubernetes after running the same job which previously left it behind.

Zalan Meggyesi added a comment - 2017-08-10 14:01 I have the specific namespace (not default ) in the cloud definition. I can confirm that replicating the namespace on the pod templates will allow the plugin to remove the slave in question, though it won't remove any that are already in the  Error state.

Viktor Gunnarson added a comment - 2017-08-10 14:17 I have namespace 'default' in the cloud definition but no namespace in the pod template. I will test if it works by adding the namespace to the pod template.

Viktor Gunnarson added a comment - 2017-08-10 14:21 Yes, adding the namespace to the pod template as well seems to do the trick. Then the DELETE request uses the namespace: Aug 10, 2017 4:19:16 PM INFO okhttp3.internal.platform.Platform log --> DELETE https: //kubernetes. default /api/v1/namespaces/ default /pods/jenkins-build-slave-pw0h7 http/1.1