Carlos Sanchez added a comment - 2017-08-10 11:17 "then the pod terminates, then waits around" is the pod terminating in kubernetes? because if it is how can it wait around?

Zalan Meggyesi added a comment - 2017-08-10 11:21 No, as far as I can tell, when it receives the termination signal in these circumstances, it remains as "Running" in kubectl's output, before switching to "Error" after the exception. The logs, however, still state that the pod was terminated (line 4). If you can put out an hpi with enhanced logging, I could install it in my cluster to tell you more.

Carlos Sanchez added a comment - 2017-08-10 11:25 ok, so it is the same issue as above, the plugin is sending the "delete" to the pod, it seems to succeed (no errors from api), but the pod continues running. okhttp3 logs would help as described in https://github.com/jenkinsci/kubernetes-plugin#debugging although there are some cases when they are not populated, may need a restart of jenkins. Would be good to see what events show up in kubectl describe pod/jenkins-slave... at the bottom

Viktor Gunnarson added a comment - 2017-08-10 12:02 csanchez Here are the events from a `kubectl describe pod`: Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 1m 1m 1 default -scheduler Normal Scheduled Successfully assigned jenkins-build-slave-pd2tx to gke- default -gci-pool-0e10fc26-j9nc 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "workspace-volume" 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "volume-1" 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "volume-0" 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume " default -token-j3cga" 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc spec.containers{jnlp} Normal Pulling pulling image "gcr.io/cloud-solutions-images/jenkins-k8s-slave" 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc spec.containers{jnlp} Normal Pulled Successfully pulled image "gcr.io/cloud-solutions-images/jenkins-k8s-slave" 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc spec.containers{jnlp} Normal Created Created container 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc spec.containers{jnlp} Normal Started Started container Here are logs from okhttp3: Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log --> GET https: //kubernetes. default /api/v1/pods?labelSelector=jenkins%3Dslave http/1.1 Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VueC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW3tajNjZ2EiLCJrdWJlcm5ldGVzLmlvL3NldnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjdiZmY0Njc5LWIxODctMTFlNS1iZTM0LTQyMDEwYWYwMDAxNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.JdJtcAa1nalzsqQnqiTlK0FmE_6OtjSrrjqeEb56Bq0TmRMwnq_Av9UOhGy6yPXF0OsJjbBOaQdFXLOKJ3K7ZBIJVm-oH5yvIu36B1kEdl4QlG82tP6XwooadmEKWSn0qczfHS3yz3ozV0yGEMJZ6CeYA32y5QAcpMwqpfuTGY5FKXpArCdgEaDT4D-XYSYyTrpZViXBkHHHbk54i7xzEj9jEFikV5S7IuwvPV1RpgQz7nQhGnBLCFQ27MAri80OGxpZkrUvihZBQN6VyKvp2D943rl8iaDGqFXJuXjEC04H_o1-a0hhM2nJlZn3mRXSXlm_xKUDB1riljZqvMlubg Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Host: kubernetes. default Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Connection: Keep-Alive Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Accept-Encoding: gzip Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log User-Agent: okhttp/3.7.0 Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log --> END GET Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log <-- 200 OK https: //kubernetes. default /api/v1/pods?labelSelector=jenkins%3Dslave (8ms) Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Date: Thu, 10 Aug 2017 11:52:04 GMT Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Content-Length: 116 Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log { "kind" : "PodList" , "apiVersion" : "v1" , "metadata" :{ "selfLink" : "/api/v1/pods" , "resourceVersion" : "41458438" }, "items" :[]} Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log <-- END HTTP (116- byte body) Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log --> GET https: //kubernetes. default /api/v1/pods?labelSelector=jenkins%3Dslave,jenkins/jenkins-build-slave%3Dtrue http/1.1 Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VueC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW3tajNjZ2EiLCJrdWJlcm5ldGVzLmlvL3NldnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjdiZmY0Njc5LWIxODctMTFlNS1iZTM0LTQyMDEwYWYwMDAxNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.JdJtcAa1nalzsqQnqiTlK0FmE_6OtjSrrjqeEb56Bq0TmRMwnq_Av9UOhGy6yPXF0OsJjbBOaQdFXLOKJ3K7ZBIJVm-oH5yvIu36B1kEdl4QlG82tP6XwooadmEKWSn0qczfHS3yz3ozV0yGEMJZ6CeYA32y5QAcpMwqpfuTGY5FKXpArCdgEaDT4D-XYSYyTrpZViXBkHHHbk54i7xzEj9jEFikV5S7IuwvPV1RpgQz7nQhGnBLCFQ27MAri80OGxpZkrUvihZBQN6VyKvp2D943rl8iaDGqFXJuXjEC04H_o1-a0hhM2nJlZn3mRXSXlm_xKUDB1riljZqvMlubg Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Host: kubernetes. default Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Connection: Keep-Alive Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Accept-Encoding: gzip Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log User-Agent: okhttp/3.7.0 Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log --> END GET Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log <-- 200 OK https: //kubernetes. default /api/v1/pods?labelSelector=jenkins%3Dslave,jenkins/jenkins-build-slave%3Dtrue (8ms) Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Date: Thu, 10 Aug 2017 11:52:04 GMT Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Content-Length: 116 Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log { "kind" : "PodList" , "apiVersion" : "v1" , "metadata" :{ "selfLink" : "/api/v1/pods" , "resourceVersion" : "41458438" }, "items" :[]} Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log <-- END HTTP (116- byte body) Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log --> POST https: //kubernetes. default /api/v1/namespaces/ default /pods http/1.1 Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json; charset=utf-8 Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Content-Length: 1475 Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VueC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW3tajNjZ2EiLCJrdWJlcm5ldGVzLmlvL3NldnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjdiZmY0Njc5LWIxODctMTFlNS1iZTM0LTQyMDEwYWYwMDAxNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.JdJtcAa1nalzsqQnqiTlK0FmE_6OtjSrrjqeEb56Bq0TmRMwnq_Av9UOhGy6yPXF0OsJjbBOaQdFXLOKJ3K7ZBIJVm-oH5yvIu36B1kEdl4QlG82tP6XwooadmEKWSn0qczfHS3yz3ozV0yGEMJZ6CeYA32y5QAcpMwqpfuTGY5FKXpArCdgEaDT4D-XYSYyTrpZViXBkHHHbk54i7xzEj9jEFikV5S7IuwvPV1RpgQz7nQhGnBLCFQ27MAri80OGxpZkrUvihZBQN6VyKvp2D943rl8iaDGqFXJuXjEC04H_o1-a0hhM2nJlZn3mRXSXlm_xKUDB1riljZqvMlubg Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Host: kubernetes. default Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Connection: Keep-Alive Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Accept-Encoding: gzip Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log User-Agent: okhttp/3.7.0 Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log { "apiVersion" : "v1" , "kind" : "Pod" , "metadata" :{ "annotations" :{}, "finalizers" :[], "labels" :{ "jenkins" : "slave" , "jenkins/jenkins-build-slave" : " true " }, "name" : "jenkins-build-slave-pd2tx" , "ownerReferences" :[]}, "spec" :{ "containers" :[{ "args" :[], "command" :[], "env" :[{ "name" : "JENKINS_LOCATION_URL" , "value" : "https: //jenkins/" },{ "name" : "JENKINS_SECRET" , "value" : "3d8b55a21d7d84f4ad54f95f24a730b858e23edfa09d2bbcbaefd85a42df15c3" },{ "name" : "JENKINS_JNLP_URL" , "value" : "http://jenkins-service/computer/jenkins-build-slave-pd2tx/slave-agent.jnlp" },{ "name" : "JENKINS_TUNNEL" , "value" : "jenkins-service:50000" },{ "name" : "JENKINS_NAME" , "value" : "jenkins-build-slave-pd2tx" },{ "name" : "JENKINS_URL" , "value" : "http://jenkins-service" },{ "name" : "HOME" , "value" : "/home/jenkins" }], "image" : "gcr.io/cloud-solutions-images/jenkins-k8s-slave" , "imagePullPolicy" : "Always" , "name" : "jnlp" , "ports" :[], "resources" :{ "limits" :{}, "requests" :{ "memory" : "2600Mi" }}, "securityContext" :{ "privileged" : false }, "tty" : false , "volumeMounts" :[{ "mountPath" : "/home/jenkins/.m2" , "name" : "volume-0" , "readOnly" : false },{ "mountPath" : "/ var /run/docker.sock" , "name" : "volume-1" , "readOnly" : false },{ "mountPath" : "/home/jenkins" , "name" : "workspace-volume" , "readOnly" : false }], "workingDir" : "/home/jenkins" }], "imagePullSecrets" :[], "nodeSelector" :{}, "restartPolicy" : "Never" , "volumes" :[{ "hostPath" :{ "path" : "/ var /tmp" }, "name" : "volume-0" },{ "hostPath" :{ "path" : "/ var /run/docker.sock" }, "name" : "volume-1" },{ "emptyDir" :{ "medium" : ""}," name ":" workspace-volume"}]}} Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log --> END POST (1475- byte body) Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log <-- 201 Created https: //kubernetes. default /api/v1/namespaces/ default /pods (7ms) Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Date: Thu, 10 Aug 2017 11:52:04 GMT Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Transfer-Encoding: chunked Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log { "kind" : "Pod" , "apiVersion" : "v1" , "metadata" :{ "name" : "jenkins-build-slave-pd2tx" , "namespace" : " default " , "selfLink" : "/api/v1/namespaces/ default /pods/jenkins-build-slave-pd2tx" , "uid" : "53c5e2e5-7dc2-11e7-b3bc-42010af0001e" , "resourceVersion" : "41458439" , "creationTimestamp" : "2017-08-10T11:52:04Z" , "labels" :{ "jenkins" : "slave" , "jenkins/jenkins-build-slave" : " true " }, "annotations" :{ "kubernetes.io/limit-ranger" : "LimitRanger plugin set: cpu request for container jnlp" }}, "spec" :{ "volumes" :[{ "name" : "volume-0" , "hostPath" :{ "path" : "/ var /tmp" }},{ "name" : "volume-1" , "hostPath" :{ "path" : "/ var /run/docker.sock" }},{ "name" : "workspace-volume" , "emptyDir" :{ "sizeLimit" : "0" }},{ "name" : " default -token-j3cga" , "secret" :{ "secretName" : " default -token-j3cga" , "defaultMode" :420}}], "containers" :[{ "name" : "jnlp" , "image" : "gcr.io/cloud-solutions-images/jenkins-k8s-slave" , "workingDir" : "/home/jenkins" , "env" :[{ "name" : "JENKINS_LOCATION_URL" , "value" : "https: //jenkins/" },{ "name" : "JENKINS_SECRET" , "value" : "3d8b55a21d7d84f4ad54f95f24a730b858e23edfa09d2bbcbaefd85a42df15c3" },{ "name" : "JENKINS_JNLP_URL" , "value" : "http://jenkins-service/computer/jenkins-build-slave-pd2tx/slave-agent.jnlp" },{ "name" : "JENKINS_TUNNEL" , "value" : "jenkins-service:50000" },{ "name" : "JENKINS_NAME" , "value" : "jenkins-build-slave-pd2tx" },{ "name" : "JENKINS_URL" , "value" : "http://jenkins-service" },{ "name" : "HOME" , "value" : "/home/jenkins" }], "resources" :{ "requests" :{ "cpu" : "100m" , "memory" : "2600Mi" }}, "volumeMounts" :[{ "name" : "volume-0" , "mountPath" : "/home/jenkins/.m2" },{ "name" : "volume-1" , "mountPath" : "/ var /run/docker.sock" },{ "name" : "workspace-volume" , "mountPath" : "/home/jenkins" },{ "name" : " default -token-j3cga" , "readOnly" : true , "mountPath" : "/ var /run/secrets/kubernetes.io/serviceaccount" }], "terminationMessagePath" : "/dev/termination-log" , "terminationMessagePolicy" : "File" , "imagePullPolicy" : "Always" , "securityContext" :{ "privileged" : false }}], "restartPolicy" : "Never" , "terminationGracePeriodSeconds" :30, "dnsPolicy" : "ClusterFirst" , "serviceAccountName" : " default " , "serviceAccount" : " default " , "securityContext" :{}, "schedulerName" : " default -scheduler" , "tolerations" :[{ "key" : "node.alpha.kubernetes.io/notReady" , " operator " : "Exists" , "effect" : "NoExecute" , "tolerationSeconds" :300},{ "key" : "node.alpha.kubernetes.io/unreachable" , " operator " : "Exists" , "effect" : "NoExecute" , "tolerationSeconds" :300}]}, "status" :{ "phase" : "Pending" , "qosClass" : "Burstable" }} Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log <-- END HTTP (2352- byte body) Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log --> GET https: //kubernetes. default /api/v1/namespaces/ default /pods/jenkins-build-slave-pd2tx http/1.1 Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VueC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW3tajNjZ2EiLCJrdWJlcm5ldGVzLmlvL3NldnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjdiZmY0Njc5LWIxODctMTFlNS1iZTM0LTQyMDEwYWYwMDAxNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.JdJtcAa1nalzsqQnqiTlK0FmE_6OtjSrrjqeEb56Bq0TmRMwnq_Av9UOhGy6yPXF0OsJjbBOaQdFXLOKJ3K7ZBIJVm-oH5yvIu36B1kEdl4QlG82tP6XwooadmEKWSn0qczfHS3yz3ozV0yGEMJZ6CeYA32y5QAcpMwqpfuTGY5FKXpArCdgEaDT4D-XYSYyTrpZViXBkHHHbk54i7xzEj9jEFikV5S7IuwvPV1RpgQz7nQhGnBLCFQ27MAri80OGxpZkrUvihZBQN6VyKvp2D943rl8iaDGqFXJuXjEC04H_o1-a0hhM2nJlZn3mRXSXlm_xKUDB1riljZqvMlubg Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Host: kubernetes. default Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Connection: Keep-Alive Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Accept-Encoding: gzip Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log User-Agent: okhttp/3.7.0 Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log --> END GET Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log <-- 200 OK https: //kubernetes. default /api/v1/namespaces/ default /pods/jenkins-build-slave-pd2tx (4ms) Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Date: Thu, 10 Aug 2017 11:52:10 GMT Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Transfer-Encoding: chunked Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log { "kind" : "Pod" , "apiVersion" : "v1" , "metadata" :{ "name" : "jenkins-build-slave-pd2tx" , "namespace" : " default " , "selfLink" : "/api/v1/namespaces/ default /pods/jenkins-build-slave-pd2tx" , "uid" : "53c5e2e5-7dc2-11e7-b3bc-42010af0001e" , "resourceVersion" : "41458444" , "creationTimestamp" : "2017-08-10T11:52:04Z" , "labels" :{ "jenkins" : "slave" , "jenkins/jenkins-build-slave" : " true " }, "annotations" :{ "kubernetes.io/limit-ranger" : "LimitRanger plugin set: cpu request for container jnlp" }}, "spec" :{ "volumes" :[{ "name" : "volume-0" , "hostPath" :{ "path" : "/ var /tmp" }},{ "name" : "volume-1" , "hostPath" :{ "path" : "/ var /run/docker.sock" }},{ "name" : "workspace-volume" , "emptyDir" :{ "sizeLimit" : "0" }},{ "name" : " default -token-j3cga" , "secret" :{ "secretName" : " default -token-j3cga" , "defaultMode" :420}}], "containers" :[{ "name" : "jnlp" , "image" : "gcr.io/cloud-solutions-images/jenkins-k8s-slave" , "workingDir" : "/home/jenkins" , "env" :[{ "name" : "JENKINS_LOCATION_URL" , "value" : "https: //jenkins/" },{ "name" : "JENKINS_SECRET" , "value" : "3d8b55a21d7d84f4ad54f95f24a730b858e23edfa09d2bbcbaefd85a42df15c3" },{ "name" : "JENKINS_JNLP_URL" , "value" : "http://jenkins-service/computer/jenkins-build-slave-pd2tx/slave-agent.jnlp" },{ "name" : "JENKINS_TUNNEL" , "value" : "jenkins-service:50000" },{ "name" : "JENKINS_NAME" , "value" : "jenkins-build-slave-pd2tx" },{ "name" : "JENKINS_URL" , "value" : "http://jenkins-service" },{ "name" : "HOME" , "value" : "/home/jenkins" }], "resources" :{ "requests" :{ "cpu" : "100m" , "memory" : "2600Mi" }}, "volumeMounts" :[{ "name" : "volume-0" , "mountPath" : "/home/jenkins/.m2" },{ "name" : "volume-1" , "mountPath" : "/ var /run/docker.sock" },{ "name" : "workspace-volume" , "mountPath" : "/home/jenkins" },{ "name" : " default -token-j3cga" , "readOnly" : true , "mountPath" : "/ var /run/secrets/kubernetes.io/serviceaccount" }], "terminationMessagePath" : "/dev/termination-log" , "terminationMessagePolicy" : "File" , "imagePullPolicy" : "Always" , "securityContext" :{ "privileged" : false }}], "restartPolicy" : "Never" , "terminationGracePeriodSeconds" :30, "dnsPolicy" : "ClusterFirst" , "serviceAccountName" : " default " , "serviceAccount" : " default " , "nodeName" : "gke- default -gci-pool-0e10fc26-j9nc" , "securityContext" :{}, "schedulerName" : " default -scheduler" , "tolerations" :[{ "key" : "node.alpha.kubernetes.io/notReady" , " operator " : "Exists" , "effect" : "NoExecute" , "tolerationSeconds" :300},{ "key" : "node.alpha.kubernetes.io/unreachable" , " operator " : "Exists" , "effect" : "NoExecute" , "tolerationSeconds" :300}]}, "status" :{ "phase" : "Running" , "conditions" :[{ "type" : "Initialized" , "status" : "True" , "lastProbeTime" : null , "lastTransitionTime" : "2017-08-10T11:52:04Z" },{ "type" : "Ready" , "status" : "True" , "lastProbeTime" : null , "lastTransitionTime" : "2017-08-10T11:52:05Z" },{ "type" : "PodScheduled" , "status" : "True" , "lastProbeTime" : null , "lastTransitionTime" : "2017-08-10T11:52:04Z" }], "hostIP" : "10.240.0.5" , "podIP" : "10.248.2.117" , "startTime" : "2017-08-10T11:52:04Z" , "containerStatuses" :[{ "name" : "jnlp" , "state" :{ "running" :{ "startedAt" : "2017-08-10T11:52:05Z" }}, "lastState" :{}, "ready" : true , "restartCount" :0, "image" : "gcr.io/cloud-solutions-images/jenkins-k8s-slave:latest" , "imageID" : "docker://sha256:a3f51422197c75f12c673f351c3da04a79d1c1508b8f5899eeb5005156f0c719" , "containerID" : "docker://9f6cf64a243b105abf35da0f1e66a03f6b625d02de35d030318fdfeb3c9ddb50" }], "qosClass" : "Burstable" }} Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log <-- END HTTP (3203- byte body) Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log --> DELETE https: //kubernetes. default /api/v1/pods/jenkins-build-slave-pd2tx http/1.1 Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json; charset=utf-8 Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Content-Length: 67 Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VueC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW3tajNjZ2EiLCJrdWJlcm5ldGVzLmlvL3NldnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjdiZmY0Njc5LWIxODctMTFlNS1iZTM0LTQyMDEwYWYwMDAxNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.JdJtcAa1nalzsqQnqiTlK0FmE_6OtjSrrjqeEb56Bq0TmRMwnq_Av9UOhGy6yPXF0OsJjbBOaQdFXLOKJ3K7ZBIJVm-oH5yvIu36B1kEdl4QlG82tP6XwooadmEKWSn0qczfHS3yz3ozV0yGEMJZ6CeYA32y5QAcpMwqpfuTGY5FKXpArCdgEaDT4D-XYSYyTrpZViXBkHHHbk54i7xzEj9jEFikV5S7IuwvPV1RpgQz7nQhGnBLCFQ27MAri80OGxpZkrUvihZBQN6VyKvp2D943rl8iaDGqFXJuXjEC04H_o1-a0hhM2nJlZn3mRXSXlm_xKUDB1riljZqvMlubg Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Host: kubernetes. default Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Connection: Keep-Alive Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Accept-Encoding: gzip Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log User-Agent: okhttp/3.7.0 Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log { "apiVersion" : "v1" , "kind" : "DeleteOptions" , "orphanDependents" : false } Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log --> END DELETE (67- byte body) Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log <-- 404 Not Found https: //kubernetes. default /api/v1/pods/jenkins-build-slave-pd2tx (3ms) Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Date: Thu, 10 Aug 2017 11:53:34 GMT Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Content-Length: 174 Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log { "kind" : "Status" , "apiVersion" : "v1" , "metadata" :{}, "status" : "Failure" , "message" : "the server could not find the requested resource" , "reason" : "NotFound" , "details" :{}, "code" :404} Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log <-- END HTTP (174- byte body)

Juha Tiensyrjä added a comment - 2017-08-10 12:11 Looks like this is missing the namespace? DELETE https://kubernetes.default/api/v1/pods/jenkins-build-slave-pd2tx http/1.1  

Carlos Sanchez added a comment - 2017-08-10 13:36 do you have a namespace in the cloud definition or in the template ? have you changed the cloud definition namespace ? it would require a restart until https://github.com/jenkinsci/kubernetes-plugin/pull/189 is released

Juha Tiensyrjä added a comment - 2017-08-10 13:40 I have the 'default' namespace in cloud definition. Tried adding the namespace to also the 'jnlp' pod, and it looks like now it's deleted from Kubernetes after running the same job which previously left it behind.

Zalan Meggyesi added a comment - 2017-08-10 14:01 I have the specific namespace (not default ) in the cloud definition. I can confirm that replicating the namespace on the pod templates will allow the plugin to remove the slave in question, though it won't remove any that are already in the  Error state.

Viktor Gunnarson added a comment - 2017-08-10 14:17 I have namespace 'default' in the cloud definition but no namespace in the pod template. I will test if it works by adding the namespace to the pod template.

Viktor Gunnarson added a comment - 2017-08-10 14:21 Yes, adding the namespace to the pod template as well seems to do the trick. Then the DELETE request uses the namespace: Aug 10, 2017 4:19:16 PM INFO okhttp3.internal.platform.Platform log --> DELETE https: //kubernetes. default /api/v1/namespaces/ default /pods/jenkins-build-slave-pw0h7 http/1.1