-
Type:
Bug
-
Resolution: Duplicate
-
Priority:
Major
-
Component/s: kubernetes-plugin
-
Environment:Jenkins 1.50.3, kubernetes plugin 1.14.8
If a pipeline has a bad containerTemplate definition, the build won't fail, but will infinitely try to complete. The build job effectively hangs.
Expected behavior would be to have the job fail if a Pod can be created for the build.
Reproduce the issue with the following pipeline (notice the bad image reference):
def label = "mypod-${UUID.randomUUID().toString()}"podTemplate(cloud: 'k8s-dc1', label: label, containers: [ containerTemplate(name: 'alpine', image: 'fakeimage', alwaysPullImage: false, ttyEnabled: true, command: 'cat') ] ) { node(label) { stage('alpine') { container('alpine') { sh "pwd && ls -la" } } } }
In my environment, most users don't have access to the namespace where build pods are created, and there is no feedback to the user in Jenkins that there is a bad containerTemplate.Â
When I run the build above, I see this in my build job, which only shows that it is waiting on the Pod (node) to become available.
Started by user Admin
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Still waiting to schedule task
‘Jenkins’ doesn’t have label ‘mypod-7000f2a8-224b-4871-8ded-543b6b1b728d’
Meanwhile, I see this in Kubernetes
~ # kubectl get pod -o wide --watch NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES jenkins-deployment-67c96975d-khpvb 2/2 Running 0 3d4h 10.233.66.66 high-memory-node-5 <none> <none> mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2 0/2 Pending 0 0s <none> <none> <none> <none> mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2 0/2 Pending 0 0s <none> high-memory-node-2 <none> <none> mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2 0/2 ContainerCreating 0 0s <none> high-memory-node-2 <none> <none> mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2 1/2 ErrImagePull 0 3s 10.233.119.96 high-memory-node-2 <none> <none> mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2 1/2 ImagePullBackOff 0 4s 10.233.119.96 high-memory-node-2 <none> <none> mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2 1/2 ErrImagePull 0 31s 10.233.119.96 high-memory-node-2 <none> <none> mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2 1/2 ImagePullBackOff 0 45s 10.233.119.96 high-memory-node-2 <none> <none> mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2 1/2 ErrImagePull 0 57s 10.233.119.96 high-memory-node-2 <none> <none> mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2 1/2 ImagePullBackOff 0 70s 10.233.119.96 high-memory-node-2 <none> <none> mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2 1/2 Terminating 0 100s 10.233.119.96 high-memory-node-2 <none> <none> mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2 1/2 Terminating 0 100s 10.233.119.96 high-memory-node-2 <none> <none> mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2 0/2 Terminating 0 102s <none> high-memory-node-2 <none> <none> mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2 0/2 Terminating 0 103s <none> high-memory-node-2 <none> <none> mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2 0/2 Terminating 0 103s <none> high-memory-node-2 <none> <none> mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk 0/2 Pending 0 0s <none> <none> <none> <none> mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk 0/2 Pending 0 0s <none> high-memory-node-2 <none> <none> mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk 0/2 ContainerCreating 0 0s <none> high-memory-node-2 <none> <none> mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk 1/2 ErrImagePull 0 3s 10.233.119.97 high-memory-node-2 <none> <none> mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk 1/2 ImagePullBackOff 0 5s 10.233.119.97 high-memory-node-2 <none> <none> ...
You can see that after failing to start the Pod several times, Kubernetes (or the plugin) terminates the first pod and starts a new pod, which exhibits the same inability to start. This continues indefinitely. Here is a "describe" of the pod
~ # kubectl describe pod mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk
Name: mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk
Namespace: jenkins-master-test
Priority: 0
PriorityClassName: <none>
Node: high-memory-node-2/10.0.40.14
Start Time: Mon, 04 Mar 2019 20:21:32 +0000
Labels: jenkins=slave
jenkins/mypod-7000f2a8-224b-4871-8ded-543b6b1b728d=true
Annotations: buildUrl: http://jenkins-service:8080/job/testmaxconn/143/
kubernetes.io/limit-ranger:
LimitRanger plugin set: cpu, memory request for container alpine; cpu, memory limit for container alpine; cpu, memory request for containe...
Status: Pending
IP: 10.233.119.97
Containers:
alpine:
Container ID:
Image: fakeimage
Image ID:
Port: <none>
Host Port: <none>
Command:
cat
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 25m
memory: 256Mi
Environment:
JENKINS_SECRET: 7ed09a2770cad6e52f0f2d0b9a47dea3e5bdde1cde75a6309c7fbc72f91cf333
JENKINS_TUNNEL: jenkins-service:50000
JENKINS_AGENT_NAME: mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk
JENKINS_NAME: mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk
JENKINS_URL: http://jenkins-service:8080/
HOME: /home/jenkins
Mounts:
/home/jenkins from workspace-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-qjfqj (ro)
jnlp:
Container ID: docker://07b2f1e8d10b13b0b2980bb53fb556f38470e49b6c94a9dc6be61d6874f06b88
Image: jenkins/jnlp-slave:alpine
Image ID: docker-pullable://jenkins/jnlp-slave@sha256:8e330d8bc461440c797d543b9872e54b328da3ef3b052180bb5aed33204d4384
Port: <none>
Host Port: <none>
State: Running
Started: Mon, 04 Mar 2019 20:21:34 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 25m
memory: 256Mi
Environment:
JENKINS_SECRET: 7ed09a2770cad6e52f0f2d0b9a47dea3e5bdde1cde75a6309c7fbc72f91cf333
JENKINS_TUNNEL: jenkins-service:50000
JENKINS_AGENT_NAME: mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk
JENKINS_NAME: mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk
JENKINS_URL: http://jenkins-service:8080/
HOME: /home/jenkins
Mounts:
/home/jenkins from workspace-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-qjfqj (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
workspace-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
default-token-qjfqj:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-qjfqj
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 15s default-scheduler Successfully assigned jenkins-master-test/mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk to high-memory-node-2
Normal Pulling 13s kubelet, high-memory-node-2 pulling image "fakeimage"
Warning Failed 13s kubelet, high-memory-node-2 Failed to pull image "fakeimage": rpc error: code = Unknown desc = Error response from daemon: pull access denied for fakeimage, repository does not exist or may require 'docker login'
Warning Failed 13s kubelet, high-memory-node-2 Error: ErrImagePull
Normal Pulled 13s kubelet, high-memory-node-2 Container image "jenkins/jnlp-slave:alpine" already present on machine
Normal Created 13s kubelet, high-memory-node-2 Created container
Normal Started 13s kubelet, high-memory-node-2 Started container
Normal BackOff 11s (x2 over 12s) kubelet, high-memory-node-2 Back-off pulling image "fakeimage"
Warning Failed 11s (x2 over 12s) kubelet, high-memory-node-2 Error: ImagePullBackOff
- duplicates
-
JENKINS-53790 Kubernetes plugin shows failing templates to only admins
-
- Resolved
-