Loading...

Type: Bug
Resolution: Not A Defect
Priority: Major
Component/s: kubernetes-plugin
Labels:
None
Environment:
Jenkins 2.235.1 running in a GKE cluster
kubernetes-plugin 1.26.3

Similar Issues:

Show

Hi,

It seems that the jnlp container fails to start whenever I use the dynamicPVC option as workspaceVolume.

podTemplate(
            namespace: "jenkins",
            serviceAccount: "jenkins",
            containers: [
                containerTemplate(name: 'docker', image: 'docker:stable', command: 'cat', ttyEnabled: true),
                containerTemplate(name: 'jnlp', image: 'jenkins/inbound-agent', runAsUser: '1000', envVars: [envVar(key: 'HOME', value: '/home/jenkins')])
            ],
            volumes: [hostPathVolume(hostPath: '/var/run/docker.sock', mountPath: '/var/run/docker.sock')],
            workspaceVolume: dynamicPVC(accessModes: 'ReadWriteOnce', requestsSize: "50Gi"),
        ) {
            node(POD_LABEL) {
                body()
            }
        }

Logs from the job:

Created Pod: jenkins/docker-image-update-55-lt7-j3fmj
 [Warning][jenkins/docker-image-update-55-lt7-j3fmj][FailedScheduling] pod has unbound immediate PersistentVolumeClaims
 [Normal][jenkins/docker-image-update-55-lt7-j3fmj][NotTriggerScaleUp] pod didn't trigger scale-up (it wouldn't fit if a new node is added): 
 [Warning][jenkins/docker-image-update-55-lt7-j3fmj][FailedScheduling] pod has unbound immediate PersistentVolumeClaims
 [Normal][jenkins/docker-image-update-55-lt7-j3fmj][Scheduled] Successfully assigned jenkins/docker-image-update-55-lt7-j3fmj to gke-firmware-devinfra-in-jenkins-pool-443f9096-59nr
 [Normal][jenkins/docker-image-update-55-lt7-j3fmj][SuccessfulAttachVolume] AttachVolume.Attach succeeded for volume "pvc-8d09711c-82d3-4daa-8162-0278d8edf5cf" 
 [Normal][jenkins/docker-image-update-55-lt7-j3fmj][Pulled] Container image "jenkins/inbound-agent" already present on machine
 [Normal][jenkins/docker-image-update-55-lt7-j3fmj][Created] Created container jnlp
 [Normal][jenkins/docker-image-update-55-lt7-j3fmj][Started] Started container jnlp
 [Normal][jenkins/docker-image-update-55-lt7-j3fmj][Pulled] Container image "docker:stable" already present on machine
 [Normal][jenkins/docker-image-update-55-lt7-j3fmj][Created] Created container docker
 [Normal][jenkins/docker-image-update-55-lt7-j3fmj][Started] Started container docker
 Created Pod: jenkins/docker-image-update-55-lt7-slg0t

In the system log:

Error in provisioning; agent=KubernetesSlave name: docker-image-update-55-lt7-j3fmj, template=PodTemplate{, name='docker-image-update_55-lt7lc-jpjwh', namespace='jenkins', label='docker-image-update_55-lt7lc', serviceAccount='jenkins', nodeUsageMode=EXCLUSIVE, workspaceVolume=org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.DynamicPVCWorkspaceVolume@6d057d0c, volumes=[HostPathVolume [mountPath=/var/run/docker.sock, hostPath=/var/run/docker.sock]], containers=[ContainerTemplate
{name='docker', image='docker:stable', command='cat', ttyEnabled=true}
, ContainerTemplate
{name='jnlp', image='jenkins/inbound-agent', runAsUser=1000, envVars=[KeyValueEnvVar [getValue()=/home/jenkins, getKey()=HOME]]}
], annotations=[org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@aab9c821, org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@c92c82e4]}
 java.lang.IllegalStateException: Pod has terminated containers: jenkins/docker-image-update-55-lt7-j3fmj (jnlp)
 at org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher.periodicAwait(AllContainersRunningPodWatcher.java:133)
 at org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher.periodicAwait(AllContainersRunningPodWatcher.java:154)
 at org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher.periodicAwait(AllContainersRunningPodWatcher.java:154)
 at org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher.periodicAwait(AllContainersRunningPodWatcher.java:154)
 at org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher.await(AllContainersRunningPodWatcher.java:94)
 at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:140)
 at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:296)
 at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
 at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

And it goes on repeating.

Looking in the system log I can see that updated pod template says that jnlp fails to start:

 containerStatuses:
  - containerID: "docker://e5772374ea0c475430095cfa113f58cd2265fc9ad076b5b1e16890234cbd0e5b"
    image: "docker:stable"
    imageID: "docker-pullable://docker@sha256:5ed3b31d3b1e5ac2f06187bb3e1aadcb6b9b9cb5ddb18f1dc25c80ce1047acb7"
    lastState: {}
    name: "docker"
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2020-07-10T16:13:30Z"
  - containerID: "docker://a3c01a55fc5f13a53ec921137924747a34cdf8e7d5b1b2053e5935381fcb9661"
    image: "jenkins/inbound-agent:latest"
    imageID: "docker-pullable://jenkins/inbound-agent@sha256:8643f4e7c586c03e7be74512fef90d5cd962dc6d1491ac2f7e95bb79cec911a1"
    lastState: {}
    name: "jnlp"
    ready: false
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: "docker://a3c01a55fc5f13a53ec921137924747a34cdf8e7d5b1b2053e5935381fcb9661"
        exitCode: 1
        finishedAt: "2020-07-10T16:13:30Z"
        reason: "Error"
        startedAt: "2020-07-10T16:13:30Z"

I've watched kubectl output and I can see that the PVC's get created and they are bound, and then they get deleted. I couldn't get logs from the pods however.

Details

Description

Attachments

Activity

People

Dates