-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
JENKINS: 2.303.2
Kubernetes Plugin: 1.30.3
We have a use case where we need to set an initContainer to do some operations before the actual job container starts.
The initContainer operation would take around 20-30 mins.
If I add slaveConnectTimeout and set it to 3600, the kubernetes plugin ends up terminating the agent (after 6 mins) and re-creates a new pod.
Are we setting it correctly?
Pipeline Spec:
podTemplate(cloud:'kubernetes', slaveConnectTimeout: '3600', yaml : """ apiVersion: v1 kind: Pod metadata: labels: some-label: some-label-value spec: initContainers: - name: init-sleep image: ubuntu command: ['bash', '-c', 'while true; do echo "hello world"; sleep 10; done'] containers: - name: job-container image: ubuntu:latest command: - cat restartPolicy: Never backoffLimit: 4 tty: true nodeSelector: kubernetes.io/os: linux """ ) { node(POD_LABEL) { stage('work') { container('job-container') { sh 'apt update' } } } }
Logs:
18:17:24 Created Pod: kubernetes dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4 18:17:24 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Scheduled] Successfully assigned dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4 to node-1 18:17:25 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Pulling] Pulling image "ubuntu" 18:17:26 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Pulled] Successfully pulled image "ubuntu" 18:17:26 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Created] Created container init-sleep 18:17:27 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Started] Started container init-sleep 18:17:34 Still waiting to schedule task 18:17:34 'init-build-26-bt2j2-ktzkh-1w9j4' is offline 18:23:34 Created Pod: kubernetes dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk 18:23:34 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Scheduled] Successfully assigned dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk to node-10 18:23:35 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Pulling] Pulling image "ubuntu" 18:23:36 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Pulled] Successfully pulled image "ubuntu" 18:23:36 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Created] Created container init-sleep 18:23:36 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Started] Started container init-sleep
Jenkins Instance Logs:
2021-10-13 01:17:24.724+0000 [id=323] INFO hudson.slaves.NodeProvisioner#update: init-build-26-bt2j2-ktzkh-1w9j4 provisioning successfully completed. We have now 3 computer(s)
2021-10-13 01:17:24.818+0000 [id=322] INFO o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4
2021-10-13 01:23:25.196+0000 [id=383] INFO o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent init-build-26-bt2j2-ktzkh-1w9j4
2021-10-13 01:23:25.202+0000 [id=383] INFO o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer init-build-26-bt2j2-ktzkh-1w9j4
Disconnected computer init-build-26-bt2j2-ktzkh-1w9j4
We are expecting the pod shouldn't get re-created for the amount of time specified by slaveConnectTimeout.
Is there something we missed? What triggers "terminate" for the pod?
Seems the default slaveConnectTimeout is 1000 seconds, i.e. over 16 minutes, so your 6-minute timeout is something else.
https://github.com/jenkinsci/kubernetes-plugin/blob/17ab0ad0f19d01ec99f154f08ea6f7f28f47b3df/src/main/java/org/csanchez/jenkins/plugins/kubernetes/PodTemplate.java#L67-L71