Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-66886

slaveConnectTimeout not honored

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • kubernetes-plugin
    • None
    • JENKINS: 2.303.2
      Kubernetes Plugin: 1.30.3

      We have a use case where we need to set an initContainer to do some operations before the actual job container starts.

      The initContainer operation would take around 20-30 mins.

      If I add slaveConnectTimeout and set it to 3600, the kubernetes plugin ends up terminating the agent (after 6 mins) and re-creates a new pod.

      Are we setting it correctly?

      Pipeline Spec:

      podTemplate(cloud:'kubernetes', 
slaveConnectTimeout: '3600', 
yaml : """
apiVersion: v1
kind: Pod
metadata:
  labels:
    some-label: some-label-value
spec:
  initContainers:
  - name: init-sleep
    image: ubuntu
    command: ['bash', '-c', 'while true; do echo "hello world"; sleep 10; done']
  containers:
  - name: job-container
    image: ubuntu:latest
    command:
    - cat
    restartPolicy: Never
    backoffLimit: 4
    tty: true
  nodeSelector:
    kubernetes.io/os: linux
"""  ) {
  node(POD_LABEL) {
    stage('work') {
        container('job-container') {
            sh 'apt update'
        }
    }
  }
}

      Logs:

      18:17:24  Created Pod: kubernetes dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4
18:17:24  [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Scheduled] Successfully assigned dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4 to node-1
18:17:25  [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Pulling] Pulling image "ubuntu"
18:17:26  [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Pulled] Successfully pulled image "ubuntu"
18:17:26  [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Created] Created container init-sleep
18:17:27  [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Started] Started container init-sleep
18:17:34  Still waiting to schedule task
18:17:34  'init-build-26-bt2j2-ktzkh-1w9j4' is offline
18:23:34  Created Pod: kubernetes dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk
18:23:34  [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Scheduled] Successfully assigned dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk to node-10
18:23:35  [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Pulling] Pulling image "ubuntu"
18:23:36  [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Pulled] Successfully pulled image "ubuntu"
18:23:36  [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Created] Created container init-sleep
18:23:36  [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Started] Started container init-sleep

      Jenkins Instance Logs:

      2021-10-13 01:17:24.724+0000 [id=323]   INFO    hudson.slaves.NodeProvisioner#update: init-build-26-bt2j2-ktzkh-1w9j4 provisioning successfully completed. We have now 3 computer(s)
2021-10-13 01:17:24.818+0000 [id=322]   INFO    o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4
2021-10-13 01:23:25.196+0000 [id=383]   INFO    o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent init-build-26-bt2j2-ktzkh-1w9j4
2021-10-13 01:23:25.202+0000 [id=383]   INFO    o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer init-build-26-bt2j2-ktzkh-1w9j4
Disconnected computer init-build-26-bt2j2-ktzkh-1w9j4

      We are expecting the pod shouldn't get re-created for the amount of time specified by slaveConnectTimeout.

       

      Is there something we missed? What triggers "terminate" for the pod?

       

       

       

            vlatombe Vincent Latombe
            hkulkar Hitesh Kulkarni
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: