[JENKINS-66886] slaveConnectTimeout not honored

Type: Bug
Resolution: Unresolved
Priority: Critical
Component/s: kubernetes-plugin
Labels:
None
Environment:
JENKINS: 2.303.2
Kubernetes Plugin: 1.30.3

Similar Issues:
Powered by SuggestiMate

Show

We have a use case where we need to set an initContainer to do some operations before the actual job container starts.

The initContainer operation would take around 20-30 mins.

If I add slaveConnectTimeout and set it to 3600, the kubernetes plugin ends up terminating the agent (after 6 mins) and re-creates a new pod.

Are we setting it correctly?

Pipeline Spec:

podTemplate(cloud:'kubernetes', 
slaveConnectTimeout: '3600', 
yaml : """
apiVersion: v1
kind: Pod
metadata:
  labels:
    some-label: some-label-value
spec:
  initContainers:
  - name: init-sleep
    image: ubuntu
    command: ['bash', '-c', 'while true; do echo "hello world"; sleep 10; done']
  containers:
  - name: job-container
    image: ubuntu:latest
    command:
    - cat
    restartPolicy: Never
    backoffLimit: 4
    tty: true
  nodeSelector:
    kubernetes.io/os: linux
"""  ) {
  node(POD_LABEL) {
    stage('work') {
        container('job-container') {
            sh 'apt update'
        }
    }
  }
}

Logs:

18:17:24  Created Pod: kubernetes dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4
18:17:24  [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Scheduled] Successfully assigned dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4 to node-1
18:17:25  [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Pulling] Pulling image "ubuntu"
18:17:26  [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Pulled] Successfully pulled image "ubuntu"
18:17:26  [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Created] Created container init-sleep
18:17:27  [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Started] Started container init-sleep
18:17:34  Still waiting to schedule task
18:17:34  'init-build-26-bt2j2-ktzkh-1w9j4' is offline
18:23:34  Created Pod: kubernetes dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk
18:23:34  [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Scheduled] Successfully assigned dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk to node-10
18:23:35  [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Pulling] Pulling image "ubuntu"
18:23:36  [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Pulled] Successfully pulled image "ubuntu"
18:23:36  [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Created] Created container init-sleep
18:23:36  [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Started] Started container init-sleep

Jenkins Instance Logs:

2021-10-13 01:17:24.724+0000 [id=323]   INFO    hudson.slaves.NodeProvisioner#update: init-build-26-bt2j2-ktzkh-1w9j4 provisioning successfully completed. We have now 3 computer(s)
2021-10-13 01:17:24.818+0000 [id=322]   INFO    o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4
2021-10-13 01:23:25.196+0000 [id=383]   INFO    o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent init-build-26-bt2j2-ktzkh-1w9j4
2021-10-13 01:23:25.202+0000 [id=383]   INFO    o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer init-build-26-bt2j2-ktzkh-1w9j4
Disconnected computer init-build-26-bt2j2-ktzkh-1w9j4

We are expecting the pod shouldn't get re-created for the amount of time specified by slaveConnectTimeout.

Is there something we missed? What triggers "terminate" for the pod?

Hitesh Kulkarni created issue - 2021-10-13 01:48

Hitesh Kulkarni made changes - 2021-10-13 17:18

Description

Original: We have a use case where we need to set an initContainer to do some operations before the actual job container starts.

The initContainer operation would take around 20-30 mins.

If I add slaveConnectTimeout and set it to 3600, the kubernetes plugin ends up terminating the agent (after 6 mins) and re-creates a new pod.

Are we setting it correctly?

Pipeline Spec:
{code:java}
podTemplate(cloud:'kubernetes',
slaveConnectTimeout: '3600',
yaml : """
apiVersion: v1
kind: Pod
metadata:
  labels:
    some-label: some-label-value
spec:
  initContainers:
  - name: init-sleep
    image: ubuntu
    command: ['bash', '-c', 'while true; do echo "hello world"; sleep 10; done']
  containers:
  - name: job-container
    image: ubuntu:latest
    command:
    - cat
    restartPolicy: Never
    backoffLimit: 4
    tty: true
  nodeSelector:
    kubernetes.io/os: linux
""" ) {
  node(POD_LABEL) {
    stage('work') {
        container('job-container') {
            sh 'apt update'
        }
    }
  }
}

{code}
Logs:
{code:java}
18:17:24 Created Pod: kubernetes dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4
18:17:24 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Scheduled] Successfully assigned dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4 to node-1
18:17:25 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Pulling] Pulling image "ubuntu"
18:17:26 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Pulled] Successfully pulled image "ubuntu"
18:17:26 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Created] Created container init-sleep
18:17:27 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Started] Started container init-sleep
18:17:34 Still waiting to schedule task
18:17:34 'init-build-26-bt2j2-ktzkh-1w9j4' is offline
18:23:34 Created Pod: kubernetes dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk
18:23:34 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Scheduled] Successfully assigned dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk to node-10
18:23:35 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Pulling] Pulling image "ubuntu"
18:23:36 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Pulled] Successfully pulled image "ubuntu"
18:23:36 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Created] Created container init-sleep
18:23:36 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Started] Started container init-sleep
{code}
Jenkins Instance Logs:
{code:java}
2021-10-13 01:17:24.724+0000 [id=323] INFO hudson.slaves.NodeProvisioner#update: init-build-26-bt2j2-ktzkh-1w9j4 provisioning successfully completed. We have now 3 computer(s)
2021-10-13 01:17:24.818+0000 [id=322] INFO o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4
2021-10-13 01:23:25.196+0000 [id=383] INFO o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent init-build-26-bt2j2-ktzkh-1w9j4
2021-10-13 01:23:25.202+0000 [id=383] INFO o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer init-build-26-bt2j2-ktzkh-1w9j4
Disconnected computer init-build-26-bt2j2-ktzkh-1w9j4

{code}
We are expecting the pod shouldn't get re-created for the amount of time specified by slaveConnectTimeout.

Is there something we missed?

New: We have a use case where we need to set an initContainer to do some operations before the actual job container starts.

The initContainer operation would take around 20-30 mins.

If I add slaveConnectTimeout and set it to 3600, the kubernetes plugin ends up terminating the agent (after 6 mins) and re-creates a new pod.

Are we setting it correctly?

Pipeline Spec:
{code:java}
podTemplate(cloud:'kubernetes',
slaveConnectTimeout: '3600',
yaml : """
apiVersion: v1
kind: Pod
metadata:
  labels:
    some-label: some-label-value
spec:
  initContainers:
  - name: init-sleep
    image: ubuntu
    command: ['bash', '-c', 'while true; do echo "hello world"; sleep 10; done']
  containers:
  - name: job-container
    image: ubuntu:latest
    command:
    - cat
    restartPolicy: Never
    backoffLimit: 4
    tty: true
  nodeSelector:
    kubernetes.io/os: linux
""" ) {
  node(POD_LABEL) {
    stage('work') {
        container('job-container') {
            sh 'apt update'
        }
    }
  }
}

{code}
Logs:
{code:java}
18:17:24 Created Pod: kubernetes dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4
18:17:24 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Scheduled] Successfully assigned dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4 to node-1
18:17:25 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Pulling] Pulling image "ubuntu"
18:17:26 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Pulled] Successfully pulled image "ubuntu"
18:17:26 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Created] Created container init-sleep
18:17:27 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4][Started] Started container init-sleep
18:17:34 Still waiting to schedule task
18:17:34 'init-build-26-bt2j2-ktzkh-1w9j4' is offline
18:23:34 Created Pod: kubernetes dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk
18:23:34 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Scheduled] Successfully assigned dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk to node-10
18:23:35 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Pulling] Pulling image "ubuntu"
18:23:36 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Pulled] Successfully pulled image "ubuntu"
18:23:36 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Created] Created container init-sleep
18:23:36 [Normal][dynamic-updater/init-build-26-bt2j2-ktzkh-0d7pk][Started] Started container init-sleep
{code}
Jenkins Instance Logs:
{code:java}
2021-10-13 01:17:24.724+0000 [id=323] INFO hudson.slaves.NodeProvisioner#update: init-build-26-bt2j2-ktzkh-1w9j4 provisioning successfully completed. We have now 3 computer(s)
2021-10-13 01:17:24.818+0000 [id=322] INFO o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes dynamic-updater/init-build-26-bt2j2-ktzkh-1w9j4
2021-10-13 01:23:25.196+0000 [id=383] INFO o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent init-build-26-bt2j2-ktzkh-1w9j4
2021-10-13 01:23:25.202+0000 [id=383] INFO o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer init-build-26-bt2j2-ktzkh-1w9j4
Disconnected computer init-build-26-bt2j2-ktzkh-1w9j4

{code}
We are expecting the pod shouldn't get re-created for the amount of time specified by slaveConnectTimeout.

Is there something we missed? What triggers "terminate" for the pod?

Andreas Müller made changes - 2021-11-18 13:56

Priority

Original: Minor [ 4 ]

New: Critical [ 2 ]

Andreas Müller made changes - 2021-11-19 14:15

Assignee

New: Vincent Latombe [ vlatombe ]

Jenkins

Details

Description

Attachments

Activity

People

Dates