[JENKINS-71875] K8s plugin seems to be incapable of intercepting SIGTERM 143 error codes and automatically launching replacement jnlp pods

Type: Bug
Resolution: Unresolved
Priority: Blocker
Component/s: kubernetes-plugin
Labels:
None

Similar Issues:
Powered by SuggestiMate

Show

Kubernetes Plugin Version:

3985.vd26d77b_2a_48a_

Expected Outcome

When draining a k8s worker node on which a `jnlp` jenkins pod is scheduled, the kubernetes plugin should be able to intercept the SIGTERM and proceed to create a replacement jnlp container.

Actual Outcome

On k8s worker node draining, the SIGTERM is received, but the plugin takes approximately 5 minutes before it attempts to launch a replacement jnlp pod.

Output

14:42:30  [Pipeline] { (SupWorld) 
14:42:30  [Pipeline] sh 
14:42:39  + echo I like to moveit moveit... 
14:42:39  I like to moveit moveit... 
14:42:40  [Pipeline] sh 
14:42:40  + sleep 3600 
14:43:46  jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Container jnlp was terminated (Exit Code: 143, Reason: Error) 
14:43:46   
14:43:46  - jnlp -- terminated (143) 
14:43:46  jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Pod just failed (Reason: null, Message: null) 
14:43:47   14:43:47  - jnlp -- terminated (143) 14:44:00  Cannot contact test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@702031c3:JNLP4-connect connection from host.amazonaws.com/1.2.3.4:12345": Remote call on JNLP4-connect connection from hostname.compute-1.amazonaws.com/1.2.3.4:57571 failed. The channel is closing down or has closed down 
14:48:46  Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 was deleted; cancelling node body 
14:48:46  Could not connect to test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 to send interrupt signal to process 
14:48:46  [Pipeline] } 
14:48:47  [Pipeline] // stage 
14:48:47  [Pipeline] }
14:48:47  [Pipeline] // node 
14:48:47  [Pipeline] } 
14:48:47  Agent was removed 
14:48:47  org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: wf23fwf-ewf23-324sdfds-2432sdfs-1234123fsdfsd 
14:48:47  Retrying 14:48:47  [Pipeline] { 
14:48:47  [Pipeline] node 
14:48:48  Created Pod: eks-cluster-name jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd 
14:48:54  Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd is provisioned from template test-kubernetes-plugin-pipeline-280a90f0-ac1a-4d50-8b5d-eed594fba22f-4mwmn

Pipeline

def randomId=UUID.randomUUID().toString()
def podLabel = "test-kubernetes-plugin-pipeline-${randomId}"
node {  
  podTemplate(
    cloud: 'testing',
    idleMinutes: 120,
    label: podLabel,
    nodeSelector: 'instance_type=jenkins',
    namespace: 'jenkins',
    yamlMergeStrategy: merge(),
    containers: [
       containerTemplate(
         name: 'jnlp',
         image: 'jenkins/inbound-agent:3107.v665000b_51092-4-jdk17',
         alwaysPullImage: false,
         ttyEnabled: true,
         workingDir: '/home/jenkins/'
      ),
    ],
    yaml: '''
apiVersion: v1
kind: Pod
spec:
  terminationGracePeriodSeconds: 1800
  priorityClassName: jenkins-pods
  tolerations:
    - key: dedicated
      operator: Equal
      value: jenkins
      effect: NoSchedule
'''  ){
  retry(count: 2, conditions: [agent(), kubernetesAgent(), nonresumable()]) {
     node(podLabel){
        stage('SupWorld'){
          sh 'echo I like to moveit moveit...'
          sh 'sleep 3600'
}      }    }  }}

bob james created issue - 2023-08-22 17:08

bob james made changes - 2023-08-22 17:11

Description

Original: *Expected Outcome*

When draining a k8s worker node on which a `jnlp` jenkins pod is scheduled, the kubernetes plugin should be able to intercept the SIGTERM and proceed to create a replacement jnlp container.

*Actual Outcome*

On k8s worker node draining, the SIGTERM is received, but the plugin takes approximately *5 minutes* before it attempts to launch a replacement jnlp pod.

Output:
```

14:42:30 [Pipeline] { (SupWorld)
14:42:30 [Pipeline] sh
14:42:39 + echo I like to moveit moveit...
14:42:39 I like to moveit moveit...
14:42:40 [Pipeline] sh
14:42:40 + sleep 3600
14:43:46 jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Container jnlp was terminated (Exit Code: 143, Reason: Error)
14:43:46
14:43:46 - jnlp -- terminated (143)
14:43:46 jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Pod just failed (Reason: null, Message: null)
14:43:47
14:43:47 - jnlp -- terminated (143)
14:44:00 Cannot contact test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@702031c3:JNLP4-connect connection from host.amazonaws.com/1.2.3.4:12345": Remote call on JNLP4-connect connection from hostname.compute-1.amazonaws.com/1.2.3.4:57571 failed. The channel is closing down or has closed down
14:48:46 Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 was deleted; cancelling node body
14:48:46 Could not connect to test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 to send interrupt signal to process
14:48:46 [Pipeline] }
14:48:47 [Pipeline] // stage
14:48:47 [Pipeline] }
14:48:47 [Pipeline] // node
14:48:47 [Pipeline] }
14:48:47 Agent was removed
14:48:47 org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: wf23fwf-ewf23-324sdfds-2432sdfs-1234123fsdfsd
14:48:47 Retrying
14:48:47 [Pipeline] {
14:48:47 [Pipeline] node
14:48:48 Created Pod: eks-cluster-name jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd
14:48:54 Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd is provisioned from template test-kubernetes-plugin-pipeline-280a90f0-ac1a-4d50-8b5d-eed594fba22f-4mwmn
```

Pipeline

```
def randomId=UUID.randomUUID().toString()
def podLabel = "test-kubernetes-plugin-pipeline-${randomId}"

node {
podTemplate(
cloud: 'testing',
idleMinutes: 120,
label: podLabel,
nodeSelector: 'instance_type=jenkins',
namespace: 'jenkins',
yamlMergeStrategy: merge(),
containers: [
containerTemplate(
name: 'jnlp',
image: 'jenkins/inbound-agent:3107.v665000b_51092-4-jdk17',
alwaysPullImage: false,
ttyEnabled: true,
workingDir: '/home/jenkins/'
),
],
yaml: '''
apiVersion: v1
kind: Pod
spec:
terminationGracePeriodSeconds: 1800
priorityClassName: jenkins-pods
tolerations:
- key: dedicated
operator: Equal
value: jenkins
effect: NoSchedule
'''
){
retry(count: 2, conditions: [agent(), kubernetesAgent(), nonresumable()]) {
node(podLabel){
stage('SupWorld') {
sh 'echo I like to moveit moveit...'
sh 'sleep 3600'
}
}
}
}
}
```

New: *Expected Outcome*

When draining a k8s worker node on which a `jnlp` jenkins pod is scheduled, the kubernetes plugin should be able to intercept the SIGTERM and proceed to create a replacement jnlp container.

*Actual Outcome*

On k8s worker node draining, the SIGTERM is received, but the plugin takes approximately *5 minutes* before it attempts to launch a replacement jnlp pod.

Output

{code:java}
14:42:30 [Pipeline] { (SupWorld) 14:42:30 [Pipeline] sh 14:42:39 + echo I like to moveit moveit... 14:42:39 I like to moveit moveit... 14:42:40 [Pipeline] sh 14:42:40 + sleep 3600 14:43:46 jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Container jnlp was terminated (Exit Code: 143, Reason: Error) 14:43:46 14:43:46 - jnlp -- terminated (143) 14:43:46 jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Pod just failed (Reason: null, Message: null) 14:43:47 14:43:47 - jnlp -- terminated (143) 14:44:00 Cannot contact test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@702031c3:JNLP4-connect connection from host.amazonaws.com/1.2.3.4:12345": Remote call on JNLP4-connect connection from hostname.compute-1.amazonaws.com/1.2.3.4:57571 failed. The channel is closing down or has closed down 14:48:46 Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 was deleted; cancelling node body 14:48:46 Could not connect to test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 to send interrupt signal to process 14:48:46 [Pipeline] } 14:48:47 [Pipeline] // stage 14:48:47 [Pipeline] } 14:48:47 [Pipeline] // node 14:48:47 [Pipeline] } 14:48:47 Agent was removed 14:48:47 org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: wf23fwf-ewf23-324sdfds-2432sdfs-1234123fsdfsd 14:48:47 Retrying 14:48:47 [Pipeline] { 14:48:47 [Pipeline] node 14:48:48 Created Pod: eks-cluster-name jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd 14:48:54 Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd is provisioned from template test-kubernetes-plugin-pipeline-280a90f0-ac1a-4d50-8b5d-eed594fba22f-4mwmn

{code}

{code:java}
def randomId=UUID.randomUUID().toString()def podLabel = "test-kubernetes-plugin-pipeline-${randomId}"
node { podTemplate( cloud: 'testing', idleMinutes: 120, label: podLabel, nodeSelector: 'instance_type=jenkins', namespace: 'jenkins', yamlMergeStrategy: merge(), containers: [ containerTemplate( name: 'jnlp', image: 'jenkins/inbound-agent:3107.v665000b_51092-4-jdk17', alwaysPullImage: false, ttyEnabled: true, workingDir: '/home/jenkins/' ), ], yaml: ''' apiVersion: v1 kind: Pod spec: terminationGracePeriodSeconds: 1800 priorityClassName: jenkins-pods tolerations: - key: dedicated operator: Equal value: jenkins effect: NoSchedule ''' ){ retry(count: 2, conditions: [agent(), kubernetesAgent(), nonresumable()]) { node(podLabel){ stage('SupWorld') { sh 'echo I like to moveit moveit...' sh 'sleep 3600' } } } }}

{code}

bob james made changes - 2023-08-22 17:12

Description

Original: *Expected Outcome*

When draining a k8s worker node on which a `jnlp` jenkins pod is scheduled, the kubernetes plugin should be able to intercept the SIGTERM and proceed to create a replacement jnlp container.

*Actual Outcome*

On k8s worker node draining, the SIGTERM is received, but the plugin takes approximately *5 minutes* before it attempts to launch a replacement jnlp pod.

Output

{code:java}
14:42:30 [Pipeline] { (SupWorld) 14:42:30 [Pipeline] sh 14:42:39 + echo I like to moveit moveit... 14:42:39 I like to moveit moveit... 14:42:40 [Pipeline] sh 14:42:40 + sleep 3600 14:43:46 jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Container jnlp was terminated (Exit Code: 143, Reason: Error) 14:43:46 14:43:46 - jnlp -- terminated (143) 14:43:46 jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Pod just failed (Reason: null, Message: null) 14:43:47 14:43:47 - jnlp -- terminated (143) 14:44:00 Cannot contact test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@702031c3:JNLP4-connect connection from host.amazonaws.com/1.2.3.4:12345": Remote call on JNLP4-connect connection from hostname.compute-1.amazonaws.com/1.2.3.4:57571 failed. The channel is closing down or has closed down 14:48:46 Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 was deleted; cancelling node body 14:48:46 Could not connect to test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 to send interrupt signal to process 14:48:46 [Pipeline] } 14:48:47 [Pipeline] // stage 14:48:47 [Pipeline] } 14:48:47 [Pipeline] // node 14:48:47 [Pipeline] } 14:48:47 Agent was removed 14:48:47 org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: wf23fwf-ewf23-324sdfds-2432sdfs-1234123fsdfsd 14:48:47 Retrying 14:48:47 [Pipeline] { 14:48:47 [Pipeline] node 14:48:48 Created Pod: eks-cluster-name jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd 14:48:54 Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd is provisioned from template test-kubernetes-plugin-pipeline-280a90f0-ac1a-4d50-8b5d-eed594fba22f-4mwmn

{code}

{code:java}
def randomId=UUID.randomUUID().toString()def podLabel = "test-kubernetes-plugin-pipeline-${randomId}"
node { podTemplate( cloud: 'testing', idleMinutes: 120, label: podLabel, nodeSelector: 'instance_type=jenkins', namespace: 'jenkins', yamlMergeStrategy: merge(), containers: [ containerTemplate( name: 'jnlp', image: 'jenkins/inbound-agent:3107.v665000b_51092-4-jdk17', alwaysPullImage: false, ttyEnabled: true, workingDir: '/home/jenkins/' ), ], yaml: ''' apiVersion: v1 kind: Pod spec: terminationGracePeriodSeconds: 1800 priorityClassName: jenkins-pods tolerations: - key: dedicated operator: Equal value: jenkins effect: NoSchedule ''' ){ retry(count: 2, conditions: [agent(), kubernetesAgent(), nonresumable()]) { node(podLabel){ stage('SupWorld') { sh 'echo I like to moveit moveit...' sh 'sleep 3600' } } } }}

{code}

New: *Expected Outcome*

When draining a k8s worker node on which a `jnlp` jenkins pod is scheduled, the kubernetes plugin should be able to intercept the SIGTERM and proceed to create a replacement jnlp container.

*Actual Outcome*

On k8s worker node draining, the SIGTERM is received, but the plugin takes approximately *5 minutes* before it attempts to launch a replacement jnlp pod.

h2. *Output*
{code:java}
14:42:30 [Pipeline] { (SupWorld) 14:42:30 [Pipeline] sh 14:42:39 + echo I like to moveit moveit... 14:42:39 I like to moveit moveit... 14:42:40 [Pipeline] sh 14:42:40 + sleep 3600 14:43:46 jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Container jnlp was terminated (Exit Code: 143, Reason: Error) 14:43:46 14:43:46 - jnlp -- terminated (143) 14:43:46 jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Pod just failed (Reason: null, Message: null) 14:43:47 14:43:47 - jnlp -- terminated (143) 14:44:00 Cannot contact test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@702031c3:JNLP4-connect connection from host.amazonaws.com/1.2.3.4:12345": Remote call on JNLP4-connect connection from hostname.compute-1.amazonaws.com/1.2.3.4:57571 failed. The channel is closing down or has closed down 14:48:46 Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 was deleted; cancelling node body 14:48:46 Could not connect to test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 to send interrupt signal to process 14:48:46 [Pipeline] } 14:48:47 [Pipeline] // stage 14:48:47 [Pipeline] } 14:48:47 [Pipeline] // node 14:48:47 [Pipeline] } 14:48:47 Agent was removed 14:48:47 org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: wf23fwf-ewf23-324sdfds-2432sdfs-1234123fsdfsd 14:48:47 Retrying 14:48:47 [Pipeline] { 14:48:47 [Pipeline] node 14:48:48 Created Pod: eks-cluster-name jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd 14:48:54 Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd is provisioned from template test-kubernetes-plugin-pipeline-280a90f0-ac1a-4d50-8b5d-eed594fba22f-4mwmn

{code}

h2. *Pipeline*
{code:java}
def randomId=UUID.randomUUID().toString()def podLabel = "test-kubernetes-plugin-pipeline-${randomId}"
node { podTemplate( cloud: 'testing', idleMinutes: 120, label: podLabel, nodeSelector: 'instance_type=jenkins', namespace: 'jenkins', yamlMergeStrategy: merge(), containers: [ containerTemplate( name: 'jnlp', image: 'jenkins/inbound-agent:3107.v665000b_51092-4-jdk17', alwaysPullImage: false, ttyEnabled: true, workingDir: '/home/jenkins/' ), ], yaml: ''' apiVersion: v1 kind: Pod spec: terminationGracePeriodSeconds: 1800 priorityClassName: jenkins-pods tolerations: - key: dedicated operator: Equal value: jenkins effect: NoSchedule ''' ){ retry(count: 2, conditions: [agent(), kubernetesAgent(), nonresumable()]) { node(podLabel){ stage('SupWorld') { sh 'echo I like to moveit moveit...' sh 'sleep 3600' } } } }}

{code}

bob james made changes - 2023-08-22 17:17

Description

Original: *Expected Outcome*

When draining a k8s worker node on which a `jnlp` jenkins pod is scheduled, the kubernetes plugin should be able to intercept the SIGTERM and proceed to create a replacement jnlp container.

*Actual Outcome*

On k8s worker node draining, the SIGTERM is received, but the plugin takes approximately *5 minutes* before it attempts to launch a replacement jnlp pod.

h2. *Output*
{code:java}
14:42:30 [Pipeline] { (SupWorld) 14:42:30 [Pipeline] sh 14:42:39 + echo I like to moveit moveit... 14:42:39 I like to moveit moveit... 14:42:40 [Pipeline] sh 14:42:40 + sleep 3600 14:43:46 jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Container jnlp was terminated (Exit Code: 143, Reason: Error) 14:43:46 14:43:46 - jnlp -- terminated (143) 14:43:46 jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Pod just failed (Reason: null, Message: null) 14:43:47 14:43:47 - jnlp -- terminated (143) 14:44:00 Cannot contact test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@702031c3:JNLP4-connect connection from host.amazonaws.com/1.2.3.4:12345": Remote call on JNLP4-connect connection from hostname.compute-1.amazonaws.com/1.2.3.4:57571 failed. The channel is closing down or has closed down 14:48:46 Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 was deleted; cancelling node body 14:48:46 Could not connect to test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 to send interrupt signal to process 14:48:46 [Pipeline] } 14:48:47 [Pipeline] // stage 14:48:47 [Pipeline] } 14:48:47 [Pipeline] // node 14:48:47 [Pipeline] } 14:48:47 Agent was removed 14:48:47 org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: wf23fwf-ewf23-324sdfds-2432sdfs-1234123fsdfsd 14:48:47 Retrying 14:48:47 [Pipeline] { 14:48:47 [Pipeline] node 14:48:48 Created Pod: eks-cluster-name jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd 14:48:54 Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd is provisioned from template test-kubernetes-plugin-pipeline-280a90f0-ac1a-4d50-8b5d-eed594fba22f-4mwmn

{code}

h2. *Pipeline*
{code:java}
def randomId=UUID.randomUUID().toString()def podLabel = "test-kubernetes-plugin-pipeline-${randomId}"
node { podTemplate( cloud: 'testing', idleMinutes: 120, label: podLabel, nodeSelector: 'instance_type=jenkins', namespace: 'jenkins', yamlMergeStrategy: merge(), containers: [ containerTemplate( name: 'jnlp', image: 'jenkins/inbound-agent:3107.v665000b_51092-4-jdk17', alwaysPullImage: false, ttyEnabled: true, workingDir: '/home/jenkins/' ), ], yaml: ''' apiVersion: v1 kind: Pod spec: terminationGracePeriodSeconds: 1800 priorityClassName: jenkins-pods tolerations: - key: dedicated operator: Equal value: jenkins effect: NoSchedule ''' ){ retry(count: 2, conditions: [agent(), kubernetesAgent(), nonresumable()]) { node(podLabel){ stage('SupWorld') { sh 'echo I like to moveit moveit...' sh 'sleep 3600' } } } }}

{code}

New: *Expected Outcome*

When draining a k8s worker node on which a `jnlp` jenkins pod is scheduled, the kubernetes plugin should be able to intercept the SIGTERM and proceed to create a replacement jnlp container.

*Actual Outcome*

On k8s worker node draining, the SIGTERM is received, but the plugin takes approximately *5 minutes* before it attempts to launch a replacement jnlp pod.

h2. *Output*
{code:java}
14:42:30 [Pipeline] { (SupWorld)
14:42:30 [Pipeline] sh
14:42:39 + echo I like to moveit moveit...
14:42:39 I like to moveit moveit...
14:42:40 [Pipeline] sh
14:42:40 + sleep 3600
14:43:46 jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Container jnlp was terminated (Exit Code: 143, Reason: Error)
14:43:46
14:43:46 - jnlp -- terminated (143)
14:43:46 jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Pod just failed (Reason: null, Message: null)
14:43:47 14:43:47 - jnlp -- terminated (143) 14:44:00 Cannot contact test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@702031c3:JNLP4-connect connection from host.amazonaws.com/1.2.3.4:12345": Remote call on JNLP4-connect connection from hostname.compute-1.amazonaws.com/1.2.3.4:57571 failed. The channel is closing down or has closed down
14:48:46 Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 was deleted; cancelling node body
14:48:46 Could not connect to test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 to send interrupt signal to process
14:48:46 [Pipeline] }
14:48:47 [Pipeline] // stage
14:48:47 [Pipeline] }
14:48:47 [Pipeline] // node
14:48:47 [Pipeline] }
14:48:47 Agent was removed
14:48:47 org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: wf23fwf-ewf23-324sdfds-2432sdfs-1234123fsdfsd
14:48:47 Retrying 14:48:47 [Pipeline] {
14:48:47 [Pipeline] node
14:48:48 Created Pod: eks-cluster-name jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd
14:48:54 Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd is provisioned from template test-kubernetes-plugin-pipeline-280a90f0-ac1a-4d50-8b5d-eed594fba22f-4mwmn

  {code}

h2. *Pipeline*
{code:java}
def randomId=UUID.randomUUID().toString()
def podLabel = "test-kubernetes-plugin-pipeline-${randomId}"
node {
  podTemplate(
    cloud: 'testing',
    idleMinutes: 120,
    label: podLabel,
    nodeSelector: 'instance_type=jenkins',
    namespace: 'jenkins',
    yamlMergeStrategy: merge(),
    containers: [
       containerTemplate(
         name: 'jnlp',
         image: 'jenkins/inbound-agent:3107.v665000b_51092-4-jdk17',
         alwaysPullImage: false,
         ttyEnabled: true,
         workingDir: '/home/jenkins/'
      ),
    ],
    yaml: '''
apiVersion: v1
kind: Pod
spec:
  terminationGracePeriodSeconds: 1800
  priorityClassName: jenkins-pods
  tolerations:
    - key: dedicated
      operator: Equal
      value: jenkins
      effect: NoSchedule
''' ){
  retry(count: 2, conditions: [agent(), kubernetesAgent(), nonresumable()]) {
     node(podLabel){
        stage('SupWorld'){
          sh 'echo I like to moveit moveit...'
          sh 'sleep 3600'
} } } }}

{code}

bob james made changes - 2023-08-22 17:20

Description

Original: *Expected Outcome*

When draining a k8s worker node on which a `jnlp` jenkins pod is scheduled, the kubernetes plugin should be able to intercept the SIGTERM and proceed to create a replacement jnlp container.

*Actual Outcome*

On k8s worker node draining, the SIGTERM is received, but the plugin takes approximately *5 minutes* before it attempts to launch a replacement jnlp pod.

h2. *Output*
{code:java}
14:42:30 [Pipeline] { (SupWorld)
14:42:30 [Pipeline] sh
14:42:39 + echo I like to moveit moveit...
14:42:39 I like to moveit moveit...
14:42:40 [Pipeline] sh
14:42:40 + sleep 3600
14:43:46 jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Container jnlp was terminated (Exit Code: 143, Reason: Error)
14:43:46
14:43:46 - jnlp -- terminated (143)
14:43:46 jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Pod just failed (Reason: null, Message: null)
14:43:47 14:43:47 - jnlp -- terminated (143) 14:44:00 Cannot contact test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@702031c3:JNLP4-connect connection from host.amazonaws.com/1.2.3.4:12345": Remote call on JNLP4-connect connection from hostname.compute-1.amazonaws.com/1.2.3.4:57571 failed. The channel is closing down or has closed down
14:48:46 Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 was deleted; cancelling node body
14:48:46 Could not connect to test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 to send interrupt signal to process
14:48:46 [Pipeline] }
14:48:47 [Pipeline] // stage
14:48:47 [Pipeline] }
14:48:47 [Pipeline] // node
14:48:47 [Pipeline] }
14:48:47 Agent was removed
14:48:47 org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: wf23fwf-ewf23-324sdfds-2432sdfs-1234123fsdfsd
14:48:47 Retrying 14:48:47 [Pipeline] {
14:48:47 [Pipeline] node
14:48:48 Created Pod: eks-cluster-name jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd
14:48:54 Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd is provisioned from template test-kubernetes-plugin-pipeline-280a90f0-ac1a-4d50-8b5d-eed594fba22f-4mwmn

  {code}

h2. *Pipeline*
{code:java}
def randomId=UUID.randomUUID().toString()
def podLabel = "test-kubernetes-plugin-pipeline-${randomId}"
node {
  podTemplate(
    cloud: 'testing',
    idleMinutes: 120,
    label: podLabel,
    nodeSelector: 'instance_type=jenkins',
    namespace: 'jenkins',
    yamlMergeStrategy: merge(),
    containers: [
       containerTemplate(
         name: 'jnlp',
         image: 'jenkins/inbound-agent:3107.v665000b_51092-4-jdk17',
         alwaysPullImage: false,
         ttyEnabled: true,
         workingDir: '/home/jenkins/'
      ),
    ],
    yaml: '''
apiVersion: v1
kind: Pod
spec:
  terminationGracePeriodSeconds: 1800
  priorityClassName: jenkins-pods
  tolerations:
    - key: dedicated
      operator: Equal
      value: jenkins
      effect: NoSchedule
''' ){
  retry(count: 2, conditions: [agent(), kubernetesAgent(), nonresumable()]) {
     node(podLabel){
        stage('SupWorld'){
          sh 'echo I like to moveit moveit...'
          sh 'sleep 3600'
} } } }}

{code}

New: *Kubernetes Plugin Version:*
{code:java}
3985.vd26d77b_2a_48a_ {code}

*Expected Outcome*

When draining a k8s worker node on which a `jnlp` jenkins pod is scheduled, the kubernetes plugin should be able to intercept the SIGTERM and proceed to create a replacement jnlp container.

*Actual Outcome*

On k8s worker node draining, the SIGTERM is received, but the plugin takes approximately *5 minutes* before it attempts to launch a replacement jnlp pod.

h2. *Output*
{code:java}
14:42:30 [Pipeline] { (SupWorld)
14:42:30 [Pipeline] sh
14:42:39 + echo I like to moveit moveit...
14:42:39 I like to moveit moveit...
14:42:40 [Pipeline] sh
14:42:40 + sleep 3600
14:43:46 jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Container jnlp was terminated (Exit Code: 143, Reason: Error)
14:43:46
14:43:46 - jnlp -- terminated (143)
14:43:46 jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Pod just failed (Reason: null, Message: null)
14:43:47 14:43:47 - jnlp -- terminated (143) 14:44:00 Cannot contact test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@702031c3:JNLP4-connect connection from host.amazonaws.com/1.2.3.4:12345": Remote call on JNLP4-connect connection from hostname.compute-1.amazonaws.com/1.2.3.4:57571 failed. The channel is closing down or has closed down
14:48:46 Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 was deleted; cancelling node body
14:48:46 Could not connect to test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 to send interrupt signal to process
14:48:46 [Pipeline] }
14:48:47 [Pipeline] // stage
14:48:47 [Pipeline] }
14:48:47 [Pipeline] // node
14:48:47 [Pipeline] }
14:48:47 Agent was removed
14:48:47 org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: wf23fwf-ewf23-324sdfds-2432sdfs-1234123fsdfsd
14:48:47 Retrying 14:48:47 [Pipeline] {
14:48:47 [Pipeline] node
14:48:48 Created Pod: eks-cluster-name jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd
14:48:54 Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd is provisioned from template test-kubernetes-plugin-pipeline-280a90f0-ac1a-4d50-8b5d-eed594fba22f-4mwmn

  {code}

h2. *Pipeline*
{code:java}
def randomId=UUID.randomUUID().toString()
def podLabel = "test-kubernetes-plugin-pipeline-${randomId}"
node {
  podTemplate(
    cloud: 'testing',
    idleMinutes: 120,
    label: podLabel,
    nodeSelector: 'instance_type=jenkins',
    namespace: 'jenkins',
    yamlMergeStrategy: merge(),
    containers: [
       containerTemplate(
         name: 'jnlp',
         image: 'jenkins/inbound-agent:3107.v665000b_51092-4-jdk17',
         alwaysPullImage: false,
         ttyEnabled: true,
         workingDir: '/home/jenkins/'
      ),
    ],
    yaml: '''
apiVersion: v1
kind: Pod
spec:
  terminationGracePeriodSeconds: 1800
  priorityClassName: jenkins-pods
  tolerations:
    - key: dedicated
      operator: Equal
      value: jenkins
      effect: NoSchedule
''' ){
  retry(count: 2, conditions: [agent(), kubernetesAgent(), nonresumable()]) {
     node(podLabel){
        stage('SupWorld'){
          sh 'echo I like to moveit moveit...'
          sh 'sleep 3600'
} } } }}

{code}

Assignee:: Unassigned

Reporter:: bob james

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Created:: 2023-08-22 17:08

Updated:: 2023-08-22 17:20

Jenkins

Details

Description

Output

Pipeline

Attachments

Activity

People

Dates