K8s plugin seems to be incapable of intercepting SIGTERM 143 error codes and automatically launching replacement jnlp pods

This issue is archived. You can view it, but you can't modify it. Learn more

XMLWordPrintable

      Kubernetes Plugin Version: 

      3985.vd26d77b_2a_48a_

       

      Expected Outcome

      When draining a k8s worker node on which a `jnlp` jenkins pod is scheduled, the kubernetes plugin should be able to intercept the SIGTERM and proceed to create a replacement jnlp container.

       

      Actual Outcome

      On k8s worker node draining, the SIGTERM is received, but the plugin takes approximately 5 minutes before it attempts to launch a replacement jnlp pod.

       

      Output

      14:42:30  [Pipeline] { (SupWorld) 
14:42:30  [Pipeline] sh 
14:42:39  + echo I like to moveit moveit... 
14:42:39  I like to moveit moveit... 
14:42:40  [Pipeline] sh 
14:42:40  + sleep 3600 
14:43:46  jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Container jnlp was terminated (Exit Code: 143, Reason: Error) 
14:43:46   
14:43:46  - jnlp -- terminated (143) 
14:43:46  jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Pod just failed (Reason: null, Message: null) 
14:43:47   14:43:47  - jnlp -- terminated (143) 14:44:00  Cannot contact test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@702031c3:JNLP4-connect connection from host.amazonaws.com/1.2.3.4:12345": Remote call on JNLP4-connect connection from hostname.compute-1.amazonaws.com/1.2.3.4:57571 failed. The channel is closing down or has closed down 
14:48:46  Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 was deleted; cancelling node body 
14:48:46  Could not connect to test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 to send interrupt signal to process 
14:48:46  [Pipeline] } 
14:48:47  [Pipeline] // stage 
14:48:47  [Pipeline] }
14:48:47  [Pipeline] // node 
14:48:47  [Pipeline] } 
14:48:47  Agent was removed 
14:48:47  org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: wf23fwf-ewf23-324sdfds-2432sdfs-1234123fsdfsd 
14:48:47  Retrying 14:48:47  [Pipeline] { 
14:48:47  [Pipeline] node 
14:48:48  Created Pod: eks-cluster-name jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd 
14:48:54  Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd is provisioned from template test-kubernetes-plugin-pipeline-280a90f0-ac1a-4d50-8b5d-eed594fba22f-4mwmn

       

      Pipeline

      def randomId=UUID.randomUUID().toString()
def podLabel = "test-kubernetes-plugin-pipeline-${randomId}"
node {  
  podTemplate(
    cloud: 'testing',
    idleMinutes: 120,
    label: podLabel,
    nodeSelector: 'instance_type=jenkins',
    namespace: 'jenkins',
    yamlMergeStrategy: merge(),
    containers: [
       containerTemplate(
         name: 'jnlp',
         image: 'jenkins/inbound-agent:3107.v665000b_51092-4-jdk17',
         alwaysPullImage: false,
         ttyEnabled: true,
         workingDir: '/home/jenkins/'
      ),
    ],
    yaml: '''
apiVersion: v1
kind: Pod
spec:
  terminationGracePeriodSeconds: 1800
  priorityClassName: jenkins-pods
  tolerations:
    - key: dedicated
      operator: Equal
      value: jenkins
      effect: NoSchedule
'''  ){
  retry(count: 2, conditions: [agent(), kubernetesAgent(), nonresumable()]) {
     node(podLabel){
        stage('SupWorld'){
          sh 'echo I like to moveit moveit...'
          sh 'sleep 3600'
}      }    }  }}

       

       

       

       

       

       

       

            Assignee:
            Unassigned
            Reporter:
            bob james
            Archiver:
            Jenkins Service Account

              Created:
              Updated:
              Archived: