Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-71875

K8s plugin seems to be incapable of intercepting SIGTERM 143 error codes and automatically launching replacement jnlp pods

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Blocker Blocker
    • kubernetes-plugin
    • None

      Kubernetes Plugin Version: 

      3985.vd26d77b_2a_48a_ 

       

      Expected Outcome

      When draining a k8s worker node on which a `jnlp` jenkins pod is scheduled, the kubernetes plugin should be able to intercept the SIGTERM and proceed to create a replacement jnlp container.

       

      Actual Outcome

      On k8s worker node draining, the SIGTERM is received, but the plugin takes approximately 5 minutes before it attempts to launch a replacement jnlp pod.

       

      Output

      14:42:30  [Pipeline] { (SupWorld) 
      14:42:30  [Pipeline] sh 
      14:42:39  + echo I like to moveit moveit... 
      14:42:39  I like to moveit moveit... 
      14:42:40  [Pipeline] sh 
      14:42:40  + sleep 3600 
      14:43:46  jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Container jnlp was terminated (Exit Code: 143, Reason: Error) 
      14:43:46   
      14:43:46  - jnlp -- terminated (143) 
      14:43:46  jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 Pod just failed (Reason: null, Message: null) 
      14:43:47   14:43:47  - jnlp -- terminated (143) 14:44:00  Cannot contact test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@702031c3:JNLP4-connect connection from host.amazonaws.com/1.2.3.4:12345": Remote call on JNLP4-connect connection from hostname.compute-1.amazonaws.com/1.2.3.4:57571 failed. The channel is closing down or has closed down 
      14:48:46  Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 was deleted; cancelling node body 
      14:48:46  Could not connect to test-kubernetes-plugin-pipeline-280a90f0-ac1a-j7kn3 to send interrupt signal to process 
      14:48:46  [Pipeline] } 
      14:48:47  [Pipeline] // stage 
      14:48:47  [Pipeline] }
      14:48:47  [Pipeline] // node 
      14:48:47  [Pipeline] } 
      14:48:47  Agent was removed 
      14:48:47  org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: wf23fwf-ewf23-324sdfds-2432sdfs-1234123fsdfsd 
      14:48:47  Retrying 14:48:47  [Pipeline] { 
      14:48:47  [Pipeline] node 
      14:48:48  Created Pod: eks-cluster-name jenkins-agents/test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd 
      14:48:54  Agent test-kubernetes-plugin-pipeline-280a90f0-ac1a-14vtd is provisioned from template test-kubernetes-plugin-pipeline-280a90f0-ac1a-4d50-8b5d-eed594fba22f-4mwmn
      
        

       

      Pipeline

      def randomId=UUID.randomUUID().toString()
      def podLabel = "test-kubernetes-plugin-pipeline-${randomId}"
      node {  
        podTemplate(
          cloud: 'testing',
          idleMinutes: 120,
          label: podLabel,
          nodeSelector: 'instance_type=jenkins',
          namespace: 'jenkins',
          yamlMergeStrategy: merge(),
          containers: [
             containerTemplate(
               name: 'jnlp',
               image: 'jenkins/inbound-agent:3107.v665000b_51092-4-jdk17',
               alwaysPullImage: false,
               ttyEnabled: true,
               workingDir: '/home/jenkins/'
            ),
          ],
          yaml: '''
      apiVersion: v1
      kind: Pod
      spec:
        terminationGracePeriodSeconds: 1800
        priorityClassName: jenkins-pods
        tolerations:
          - key: dedicated
            operator: Equal
            value: jenkins
            effect: NoSchedule
      '''  ){
        retry(count: 2, conditions: [agent(), kubernetesAgent(), nonresumable()]) {
           node(podLabel){
              stage('SupWorld'){
                sh 'echo I like to moveit moveit...'
                sh 'sleep 3600'
      }      }    }  }}
      
      
      

       

       

       

       

       

       

       

          [JENKINS-71875] K8s plugin seems to be incapable of intercepting SIGTERM 143 error codes and automatically launching replacement jnlp pods

          There are no comments yet on this issue.

            Unassigned Unassigned
            rbob bob james
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: