Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46444

Slave Node is Left in "offline" while Slave Pod is terminated

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major
    • kubernetes-plugin
    • None

      Using Kubernetes-Plugin 0.12 to start a slave pod in Kubernetes Cluster.

      Upon job completion, slave pod receives 'Kill' signal and pod terminates.

      However, Jenkins master still shows Slave Node as active, and marks it in "offline" status.

      Expected behaviour, Jenkins master removes the node of the terminated slave pod.

      Events Received by Jenkins Slave Pod:

      2017-08-24 23:22:08 -0700 PDT   2017-08-24 23:22:08 -0700 PDT   1         us-west-2-prod-tsshv   Pod                 Normal    Scheduled   default-scheduler   Successfully assigned us-west-2-prod-tsshv to ip-10-7
2-32-128.us-west-2.compute.internal
2017-08-24 23:22:08 -0700 PDT   2017-08-24 23:22:08 -0700 PDT   1         us-west-2-prod-tsshv   Pod       spec.containers{jnlp}   Normal    Pulling   kubelet, ip-10-72-32-128.us-west-2.compute.internal   pulling
 image "jenkinsci/jnlp-slave:alpine"
2017-08-24 23:22:16 -0700 PDT   2017-08-24 23:22:16 -0700 PDT   1         us-west-2-prod-tsshv   Pod       spec.containers{jnlp}   Normal    Pulled    kubelet, ip-10-72-32-128.us-west-2.compute.internal   Success
fully pulled image "jenkinsci/jnlp-slave:alpine"
2017-08-24 23:22:16 -0700 PDT   2017-08-24 23:22:16 -0700 PDT   1         us-west-2-prod-tsshv   Pod       spec.containers{jnlp}   Normal    Created   kubelet, ip-10-72-32-128.us-west-2.compute.internal   Created
 container with docker id 5790ffa32575; Security:[seccomp=unconfined]
2017-08-24 23:22:16 -0700 PDT   2017-08-24 23:22:16 -0700 PDT   1         us-west-2-prod-tsshv   Pod       spec.containers{jnlp}   Normal    Started   kubelet, ip-10-72-32-128.us-west-2.compute.internal   Started
 container with docker id 5790ffa32575
2017-08-24 23:22:21 -0700 PDT   2017-08-24 23:22:21 -0700 PDT   1         us-west-2-prod-tsshv   Pod       spec.containers{jnlp}   Normal    Killing   kubelet, ip-10-72-32-128.us-west-2.compute.internal   Killing

       

      Jenkins Master Log

      Aug 25, 2017 6:22:08 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Spot instances: 1
Aug 25, 2017 6:22:08 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template: Kubernetes Pod Template
Aug 25, 2017 6:22:08 AM okhttp3.internal.platform.Platform log
INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
Aug 25, 2017 6:22:08 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from us-west-2-prod with 1 executors. Remaining excess workload: 0
Aug 25, 2017 6:22:08 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
INFO: Created Pod: us-west-2-prod-tsshv
Aug 25, 2017 6:22:08 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
INFO: Waiting for Pod to be scheduled (0/100): us-west-2-prod-tsshv
Aug 25, 2017 6:22:14 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
INFO: Container is waiting us-west-2-prod-tsshv [jnlp]: ContainerStateWaiting(message=null, reason=ContainerCreating, additionalProperties={})
Aug 25, 2017 6:22:14 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
INFO: Waiting for Pod to be scheduled (1/100): us-west-2-prod-tsshv
Aug 25, 2017 6:22:17 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #50 from /10.4.93.0:53810
Aug 25, 2017 6:22:21 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminating Kubernetes instance for slave us-west-2-prod-tsshv
Aug 25, 2017 6:22:21 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminated Kubernetes instance for slave us-west-2-prod-tsshv
Aug 25, 2017 6:22:21 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Disconnected computer us-west-2-prod-tsshv
Terminated Kubernetes instance for slave us-west-2-prod-tsshv
Aug 25, 2017 6:22:21 AM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
WARNING: Computer.threadPoolForRemoting [#262] for us-west-2-prod-tsshv terminated
java.nio.channels.ClosedChannelException
        at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208)
        at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
        at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
        at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
        at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800)
        at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
        at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:311)
        at hudson.remoting.Channel.close(Channel.java:1304)
        at hudson.remoting.Channel.close(Channel.java:1272)
        at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:708)
        at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:96)
        at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:626)
        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Aug 25, 2017 6:22:22 AM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: test #70 completed: SUCCESS
Aug 25, 2017 6:22:28 AM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 3 computer(s)

       

      After this, Jenkins nodes show the "offline" node

      I can delete the "offline" node manually via UI, upon which master log repors:

      Aug 25, 2017 6:26:47 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminating Kubernetes instance for slave us-west-2-prod-tsshv
Aug 25, 2017 6:26:47 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminated Kubernetes instance for slave us-west-2-prod-tsshv
Terminated Kubernetes instance for slave us-west-2-prod-tsshv
Aug 25, 2017 6:26:47 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Disconnected computer us-west-2-prod-tsshv

        1. image-2017-08-24-23-25-48-535.png
          image-2017-08-24-23-25-48-535.png
          79 kB
        2. image-2019-03-27-16-16-22-818.png
          image-2019-03-27-16-16-22-818.png
          65 kB

            csanchez Carlos Sanchez
            ichekrygin Illya Chekrygin
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: