Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56889

Kubernetes creates too many slaves

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major
    • kubernetes-plugin
    • None
    • Jenkins LTS 2.164.1
      Kubernetes plugin 1.14.9

      The kubernetes plugin creates multiple containers for each job in queue.

      Second to wait for pod to be running is set to 600, but seems to be ignored.

      INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Apr 04, 2019 5:46:50 PM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 2 computer(s)
Apr 04, 2019 5:46:50 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Created Pod: kube-molecule-489gg in namespace jenkins-test
Apr 04, 2019 5:47:00 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 1
Apr 04, 2019 5:47:00 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label kube-molecule-pod: Kubernetes Pod Template
Apr 04, 2019 5:47:00 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Apr 04, 2019 5:47:02 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
WARNING: Connection #1379 failed
java.io.EOFException
    at java.io.DataInputStream.readFully(DataInputStream.java:197)
    at java.io.DataInputStream.readFully(DataInputStream.java:169)
    at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:244)

Apr 04, 2019 5:47:03 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #1,380 from /10.248.101.174:40962
Apr 04, 2019 5:47:10 PM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 3 computer(s)
Apr 04, 2019 5:47:10 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Created Pod: kube-molecule-76fxj in namespace jenkins-test
Apr 04, 2019 5:47:20 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 1
Apr 04, 2019 5:47:20 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label kube-molecule-pod: Kubernetes Pod Template
Apr 04, 2019 5:47:20 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Apr 04, 2019 5:47:22 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
WARNING: Connection #1381 failed
java.io.EOFException
    at java.io.DataInputStream.readFully(DataInputStream.java:197)
    at java.io.DataInputStream.readFully(DataInputStream.java:169)
    at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:244)

Apr 04, 2019 5:47:23 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #1,382 from /10.248.101.173:48718
Apr 04, 2019 5:47:30 PM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 4 computer(s)
Apr 04, 2019 5:47:30 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Created Pod: kube-molecule-qsbbr in namespace jenkins-test
Apr 04, 2019 5:47:42 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
WARNING: Connection #1383 failed
java.io.EOFException
    at java.io.DataInputStream.readFully(DataInputStream.java:197)
    at java.io.DataInputStream.readFully(DataInputStream.java:169)
    at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:244)

Apr 04, 2019 5:47:43 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #1,384 from /10.248.101.174:40116
Apr 04, 2019 5:47:47 PM hudson.model.AsyncPeriodicWork$1 run

          [JENKINS-56889] Kubernetes creates too many slaves

          Tony Wen added a comment -

          I'm having the same issue.

          jenkins ver 2.174, Kubernetes plugin 1.15.1.

          Looks like jenkins is provisioning pod continuously using over-provisioning setting. but it is not applicable here with Kubernetes plugin when each job pipeline is configured with unique pod label.  Only 1 pod will be used to execute 'real' command and all others will never get used by any other build.

          Logs

          Apr 19, 2019 8:11:28 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud pro                              vision
INFO: Excess workload after pending Kubernetes agents: 1
Apr 19, 2019 8:11:28 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud pro                              vision
INFO: Template for label k8s-csf-cctf-test-CCTF-1-1: Kubernetes Pod Template
Apr 19, 2019 8:11:28 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesClientPro                              vider createClient
INFO: Created new Kubernetes client: kubernetes io.fabric8.kubernetes.client.Defaul                              tKubernetesClient@49f45b70
Apr 19, 2019 8:11:28 AM okhttp3.internal.platform.Platform log
INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class pat                              h?
Apr 19, 2019 8:11:28 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors                              . Remaining excess workload: 0
Apr 19, 2019 8:11:38 AM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 2 co                              mputer(s)
Apr 19, 2019 8:11:38 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher                               launch
INFO: Created Pod: k8s-csf-cctf-test-cctf-1-1-8g2p1-5tx97 in namespace cctf
Apr 19, 2019 8:11:39 AM okhttp3.internal.platform.Platform log
INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class pat                              h?
Apr 19, 2019 8:11:47 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #1 from /192.168.1.49:53436
Apr 19, 2019 8:11:58 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud pro                              vision
INFO: Excess workload after pending Kubernetes agents: 1
Apr 19, 2019 8:11:58 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud pro                              vision
INFO: Template for label k8s-csf-cctf-test-CCTF-1-1: Kubernetes Pod Template
Apr 19, 2019 8:11:58 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors                              . Remaining excess workload: 0
Apr 19, 2019 8:12:08 AM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 3 co                              mputer(s)
Apr 19, 2019 8:12:08 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud pro                              vision
INFO: Excess workload after pending Kubernetes agents: 1
Apr 19, 2019 8:12:08 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud pro                              vision
INFO: Template for label k8s-csf-cctf-test-CCTF-2-1: Kubernetes Pod Template
Apr 19, 2019 8:12:08 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher                               launch
INFO: Created Pod: k8s-csf-cctf-test-cctf-1-1-8g2p1-7gfkw in namespace cctf
Apr 19, 2019 8:12:08 AM okhttp3.internal.platform.Platform log
INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class pat                              h?
Apr 19, 2019 8:12:08 AM okhttp3.internal.platform.Platform log
INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class pat                              h?
Apr 19, 2019 8:12:08 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors                              . Remaining excess workload: 0
Apr 19, 2019 8:12:13 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #2 from /192.168.1.43:56560
Apr 19, 2019 8:12:18 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud pro                              vision
INFO: Excess workload after pending Kubernetes agents: 1
Apr 19, 2019 8:12:18 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud pro                              vision
INFO: Template for label k8s-csf-cctf-test-CCTF-1-1: Kubernetes Pod Template
Apr 19, 2019 8:12:18 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors                              . Remaining excess workload: 0
Apr 19, 2019 8:12:18 AM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 4 co                              mputer(s)

          Tony Wen added a comment - I'm having the same issue. jenkins ver 2.174, Kubernetes plugin 1.15.1. Looks like jenkins is provisioning pod continuously using over-provisioning setting. but it is not applicable here with Kubernetes plugin when each job pipeline is configured with unique pod label.  Only 1 pod will be used to execute 'real' command and all others will never get used by any other build. Logs Apr 19, 2019 8:11:28 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud pro vision INFO: Excess workload after pending Kubernetes agents: 1 Apr 19, 2019 8:11:28 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud pro vision INFO: Template for label k8s-csf-cctf-test-CCTF-1-1: Kubernetes Pod Template Apr 19, 2019 8:11:28 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesClientPro vider createClient INFO: Created new Kubernetes client: kubernetes io.fabric8.kubernetes.client.Defaul tKubernetesClient@49f45b70 Apr 19, 2019 8:11:28 AM okhttp3.internal.platform.Platform log INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class pat h? Apr 19, 2019 8:11:28 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors . Remaining excess workload: 0 Apr 19, 2019 8:11:38 AM hudson.slaves.NodeProvisioner$2 run INFO: Kubernetes Pod Template provisioning successfully completed. We have now 2 co mputer(s) Apr 19, 2019 8:11:38 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch INFO: Created Pod: k8s-csf-cctf-test-cctf-1-1-8g2p1-5tx97 in namespace cctf Apr 19, 2019 8:11:39 AM okhttp3.internal.platform.Platform log INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class pat h? Apr 19, 2019 8:11:47 AM hudson.TcpSlaveAgentListener$ConnectionHandler run INFO: Accepted JNLP4-connect connection #1 from /192.168.1.49:53436 Apr 19, 2019 8:11:58 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud pro vision INFO: Excess workload after pending Kubernetes agents: 1 Apr 19, 2019 8:11:58 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud pro vision INFO: Template for label k8s-csf-cctf-test-CCTF-1-1: Kubernetes Pod Template Apr 19, 2019 8:11:58 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors . Remaining excess workload: 0 Apr 19, 2019 8:12:08 AM hudson.slaves.NodeProvisioner$2 run INFO: Kubernetes Pod Template provisioning successfully completed. We have now 3 co mputer(s) Apr 19, 2019 8:12:08 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud pro vision INFO: Excess workload after pending Kubernetes agents: 1 Apr 19, 2019 8:12:08 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud pro vision INFO: Template for label k8s-csf-cctf-test-CCTF-2-1: Kubernetes Pod Template Apr 19, 2019 8:12:08 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch INFO: Created Pod: k8s-csf-cctf-test-cctf-1-1-8g2p1-7gfkw in namespace cctf Apr 19, 2019 8:12:08 AM okhttp3.internal.platform.Platform log INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class pat h? Apr 19, 2019 8:12:08 AM okhttp3.internal.platform.Platform log INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class pat h? Apr 19, 2019 8:12:08 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors . Remaining excess workload: 0 Apr 19, 2019 8:12:13 AM hudson.TcpSlaveAgentListener$ConnectionHandler run INFO: Accepted JNLP4-connect connection #2 from /192.168.1.43:56560 Apr 19, 2019 8:12:18 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud pro vision INFO: Excess workload after pending Kubernetes agents: 1 Apr 19, 2019 8:12:18 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud pro vision INFO: Template for label k8s-csf-cctf-test-CCTF-1-1: Kubernetes Pod Template Apr 19, 2019 8:12:18 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors . Remaining excess workload: 0 Apr 19, 2019 8:12:18 AM hudson.slaves.NodeProvisioner$2 run INFO: Kubernetes Pod Template provisioning successfully completed. We have now 4 co mputer(s)

            csanchez Carlos Sanchez
            denraf Raf Nijskens
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: