Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47062

Kubernetes template instance cap not respected when multiple templates have same label

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • kubernetes-plugin
    • None
    • Jenkins 2.73.1 kubernetes-plugin 1.0

      If there are two templates with different names but the same labels with a maximum of 1 pod each the scheduling system seems to only allow 1 pod to be spun up instead of 2.

      Kubernetes (and thus openshift) does not seem to support dynamic persistent volume claims. Hence all pods from the same template will share the same persistent volume claim. To work around this it is suggested to create a different template (replication controller or deployment config) for each pvc with the replication count being 1. It won't be sufficient to simply use subpathing on a single pvc for all replicated pods due to performance concerns.

      I have done this in the config:
      template1:
      name: maven
      labels: mavenlabel
      Time in minutes to retain slave when idle: 1 min
      max pods: 1

      template2:
      name: maven2
      labels: mavenlabel
      Time in minutes to retain slave when idle: 1 min
      max pods: 1

      I have two jobs:
      kube1 restrict to label "mavenlabel"
      sleep for 30 seconds

      kube2 restrict to label "mavenlabel"
      sleep for 30 seconds

      After launching kube1 it creates a pod from the maven template and sleeps. Launching kube2 ends up waiting for the pod from kube1 to finish instead of creating another one from the "maven2" template. .

      (pending—Waiting for next available executor on maven-2c3qv)
      

          [JENKINS-47062] Kubernetes template instance cap not respected when multiple templates have same label

          Mark R added a comment -

          Code may have to be changed to support multiple templates using the same label. Another use case that this could come up is if there are subsets of node labels used. Example label combos:

          • linux, docker
          • linux, java
          • linux, nodejs

          https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/PodTemplateUtils.java#L194

          Mark R added a comment - Code may have to be changed to support multiple templates using the same label. Another use case that this could come up is if there are subsets of node labels used. Example label combos: linux, docker linux, java linux, nodejs https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/PodTemplateUtils.java#L194

          Carlos Sanchez added a comment - please provide the logs from https://github.com/jenkinsci/kubernetes-plugin#debugging

          Mark R added a comment -

          Attached log

          Mark R added a comment - Attached log

          that log only shows one agent running

          Carlos Sanchez added a comment - that log only shows one agent running

          Mark R added a comment - - edited

          Correct. It should end up launching another slave node because there are two templates with the label "mavenlabel". However only 1 gets launched causing the other job to wait in a queue.

          Mark R added a comment - - edited Correct. It should end up launching another slave node because there are two templates with the label "mavenlabel". However only 1 gets launched causing the other job to wait in a queue.

          what the log shows is that the plugin never gets a request for more than one agent, so it is probably a misconfiguration in jenkins core

          have you checked https://github.com/jenkinsci/kubernetes-plugin#over-provisioning-flags ?

          Carlos Sanchez added a comment - what the log shows is that the plugin never gets a request for more than one agent, so it is probably a misconfiguration in jenkins core have you checked https://github.com/jenkinsci/kubernetes-plugin#over-provisioning-flags ?

          Mark R added a comment -

          After using command line args:

          -Dhudson.model.LoadStatistics.decay=0.2 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
          

          The kubernetes plugin provisioner is now called (see log2) jkubeplugin2.log

          It does not function correctly however. Notice this:

          Sep 22, 2017 1:59:17 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
          Template instance cap of 1 reached for template maven, not provisioning: 1 running or errored in namespace test1-jenkins with label mavenlabel
          ...
          Sep 22, 2017 1:59:17 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
          Template instance cap of 1 reached for template maven2, not provisioning: 1 running or errored in namespace test1-jenkins with label mavenlabel
          

          The problem is both templates have exactly the same labels.

          {jenkins=slave, jenkins/mavenlabel=true}
          

          The namedList variable here does not account for which template a pod comes from. The namedList for maven is 1 pod (which is correct). The namedList of template "maven2" also returns 1 pod when it should instead return 0. Hence an extra pod is not spun up due to the false positive for the maven2 template.

          Mark R added a comment - After using command line args: -Dhudson.model.LoadStatistics.decay=0.2 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 The kubernetes plugin provisioner is now called (see log2) jkubeplugin2.log It does not function correctly however. Notice this: Sep 22, 2017 1:59:17 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave Template instance cap of 1 reached for template maven, not provisioning: 1 running or errored in namespace test1-jenkins with label mavenlabel ... Sep 22, 2017 1:59:17 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave Template instance cap of 1 reached for template maven2, not provisioning: 1 running or errored in namespace test1-jenkins with label mavenlabel The problem is both templates have exactly the same labels. {jenkins=slave, jenkins/mavenlabel=true} The namedList variable here does not account for which template a pod comes from. The namedList for maven is 1 pod (which is correct). The namedList of template "maven2" also returns 1 pod when it should instead return 0. Hence an extra pod is not spun up due to the false positive for the maven2 template.

            Unassigned Unassigned
            mrichar2 Mark R
            Votes:
            3 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: