Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67100

KubernetesProvisioningLimits has race condition during initialization

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • kubernetes-plugin
    • None
    • 3546.v6103d89542d6

      After a Jenkins restart, some of our Jenkins instances are stuck, unable to spawn any new Kubernetes agent. As we already faced some issues with the KubernetesProvisioningLimits class (see https://issues.jenkins.io/browse/JENKINS-66484 and https://github.com/jenkinsci/kubernetes-plugin/pull/1028), I re-created a logger for {{org.csanchez.jenkins.plugins.kubernetes.KubernetesProvisioningLimits }}and indeed, the stuck instance were showing "impossible" data, e.g.:

       

      Nov 10, 2021 2:57:09 AM FINEST org.csanchez.jenkins.plugins.kubernetes.KubernetesProvisioningLimits
kubernetes global limit reached: 32/4. Cannot add 1 more!

       

      My best guess is that KubernetesProvisioningLimits initialization phase has some race condition and more specifically when the instance is restarted with some jobs in the queue. It seems that KubernetesSlave are being created for the elements in the queue before the KubernetesProvisioningLimits#init method is invoked.

          [JENKINS-67100] KubernetesProvisioningLimits has race condition during initialization

          Mikaël Barbero created issue -
          Vincent Latombe made changes -
          Assignee New: Vincent Latombe [ vlatombe ]
          Vincent Latombe made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]
          Vincent Latombe made changes -
          Status Original: In Progress [ 3 ] New: In Review [ 10005 ]
          Vincent Latombe made changes -
          Released As New: 3546.v6103d89542d6
          Resolution New: Fixed [ 1 ]
          Status Original: In Review [ 10005 ] New: Resolved [ 5 ]
          Piotrek Zygielo made changes -
          Remote Link New: This issue links to "GH PR #1144 (Web Link)" [ 27442 ]
          Fred G made changes -
          Resolution Original: Fixed [ 1 ]
          Status Original: Resolved [ 5 ] New: Reopened [ 4 ]

            vlatombe Vincent Latombe
            mbarbero Mikaël Barbero
            Votes:
            3 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: