Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65501

high queue lock contention when provisioning large number of k8s nodes

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: core, kubernetes-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.277.3
      Kubernetes Plugin 1.29.2
    • Similar Issues:
    • Released As:
      2.294

      Description

      When requesting many nodes at once, provisioning locks Queue and prevents regular calls to maintain. Jobs are stuck for minutes.

      In our case we have jobs which request 100+ nodes of different type from k8s. Each REST call to k8s api takes ~2sec. All of them are executed within one withLock which basically blocks everything else from happening on jenkins for that time. To make it worse it seems it then recurses down and does the same again.

      It even gets worse when cluster is at high load and pods can not be scheduled anymore, then it seems like waiting for the pod startup timeout also adds to the time.

      As soon as the nodes are available or load decreases, calls to maintain get back to normal levels.

        Attachments

          Activity

          Hide
          raihaan Raihaan Shouhell added a comment -

          Recently some work has been put in to reduce lock contention when adding nodes have you tried this on 2.289 or 2.288?

          Show
          raihaan Raihaan Shouhell added a comment - Recently some work has been put in to reduce lock contention when adding nodes have you tried this on 2.289 or 2.288?
          Hide
          raihaan Raihaan Shouhell added a comment -

          Jenkins 2.294 and onwards do not lock the queue at all when adding nodes

          Show
          raihaan Raihaan Shouhell added a comment - Jenkins 2.294 and onwards do not lock the queue at all when adding nodes

            People

            Assignee:
            raihaan Raihaan Shouhell
            Reporter:
            scddev Dietmar Scheidl
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: