Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-71389

Kubernetes Plugin consumes a lot of threads when managing many Kubernetes Clouds

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • kubernetes-plugin
    • None

      The Kubernetes Plugin spins up a lot of Okhttp threads when many Kubernetes clouds are defined. Even when there is no activity.

      Quickly testing this with the current latest version 3937.vd7b_82db_e347b_, 100 Kubernetes clouds results in 400 Okhttp threads, without build activity.

      Kubernetes plugin keeps a cache of one k8s client per Kubernetes cloud as per https://github.com/jenkinsci/kubernetes-plugin/blob/3937.vd7b_82db_e347b_/src/main/java/org/csanchez/jenkins/plugins/kubernetes/KubernetesClientProvider.java#L54-L55. And per my findings, each client has 2 Okhttp Dispatcher thread (not sure why 2 dispatchers) and 2 other threads (one websocket connection and one http2 connection) at all time.
      Then when we see activity on the controllers, more threads are spun up obviously.

      While managing 100s of k8s clouds sounds like a lot, it is viable requirement: K8s cloud may be restricted / associated to folder so teams can only be given access to specific k8s clouds that are configured globally.

      Therefore I think it qualifies as a scalability / performance problem.
      So this seems like a scalability problem.

          [JENKINS-71389] Kubernetes Plugin consumes a lot of threads when managing many Kubernetes Clouds

          The Okhttp Implementation might be greedy. Since k8s client 6.x, jetty and native java are other alternatives. Maybe those alternatives could be tested at some point. Also note that at the moment, k8s client okhttp implementation uses okhttp 3...

          Maybe another solution is to manage an ExecutorService in the Kubernetes plugin that we pass to those clients. Since https://github.com/fabric8io/kubernetes-client/blob/v6.5.1/CHANGELOG.md#note-breaking-changes-1, maxConcurrentRequest is not used anymore anyway. But I am actually not sure if it would work well in the case of Okhttp. I think it would still spun up the Dispatcher at least.

          Allan BURDAJEWICZ added a comment - The Okhttp Implementation might be greedy. Since k8s client 6.x, jetty and native java are other alternatives. Maybe those alternatives could be tested at some point. Also note that at the moment, k8s client okhttp implementation uses okhttp 3... Maybe another solution is to manage an ExecutorService in the Kubernetes plugin that we pass to those clients. Since https://github.com/fabric8io/kubernetes-client/blob/v6.5.1/CHANGELOG.md#note-breaking-changes-1 , maxConcurrentRequest is not used anymore anyway. But I am actually not sure if it would work well in the case of Okhttp. I think it would still spun up the Dispatcher at least.

            Unassigned Unassigned
            allan_burdajewicz Allan BURDAJEWICZ
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: