Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58463

Job build failed by "Interrupted while waiting for websocket connection, you should increase the Max connections to Kubernetes API"

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: kubernetes-plugin
    • Labels:
      None
    • Environment:
      Plugin version: 1.16.3, Centos; Jenkins start with java ops: -Dorg.csanchez.jenkins.plugins.kubernetes.clients.cacheExpiration=2592000
      Kubernetes: AWS EKS
    • Similar Issues:

      Description

       Job will failed randomly by message "Interrupted while waiting for websocket connection, you should increase the Max connections to Kubernetes API".

      The metrics total requests reached 64 when the issue happen. 

      It seems in Kubernetes-plugin codes, the k8s client maxRequests is set to 64 by hard code. No configuration can change it. This java parameter also can not change it: kubernetes.max.concurrent.requests.

      This issue will happen even set MaxConcurrentRequestsPerHost to a big number.

      Once change MaxConcurrentRequestsPerHost in jenkins, the issue will disappear some times, but will occur again after running for a period. It seems it due to that the k8s API client recreated when change the configuration.

      Is it possible add a configuration for k8s client maxRequests? 

       

        Attachments

          Issue Links

            Activity

            Hide
            shott85 David Schott added a comment -

            Alec if you have a file to share I think you can attach it to the ticket, or use https://gist.github.com (or similar) if you wish.

            Show
            shott85 David Schott added a comment - Alec  if you have a file to share I think you can attach it to the ticket, or use https://gist.github.com  (or similar) if you wish.
            Hide
            akloss Alec added a comment -

            Attached.  I replaced parts of a bunch of the names with X to obfuscate the job names.  I hope that's not too disorienting.

            Show
            akloss Alec added a comment - Attached.  I replaced parts of a bunch of the names with X to obfuscate the job names.  I hope that's not too disorienting.
            Hide
            akloss Alec added a comment -

            How large a number should we expect to set this?  The helm chart for Jenkins has it semi-hard-coded at 32, which doesn't seem sufficient for us.

            Show
            akloss Alec added a comment - How large a number should we expect to set this?  The helm chart for Jenkins has it semi-hard-coded at 32, which doesn't seem sufficient for us.
            Hide
            allan_burdajewicz Allan BURDAJEWICZ added a comment -

            Alec what version of kubernetes plugin are you using ?
            Maybe David Schott can confirm but I think that those /pods?fieldSelector and /events?fieldSelector are watches that wait for pods to be ready and maybe connected: https://github.com/jenkinsci/kubernetes-plugin/blob/kubernetes-1.28.4/src/main/java/org/csanchez/jenkins/plugins/kubernetes/KubernetesLauncher.java#L158-L163. Like if the provisionning of many agents was triggered at once. If such build storm is expected in the environment, then raising the request might be the best option.

            Show
            allan_burdajewicz Allan BURDAJEWICZ added a comment - Alec what version of kubernetes plugin are you using ? Maybe David Schott can confirm but I think that those /pods?fieldSelector and /events?fieldSelector are watches that wait for pods to be ready and maybe connected: https://github.com/jenkinsci/kubernetes-plugin/blob/kubernetes-1.28.4/src/main/java/org/csanchez/jenkins/plugins/kubernetes/KubernetesLauncher.java#L158-L163 . Like if the provisionning of many agents was triggered at once. If such build storm is expected in the environment, then raising the request might be the best option.
            Hide
            akloss Alec added a comment -

            We're using 1.27.4.  It seems unfortunate that even though we have the build concurrency limit set to 50, we need more connections than that.  This is a new Jenkins building a handful of existing repositories, some of which have many branches.

            Show
            akloss Alec added a comment - We're using 1.27.4.  It seems unfortunate that even though we have the build concurrency limit set to 50, we need more connections than that.  This is a new Jenkins building a handful of existing repositories, some of which have many branches.

              People

              Assignee:
              shott85 David Schott
              Reporter:
              bibline He Bihong
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: