Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58463

Job build failed by "Interrupted while waiting for websocket connection, you should increase the Max connections to Kubernetes API"

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Major
    • Resolution: Fixed
    • kubernetes-plugin
    • None
    • Plugin version: 1.16.3, Centos; Jenkins start with java ops: -Dorg.csanchez.jenkins.plugins.kubernetes.clients.cacheExpiration=2592000
      Kubernetes: AWS EKS

    Description

       Job will failed randomly by message "Interrupted while waiting for websocket connection, you should increase the Max connections to Kubernetes API".

      The metrics total requests reached 64 when the issue happen. 

      It seems in Kubernetes-plugin codes, the k8s client maxRequests is set to 64 by hard code. No configuration can change it. This java parameter also can not change it: kubernetes.max.concurrent.requests.

      This issue will happen even set MaxConcurrentRequestsPerHost to a big number.

      Once change MaxConcurrentRequestsPerHost in jenkins, the issue will disappear some times, but will occur again after running for a period. It seems it due to that the k8s API client recreated when change the configuration.

      Is it possible add a configuration for k8s client maxRequests? 

       

      Attachments

        Issue Links

          Activity

            bibline He Bihong created issue -

            where do you see the hardcoded 64 number ?

            if you set the k8s cloud parameter max Requests Per Host to a value it will be used every time the client is recreated

            csanchez Carlos Sanchez added a comment - where do you see the hardcoded 64 number ? if you set the k8s cloud parameter max Requests Per Host to a value it will be used every time the client is recreated
            csanchez Carlos Sanchez made changes -
            Field Original Value New Value
            Link This issue duplicates JENKINS-40825 [ JENKINS-40825 ]
            jglick Jesse Glick added a comment -

            Possibly a symptom of JENKINS-58290. Please ensure that you are running the latest plugin release before filing a bug report.

            jglick Jesse Glick added a comment - Possibly a symptom of JENKINS-58290 . Please ensure that you are running the latest plugin release before filing a bug report.
            jglick Jesse Glick made changes -
            Link This issue duplicates JENKINS-58290 [ JENKINS-58290 ]
            jglick Jesse Glick made changes -
            Resolution Duplicate [ 3 ]
            Status Open [ 1 ] Resolved [ 5 ]
            bibline He Bihong added a comment - - edited

            The maxConcurrentRequests number 64 I mentioned in this class:  io.fabric8.kubernetes.client.Config

            It defined parameter maxConcurrentRequests and default value is 64, but not changed, even it provides setMaxConcurrentRequests method.

            It also not set in this class when init the Config: org.csanchez.jenkins.plugins.kubernetes.KubernetesFactoryAdapter, but  set MaxConcurrentRequestsPerHost:

            builder.withMaxConcurrentRequestsPerHost(maxRequestsPerHost);

            Is it possible add a configure for maxRequests and set it here:

            builder.withMaxConcurrentRequests(maxRequests);

             

             

             

            bibline He Bihong added a comment - - edited The maxConcurrentRequests number 64 I mentioned in this class:  io.fabric8.kubernetes.client.Config It defined parameter maxConcurrentRequests and default value is 64, but not changed, even it provides setMaxConcurrentRequests method. It also not set in this class when init the Config: org.csanchez.jenkins.plugins.kubernetes.KubernetesFactoryAdapter, but  set MaxConcurrentRequestsPerHost: builder.withMaxConcurrentRequestsPerHost(maxRequestsPerHost); Is it possible add a configure for maxRequests and set it here: builder.withMaxConcurrentRequests(maxRequests);      
            bibline He Bihong made changes -
            Environment Plugin version: 1.16.3, Centos; Jenkins start with java ops: -Dorg.csanchez.jenkins.plugins.kubernetes.clients.cacheExpiration=2592000 Plugin version: 1.16.3, Centos; Jenkins start with java ops: -Dorg.csanchez.jenkins.plugins.kubernetes.clients.cacheExpiration=2592000
            Kubernetes: AWS EKS
            sermelo Sergio Merino added a comment -

            I am reopening the ticket because I am having this issue and bibline is right.  You can change the max requests per host, but given that the default max requests is 64, there is no way of going over that

            sermelo Sergio Merino added a comment - I am reopening the ticket because I am having this issue and bibline is right.  You can change the max requests per host, but given that the default max requests is 64, there is no way of going over that
            sermelo Sergio Merino made changes -
            Assignee Carlos Sanchez [ csanchez ]
            Resolution Duplicate [ 3 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            shott85 David Schott made changes -
            Assignee David Schott [ shott85 ]
            shott85 David Schott added a comment - https://github.com/jenkinsci/kubernetes-plugin/pull/861  allows us to go over 64
            shott85 David Schott added a comment - - edited

            The PR build has passed and the artifact is available here if anyone wants to validate the new behavior.

            shott85 David Schott added a comment - - edited The PR build has passed and the artifact is available here  if anyone wants to validate the new behavior.
            shott85 David Schott made changes -
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Fixed but Unreleased [ 10203 ]
            allan_burdajewicz Allan BURDAJEWICZ made changes -
            Attachment dumpK8sClientsRequests.groovy [ 52996 ]

            For those impacted, would you be able to run a script like dumpK8sClientsRequests.groovy under Manage Jenkins > Script Console to check what kind of requests are queuing and monopolizing the k8s clients dispatchers ?

            The fix brought here will help raise the limit and work around the problem but I think it is still valuable to investigate what causes the number or concurrent requests to reach high numbers.

            allan_burdajewicz Allan BURDAJEWICZ added a comment - For those impacted, would you be able to run a script like dumpK8sClientsRequests.groovy under Manage Jenkins > Script Console to check what kind of requests are queuing and monopolizing the k8s clients dispatchers ? The fix brought here will help raise the limit and work around the problem but I think it is still valuable to investigate what causes the number or concurrent requests to reach high numbers.

            shott85 Should we mark this a solved (ability to set the max concurrent requests released in 1.27.3) ?

            allan_burdajewicz Allan BURDAJEWICZ added a comment - shott85 Should we mark this a solved (ability to set the max concurrent requests released in 1.27.3 ) ?
            akloss Alec added a comment -

            What's the best way to get this to folks?  My output is about 400 lines.

            akloss Alec added a comment - What's the best way to get this to folks?  My output is about 400 lines.
            shott85 David Schott added a comment -

            Released in 1.27.3

            shott85 David Schott added a comment - Released in 1.27.3
            shott85 David Schott made changes -
            Released As https://github.com/jenkinsci/kubernetes-plugin/releases/tag/kubernetes-1.27.3
            Status Fixed but Unreleased [ 10203 ] Resolved [ 5 ]
            shott85 David Schott added a comment -

            akloss if you have a file to share I think you can attach it to the ticket, or use https://gist.github.com (or similar) if you wish.

            shott85 David Schott added a comment - akloss  if you have a file to share I think you can attach it to the ticket, or use https://gist.github.com  (or similar) if you wish.
            akloss Alec made changes -
            Attachment dumpK8sClientsRequests-out.txt [ 53627 ]
            akloss Alec added a comment -

            Attached.  I replaced parts of a bunch of the names with X to obfuscate the job names.  I hope that's not too disorienting.

            akloss Alec added a comment - Attached.  I replaced parts of a bunch of the names with X to obfuscate the job names.  I hope that's not too disorienting.
            akloss Alec added a comment -

            How large a number should we expect to set this?  The helm chart for Jenkins has it semi-hard-coded at 32, which doesn't seem sufficient for us.

            akloss Alec added a comment - How large a number should we expect to set this?  The helm chart for Jenkins has it semi-hard-coded at 32, which doesn't seem sufficient for us.
            akloss Alec made changes -
            Description Job will failed randomly by message "Interrupted while waiting for websocket connection, you should increase the Max connections to Kubernetes API".

            The metrics total requests reached 64 when the issue happen. 

            It seems in Kubernetes-plugin codes, the k8s client maxRequests is set to 64 by hard code. No configuration can change it. This java parameter also can not change it: kubernetes.max.concurrent.requests.

            This issue will happen even set MaxConcurrentRequestsPerHost to a big number.

            Once change MaxConcurrentRequestsPerHost in jenkins, the issue will disappear some times, but will occur again after running for a period. It seems it due to that the k8s API client recreated when change the configuration.

            Is it possible add a configuration for k8s client maxRequests? 

             
             Job will failed randomly by message "Interrupted while waiting for websocket connection, you should increase the Max connections to Kubernetes API".

            The metrics total requests reached 64 when the issue happen. 

            It seems in Kubernetes-plugin codes, the k8s client maxRequests is set to 64 by hard code. No configuration can change it. This java parameter also can not change it: kubernetes.max.concurrent.requests.

            This issue will happen even set MaxConcurrentRequestsPerHost to a big number.

            Once change MaxConcurrentRequestsPerHost in jenkins, the issue will disappear some times, but will occur again after running for a period. It seems it due to that the k8s API client recreated when change the configuration.

            Is it possible add a configuration for k8s client maxRequests? 

             

            akloss what version of kubernetes plugin are you using ?
            Maybe shott85 can confirm but I think that those /pods?fieldSelector and /events?fieldSelector are watches that wait for pods to be ready and maybe connected: https://github.com/jenkinsci/kubernetes-plugin/blob/kubernetes-1.28.4/src/main/java/org/csanchez/jenkins/plugins/kubernetes/KubernetesLauncher.java#L158-L163. Like if the provisionning of many agents was triggered at once. If such build storm is expected in the environment, then raising the request might be the best option.

            allan_burdajewicz Allan BURDAJEWICZ added a comment - akloss what version of kubernetes plugin are you using ? Maybe shott85 can confirm but I think that those /pods?fieldSelector and /events?fieldSelector are watches that wait for pods to be ready and maybe connected: https://github.com/jenkinsci/kubernetes-plugin/blob/kubernetes-1.28.4/src/main/java/org/csanchez/jenkins/plugins/kubernetes/KubernetesLauncher.java#L158-L163 . Like if the provisionning of many agents was triggered at once. If such build storm is expected in the environment, then raising the request might be the best option.
            akloss Alec added a comment -

            We're using 1.27.4.  It seems unfortunate that even though we have the build concurrency limit set to 50, we need more connections than that.  This is a new Jenkins building a handful of existing repositories, some of which have many branches.

            akloss Alec added a comment - We're using 1.27.4.  It seems unfortunate that even though we have the build concurrency limit set to 50, we need more connections than that.  This is a new Jenkins building a handful of existing repositories, some of which have many branches.

            People

              shott85 David Schott
              bibline He Bihong
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: