Details
-
Bug
-
Status: Resolved (View Workflow)
-
Major
-
Resolution: Fixed
-
None
-
Plugin version: 1.16.3, Centos; Jenkins start with java ops: -Dorg.csanchez.jenkins.plugins.kubernetes.clients.cacheExpiration=2592000
Kubernetes: AWS EKS
Description
Job will failed randomly by message "Interrupted while waiting for websocket connection, you should increase the Max connections to Kubernetes API".
The metrics total requests reached 64 when the issue happen.
It seems in Kubernetes-plugin codes, the k8s client maxRequests is set to 64 by hard code. No configuration can change it. This java parameter also can not change it: kubernetes.max.concurrent.requests.
This issue will happen even set MaxConcurrentRequestsPerHost to a big number.
Once change MaxConcurrentRequestsPerHost in jenkins, the issue will disappear some times, but will occur again after running for a period. It seems it due to that the k8s API client recreated when change the configuration.
Is it possible add a configuration for k8s client maxRequests?
Attachments
Issue Links
- duplicates
-
-
- Resolved
-
-
-
- Resolved
-
Activity
Alec
added a comment - Attached. I replaced parts of a bunch of the names with X to obfuscate the job names. I hope that's not too disorienting.
Alec
added a comment - Attached. I replaced parts of a bunch of the names with X to obfuscate the job names. I hope that's not too disorienting. Alec
added a comment - How large a number should we expect to set this? The helm chart for Jenkins has it semi-hard-coded at 32, which doesn't seem sufficient for us.
Alec
added a comment - How large a number should we expect to set this? The helm chart for Jenkins has it semi-hard-coded at 32, which doesn't seem sufficient for us. 
Allan BURDAJEWICZ
added a comment - akloss what version of kubernetes plugin are you using ?
Maybe shott85 can confirm but I think that those /pods?fieldSelector and /events?fieldSelector are watches that wait for pods to be ready and maybe connected: https://github.com/jenkinsci/kubernetes-plugin/blob/kubernetes-1.28.4/src/main/java/org/csanchez/jenkins/plugins/kubernetes/KubernetesLauncher.java#L158-L163. Like if the provisionning of many agents was triggered at once. If such build storm is expected in the environment, then raising the request might be the best option.
Allan BURDAJEWICZ
added a comment - akloss what version of kubernetes plugin are you using ?
Maybe shott85 can confirm but I think that those /pods?fieldSelector and /events?fieldSelector are watches that wait for pods to be ready and maybe connected: https://github.com/jenkinsci/kubernetes-plugin/blob/kubernetes-1.28.4/src/main/java/org/csanchez/jenkins/plugins/kubernetes/KubernetesLauncher.java#L158-L163 . Like if the provisionning of many agents was triggered at once. If such build storm is expected in the environment, then raising the request might be the best option. Alec
added a comment - We're using 1.27.4. It seems unfortunate that even though we have the build concurrency limit set to 50, we need more connections than that. This is a new Jenkins building a handful of existing repositories, some of which have many branches.
Alec
added a comment - We're using 1.27.4. It seems unfortunate that even though we have the build concurrency limit set to 50, we need more connections than that. This is a new Jenkins building a handful of existing repositories, some of which have many branches.
akloss if you have a file to share I think you can attach it to the ticket, or use https://gist.github.com (or similar) if you wish.