-
Bug
-
Resolution: Fixed
-
Minor
-
None
-
-
kubernetes 1.24.0
I'm using a Jenkins installation which is run on a standard ec2 instance. This Jenkins instance is supposed to connect to an EKS cluster. EKS cluster auth is done via aws-iam-authenticator which is correctly configured on the machine. I've put all the config in the .kube/config for the jenkins user.
The kubernetes plugin will pick up the k8s config just fine and succeed through all connection tests. It will also run builds just fine for about 15 minutes. At that point, the token issues from aws iam authenticator will be expired and no more pods can be created for my builds. Configuration tests do still work after that, so I guess the plugin creates a new client for those.
The plugin will print the following error message:
2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetes2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetesio.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: [https://....eu-west-1.eks.amazonaws.com/api/v1/namespaces/jenkins/pods?labelSelector=app.kubernetes.io%2Fpart-of%3Djenkins%2Capp.kubernetes.io%2Fcomponent%3Dslave]. Message: Unauthorized! Token may have expired! Please log-in again. Unauthorized. at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:354) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:153) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:620) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:69) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.getActiveSlavePods(KubernetesCloud.java:615) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:590) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:542) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:729) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:332) at hudson.slaves.NodeProvisioner.access$900(NodeProvisioner.java:63) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
- relates to
-
-
- Open
-
- links to
[JENKINS-60105] Kubernetes plugin fails with EKS
Jochen Niebuhr
created issue -
Jochen Niebuhr
made changes -
| Description |
Original:
I'm using a Jenkins installation which is run on a standard ec2 instance. This Jenkins instance is supposed to connect to an EKS cluster. EKS cluster auth is done via aws-iam-authenticator which is correctly configured on the machine. I've put all the config in the .kube/config for the jenkins user. The kubernetes plugin will pick up the k8s config just fine and succeed through all connection tests. It will also run builds just fine for about 15 minutes. At that point, the token issues from aws iam authenticator will be expired and no more pods can be created for my builds. Configuration tests do still work after that, so I guess the plugin creates a new client for those. The plugin will print the following error message: ``` 2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetes2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetesio.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://296cb72614fca8d61a1b611020af335c.yl4.eu-west-1.eks.amazonaws.com/api/v1/namespaces/jenkins/pods?labelSelector=app.kubernetes.io%2Fpart-of%3Djenkins%2Capp.kubernetes.io%2Fcomponent%3Dslave. Message: Unauthorized! Token may have expired! Please log-in again. Unauthorized. at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:354) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:153) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:620) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:69) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.getActiveSlavePods(KubernetesCloud.java:615) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:590) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:542) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:729) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:332) at hudson.slaves.NodeProvisioner.access$900(NodeProvisioner.java:63) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ``` |
New:
I'm using a Jenkins installation which is run on a standard ec2 instance. This Jenkins instance is supposed to connect to an EKS cluster. EKS cluster auth is done via aws-iam-authenticator which is correctly configured on the machine. I've put all the config in the .kube/config for the jenkins user. The kubernetes plugin will pick up the k8s config just fine and succeed through all connection tests. It will also run builds just fine for about 15 minutes. At that point, the token issues from aws iam authenticator will be expired and no more pods can be created for my builds. Configuration tests do still work after that, so I guess the plugin creates a new client for those. The plugin will print the following error message: {code} 2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetes2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetesio.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: [https://296cb72614fca8d61a1b611020af335c.yl4.eu-west-1.eks.amazonaws.com/api/v1/namespaces/jenkins/pods?labelSelector=app.kubernetes.io%2Fpart-of%3Djenkins%2Capp.kubernetes.io%2Fcomponent%3Dslave]. Message: Unauthorized! Token may have expired! Please log-in again. Unauthorized. at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:354) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:153) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:620) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:69) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.getActiveSlavePods(KubernetesCloud.java:615) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:590) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:542) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:729) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:332) at hudson.slaves.NodeProvisioner.access$900(NodeProvisioner.java:63) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {code} |
Jochen Niebuhr
made changes -
| Description |
Original:
I'm using a Jenkins installation which is run on a standard ec2 instance. This Jenkins instance is supposed to connect to an EKS cluster. EKS cluster auth is done via aws-iam-authenticator which is correctly configured on the machine. I've put all the config in the .kube/config for the jenkins user. The kubernetes plugin will pick up the k8s config just fine and succeed through all connection tests. It will also run builds just fine for about 15 minutes. At that point, the token issues from aws iam authenticator will be expired and no more pods can be created for my builds. Configuration tests do still work after that, so I guess the plugin creates a new client for those. The plugin will print the following error message: {code} 2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetes2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetesio.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: [https://296cb72614fca8d61a1b611020af335c.yl4.eu-west-1.eks.amazonaws.com/api/v1/namespaces/jenkins/pods?labelSelector=app.kubernetes.io%2Fpart-of%3Djenkins%2Capp.kubernetes.io%2Fcomponent%3Dslave]. Message: Unauthorized! Token may have expired! Please log-in again. Unauthorized. at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:354) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:153) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:620) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:69) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.getActiveSlavePods(KubernetesCloud.java:615) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:590) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:542) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:729) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:332) at hudson.slaves.NodeProvisioner.access$900(NodeProvisioner.java:63) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {code} |
New:
I'm using a Jenkins installation which is run on a standard ec2 instance. This Jenkins instance is supposed to connect to an EKS cluster. EKS cluster auth is done via aws-iam-authenticator which is correctly configured on the machine. I've put all the config in the .kube/config for the jenkins user. The kubernetes plugin will pick up the k8s config just fine and succeed through all connection tests. It will also run builds just fine for about 15 minutes. At that point, the token issues from aws iam authenticator will be expired and no more pods can be created for my builds. Configuration tests do still work after that, so I guess the plugin creates a new client for those. The plugin will print the following error message: {code:java} 2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetes2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetesio.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: [https://....eu-west-1.eks.amazonaws.com/api/v1/namespaces/jenkins/pods?labelSelector=app.kubernetes.io%2Fpart-of%3Djenkins%2Capp.kubernetes.io%2Fcomponent%3Dslave]. Message: Unauthorized! Token may have expired! Please log-in again. Unauthorized. at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:354) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:153) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:620) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:69) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.getActiveSlavePods(KubernetesCloud.java:615) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:590) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:542) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:729) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:332) at hudson.slaves.NodeProvisioner.access$900(NodeProvisioner.java:63) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {code} |
Caleb is correct, but as this is a recurring problem, I believe the plugin could use a shorter timeout to avoid special tuning when using EKS.
| Assignee | New: Vincent Latombe [ vlatombe ] |
| Status | Original: Open [ 1 ] | New: In Progress [ 3 ] |
| Status | Original: In Progress [ 3 ] | New: In Review [ 10005 ] |
| Remote Link | New: This issue links to "PR #695 (Web Link)" [ 24330 ] |
| Link | New: This issue relates to JENKINS-59087 [ JENKINS-59087 ] |
jniebuhr I think this is covered in the plugin readme under the EKS section (https://plugins.jenkins.io/kubernetes)
Copy/pasting from that readme for convenience:
Running with a remote Kubernetes Cloud in AWS EKS
EKS enforces authentication to the cluster through aws-iam-authenticator. The token expires after 15 minutes so the kubernetes client cache needs to be set to something below this by setting a java argument, like so:
JAVA_ARGS="-Dorg.csanchez.jenkins.plugins.kubernetes.clients.cacheExpiration=60"