-
Bug
-
Resolution: Fixed
-
Minor
-
None
-
-
kubernetes 1.24.0
I'm using a Jenkins installation which is run on a standard ec2 instance. This Jenkins instance is supposed to connect to an EKS cluster. EKS cluster auth is done via aws-iam-authenticator which is correctly configured on the machine. I've put all the config in the .kube/config for the jenkins user.
The kubernetes plugin will pick up the k8s config just fine and succeed through all connection tests. It will also run builds just fine for about 15 minutes. At that point, the token issues from aws iam authenticator will be expired and no more pods can be created for my builds. Configuration tests do still work after that, so I guess the plugin creates a new client for those.
The plugin will print the following error message:
2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetes2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetesio.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: [https://....eu-west-1.eks.amazonaws.com/api/v1/namespaces/jenkins/pods?labelSelector=app.kubernetes.io%2Fpart-of%3Djenkins%2Capp.kubernetes.io%2Fcomponent%3Dslave]. Message: Unauthorized! Token may have expired! Please log-in again. Unauthorized. at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:354) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:153) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:620) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:69) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.getActiveSlavePods(KubernetesCloud.java:615) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:590) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:542) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:729) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:332) at hudson.slaves.NodeProvisioner.access$900(NodeProvisioner.java:63) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
- relates to
-
-
- Open
-
- links to
[JENKINS-60105] Kubernetes plugin fails with EKS
Jochen Niebuhr
created issue -
Jochen Niebuhr
made changes -
| Description |
Original:
I'm using a Jenkins installation which is run on a standard ec2 instance. This Jenkins instance is supposed to connect to an EKS cluster. EKS cluster auth is done via aws-iam-authenticator which is correctly configured on the machine. I've put all the config in the .kube/config for the jenkins user. The kubernetes plugin will pick up the k8s config just fine and succeed through all connection tests. It will also run builds just fine for about 15 minutes. At that point, the token issues from aws iam authenticator will be expired and no more pods can be created for my builds. Configuration tests do still work after that, so I guess the plugin creates a new client for those. The plugin will print the following error message: ``` 2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetes2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetesio.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://296cb72614fca8d61a1b611020af335c.yl4.eu-west-1.eks.amazonaws.com/api/v1/namespaces/jenkins/pods?labelSelector=app.kubernetes.io%2Fpart-of%3Djenkins%2Capp.kubernetes.io%2Fcomponent%3Dslave. Message: Unauthorized! Token may have expired! Please log-in again. Unauthorized. at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:354) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:153) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:620) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:69) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.getActiveSlavePods(KubernetesCloud.java:615) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:590) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:542) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:729) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:332) at hudson.slaves.NodeProvisioner.access$900(NodeProvisioner.java:63) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ``` |
New:
I'm using a Jenkins installation which is run on a standard ec2 instance. This Jenkins instance is supposed to connect to an EKS cluster. EKS cluster auth is done via aws-iam-authenticator which is correctly configured on the machine. I've put all the config in the .kube/config for the jenkins user. The kubernetes plugin will pick up the k8s config just fine and succeed through all connection tests. It will also run builds just fine for about 15 minutes. At that point, the token issues from aws iam authenticator will be expired and no more pods can be created for my builds. Configuration tests do still work after that, so I guess the plugin creates a new client for those. The plugin will print the following error message: {code} 2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetes2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetesio.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: [https://296cb72614fca8d61a1b611020af335c.yl4.eu-west-1.eks.amazonaws.com/api/v1/namespaces/jenkins/pods?labelSelector=app.kubernetes.io%2Fpart-of%3Djenkins%2Capp.kubernetes.io%2Fcomponent%3Dslave]. Message: Unauthorized! Token may have expired! Please log-in again. Unauthorized. at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:354) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:153) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:620) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:69) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.getActiveSlavePods(KubernetesCloud.java:615) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:590) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:542) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:729) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:332) at hudson.slaves.NodeProvisioner.access$900(NodeProvisioner.java:63) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {code} |
Jochen Niebuhr
made changes -
| Description |
Original:
I'm using a Jenkins installation which is run on a standard ec2 instance. This Jenkins instance is supposed to connect to an EKS cluster. EKS cluster auth is done via aws-iam-authenticator which is correctly configured on the machine. I've put all the config in the .kube/config for the jenkins user. The kubernetes plugin will pick up the k8s config just fine and succeed through all connection tests. It will also run builds just fine for about 15 minutes. At that point, the token issues from aws iam authenticator will be expired and no more pods can be created for my builds. Configuration tests do still work after that, so I guess the plugin creates a new client for those. The plugin will print the following error message: {code} 2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetes2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetesio.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: [https://296cb72614fca8d61a1b611020af335c.yl4.eu-west-1.eks.amazonaws.com/api/v1/namespaces/jenkins/pods?labelSelector=app.kubernetes.io%2Fpart-of%3Djenkins%2Capp.kubernetes.io%2Fcomponent%3Dslave]. Message: Unauthorized! Token may have expired! Please log-in again. Unauthorized. at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:354) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:153) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:620) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:69) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.getActiveSlavePods(KubernetesCloud.java:615) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:590) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:542) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:729) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:332) at hudson.slaves.NodeProvisioner.access$900(NodeProvisioner.java:63) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {code} |
New:
I'm using a Jenkins installation which is run on a standard ec2 instance. This Jenkins instance is supposed to connect to an EKS cluster. EKS cluster auth is done via aws-iam-authenticator which is correctly configured on the machine. I've put all the config in the .kube/config for the jenkins user. The kubernetes plugin will pick up the k8s config just fine and succeed through all connection tests. It will also run builds just fine for about 15 minutes. At that point, the token issues from aws iam authenticator will be expired and no more pods can be created for my builds. Configuration tests do still work after that, so I guess the plugin creates a new client for those. The plugin will print the following error message: {code:java} 2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetes2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetesio.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: [https://....eu-west-1.eks.amazonaws.com/api/v1/namespaces/jenkins/pods?labelSelector=app.kubernetes.io%2Fpart-of%3Djenkins%2Capp.kubernetes.io%2Fcomponent%3Dslave]. Message: Unauthorized! Token may have expired! Please log-in again. Unauthorized. at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:354) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:153) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:620) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:69) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.getActiveSlavePods(KubernetesCloud.java:615) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:590) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:542) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:729) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:332) at hudson.slaves.NodeProvisioner.access$900(NodeProvisioner.java:63) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {code} |
| Assignee | New: Vincent Latombe [ vlatombe ] |
| Status | Original: Open [ 1 ] | New: In Progress [ 3 ] |
| Status | Original: In Progress [ 3 ] | New: In Review [ 10005 ] |
| Remote Link | New: This issue links to "PR #695 (Web Link)" [ 24330 ] |
| Link | New: This issue relates to JENKINS-59087 [ JENKINS-59087 ] |
| Released As | New: kubernetes 1.24.0 | |
| Resolution | New: Fixed [ 1 ] | |
| Status | Original: In Review [ 10005 ] | New: Resolved [ 5 ] |