-
Bug
-
Resolution: Fixed
-
Blocker
-
None
-
Jenkins Controller: 2.375.1
kubernetes Client API Plugin: 6.3.1-206.v76d3b_6b_14db_b
Kubernetes plugin: 3802.vb_b_600831fcb_3
Helm installation on GKE
-
-
Kubernetes 3842.v7ff395ed0cf3
From the Cloud menu I am able to authenticate the connection to the cluster
This functionally was working but it stoped working after various updates
2023-01-25 21:21:15.037+0000 [id=244] WARNING o.c.j.p.k.KubernetesLauncher#launch: Error in provisioning; agent=KubernetesSlave name: test-remote-agent-pod-1-w1n43-6trz0-1x4cm, template=PodTemplate{id='74f3a8f6-23a5-443d-a312-d6f470b81c39', name='test-remote-agent-pod_1-w1n43-6trz0', namespace='jenkins', label='test-remote-agent-pod_1-w1n43', annotations=[PodAnnotation
{key='buildUrl', value='https://gke-test-jenkins.corp.quotient.com/job/test-remote-agent-pod/1/'}, PodAnnotation{key='runUrl', value='job/test-remote-agent-pod/1/'}]}
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://35.186.174.127/api/v1/namespaces/jenkins/pods. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:742)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:722)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:671)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:601)
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$retryWithExponentialBackoff$2(OperationSupport.java:643)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:52)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:133)
Caused: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://35.186.174.127/api/v1/namespaces/jenkins/pods. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.
at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:536)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:570)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:554)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleCreate(OperationSupport.java:347)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:704)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:93)
at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:42)
at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:135)
at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:298)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:48)
at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:82)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-01-25 21:21:15.045+0000 [id=244] INFO o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent test-remote-agent-pod-1-w1n43-6trz0-1x4cm
- is duplicated by
-
-
- Resolved
-
- links to
[JENKINS-70493] Pod not able provision inbound-agent
-Dkubernetes.disable.autoConfig=true
Should be enough to workaround. Really looks like a regression in kubernetes-client 6.x
allan_burdajewicz do you have a clear reproducer ? I'm not able to reproduce systematically on my side (sometimes it works, sometimes it doesn't). Makes me think about some caching issue, but I can't precisely pinpoint it.
Allan BURDAJEWICZ
added a comment - vlatombe I can reproduce pretty consistently when scheduling several builds simultaneously. My scenario was: running a controller locally and configure a Kubernetes Cloud to a remote EKS 1.21. Using either Secret Text or Secret file as credentials for a serviceaccount created in EKS. Locally I where I do have a ~/.kube/config but it does not have any context for that EKS.
Allan BURDAJEWICZ
added a comment - vlatombe I can reproduce pretty consistently when scheduling several builds simultaneously. My scenario was: running a controller locally and configure a Kubernetes Cloud to a remote EKS 1.21. Using either Secret Text or Secret file as credentials for a serviceaccount created in EKS. Locally I where I do have a ~/.kube/config but it does not have any context for that EKS. This PR looks suspicious https://github.com/fabric8io/kubernetes-client/pull/4528
I got down to the root cause, the problem is the `TokenRefreshInterceptor` from kubernetes-client that is refreshing credentials every minute. And if the client is configured with "autoConfiguration", it takes injected configuration instead of the initial token.
Disabling autoConfiguration unless an explicit Kubernetes URL is given seems the most reliable fix.
However, https://github.com/fabric8io/kubernetes-client/pull/4528 fixes the case where you use a kubeconfig as credentials (using secret file).
The problem occurs if you use a token specified as a secret text credentials. In this case, the refresh erases the existing value and uses wrongly the value from autoconfiguration.
https://github.com/jenkinsci/kubernetes-plugin/pull/1302 should be working around the problem from kubernetes-client in a better way than the system property.
Filed https://github.com/fabric8io/kubernetes-client/issues/4802 upstream.
Hello Folks
Not sure in my setup where would I pass/set
-Dkubernetes.auth.tryKubeConfig=false and -Dkubernetes.auth.tryServiceAccount=false
I am using Helm for the Controller installation and JCasC configuration
Also am I to assume the next rev of the Kubernetes plugin will have the fix
Thanks
Samuel Lopez
added a comment - - edited Hello Folks
Not sure in my setup where would I pass/set
-Dkubernetes.auth.tryKubeConfig=false and -Dkubernetes.auth.tryServiceAccount=false
I am using Helm for the Controller installation and JCasC configuration
Also am I to assume the next rev of the Kubernetes plugin will have the fix
Thanks Samuel Lopez
added a comment - Hello Vincent
I have installed the new Kubernetes Plugin Version and initial testing looks good.
Will be doing running more jobs in a bit.
Thank you so very much for helping with this issue
You help and support are very much appreciated
Samuel Lopez
added a comment - Hello Vincent
I have installed the new Kubernetes Plugin Version and initial testing looks good.
Will be doing running more jobs in a bit.
Thank you so very much for helping with this issue
You help and support are very much appreciated 
This seems related to the upgrade to k8s client 6.x. Apparently even though you configure a Kubernetes Cloud with credentials, the k8s client try to infer auth details from the environment (such as the home KUBECONFIG or the in cluster details at /var/run/secrets/kubernetes.io).
Workaround would be to disable the autoconfigure behavior by adding the system property -Dkubernetes.auth.tryKubeConfig=false and -Dkubernetes.auth.tryServiceAccount=false.
So this would break authentication to Kubernetes Cloud that uses a local cluster (that use the serviceaccount attached to your controller). If your controller is running in a pod with a service account used to provisioning agent in the same cluster, you would need to configure that local Kubernetes Cloud with Credentials as well..