-
Bug
-
Resolution: Fixed
-
Blocker
-
None
-
-
kubernetes 3900.va_dce992317b_4
I have separate namespaces for the Jenkins controller and agent pods in the same Kubernetes cluster. When I start Jenkins, I can always successfully run a single job. The following jobs will hang waiting for a pod to provision. Logs contain:
WARNING: Error in provisioning; [snip] Caused by: io.fabric8.kubernetes.client.KubernetesClientException [snip] Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:CONTROLLER_NAMESPACE:CONTROLLER_SERVICEACCOUNT" cannot create resource "pods" in API group "" in the namespace "AGENT_NAMESPACE"
The plugin is using the controller service account in the controller namespace instead of the service account from the credential token.
Things get a bit random at this point. A pod may start successfully but then an error is logged that the controller SA cannot get pods while the job is running. If I wait long enough a pending job might finally get through after 9-10 minutes of waiting and complete. The following job will have errors again.
I've tried hard coding the service account to the pod template, but this has not helped.
Problem combo:
kubernetes-plugin: 3802.vb_b_600831fcb_3
kubernetes-client-api-plugin: 6.3.1-206.v76d3b_6b_14db_b
Kubernetes-credentials-plugin: 0.10.0
After reverting to previous combination things work fine:
kubernetes-plugin: 3743.v1fa_4c724c3b_7
kubernetes-client-api-plugin: 5.12.2-193.v26a_6078f65a_9
Kubernetes-credentials-plugin: 0.9.0
- duplicates
-
-
- Resolved
-
- is duplicated by
-
-
- Reopened
-
-
-
- Closed
-
- links to
[JENKINS-70416] Kubernetes plugin uses controller service account instead of configured credentials
Markus
created issue -
Markus
made changes -
| Description |
Original:
I have separate namespaces for the Jenkins controller and agent pods in the same Kubernetes cluster. When I start Jenkins, I can always successfully run a single job. After this things get a bit random. Logs contain:
{code:java} WARNING: Error in provisioning; [snip] Caused by: io.fabric8.kubernetes.client.KubernetesClientException [snip] Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:CONTROLLER_NAMESPACE:CONTROLLER_SERVICEACCOUNT" cannot create resource "pods" in API group "" in the namespace "AGENT_NAMESPACE"{code} As said, things get a bit random. A pod may start successfully but then an error is logged that the controller SA cannot *get* pods while the job is running. If I wait long enough a pending job might finally get through after 9-10 minutes of waiting and completes. The following job will have errors again. I've tried hard coding the service account to the pod template, but this has not helped. Problem combo: kubernetes-plugin: 3802.vb_b_600831fcb_3 kubernetes-client-api-plugin: 6.3.1-206.v76d3b_6b_14db_b Kubernetes-credentials-plugin: 0.10.0 After reverting to previous combination things work fine: kubernetes-plugin: 3743.v1fa_4c724c3b_7 kubernetes-client-api-plugin: 5.12.2-193.v26a_6078f65a_9 Kubernetes-credentials-plugin: 0.9.0 Updating plugins one by one suggests it's specifically the kubernetes plugin update, not the credentials plugin that causes the error. |
New:
I have separate namespaces for the Jenkins controller and agent pods in the same Kubernetes cluster. When I start Jenkins, I can always successfully run a single job. After this things get a bit random. Logs contain:
{code:java} WARNING: Error in provisioning; [snip] Caused by: io.fabric8.kubernetes.client.KubernetesClientException [snip] Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:CONTROLLER_NAMESPACE:CONTROLLER_SERVICEACCOUNT" cannot create resource "pods" in API group "" in the namespace "AGENT_NAMESPACE"{code} As said, things get a bit random. A pod may start successfully but then an error is logged that the controller SA cannot *get* pods while the job is running. If I wait long enough a pending job might finally get through after 9-10 minutes of waiting and completes. The following job will have errors again. I've tried hard coding the service account to the pod template, but this has not helped. Problem combo: kubernetes-plugin: 3802.vb_b_600831fcb_3 kubernetes-client-api-plugin: 6.3.1-206.v76d3b_6b_14db_b Kubernetes-credentials-plugin: 0.10.0 After reverting to previous combination things work fine: kubernetes-plugin: 3743.v1fa_4c724c3b_7 kubernetes-client-api-plugin: 5.12.2-193.v26a_6078f65a_9 Kubernetes-credentials-plugin: 0.9.0 |
Markus
made changes -
| Description |
Original:
I have separate namespaces for the Jenkins controller and agent pods in the same Kubernetes cluster. When I start Jenkins, I can always successfully run a single job. After this things get a bit random. Logs contain:
{code:java} WARNING: Error in provisioning; [snip] Caused by: io.fabric8.kubernetes.client.KubernetesClientException [snip] Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:CONTROLLER_NAMESPACE:CONTROLLER_SERVICEACCOUNT" cannot create resource "pods" in API group "" in the namespace "AGENT_NAMESPACE"{code} As said, things get a bit random. A pod may start successfully but then an error is logged that the controller SA cannot *get* pods while the job is running. If I wait long enough a pending job might finally get through after 9-10 minutes of waiting and completes. The following job will have errors again. I've tried hard coding the service account to the pod template, but this has not helped. Problem combo: kubernetes-plugin: 3802.vb_b_600831fcb_3 kubernetes-client-api-plugin: 6.3.1-206.v76d3b_6b_14db_b Kubernetes-credentials-plugin: 0.10.0 After reverting to previous combination things work fine: kubernetes-plugin: 3743.v1fa_4c724c3b_7 kubernetes-client-api-plugin: 5.12.2-193.v26a_6078f65a_9 Kubernetes-credentials-plugin: 0.9.0 |
New:
I have separate namespaces for the Jenkins controller and agent pods in the same Kubernetes cluster. When I start Jenkins, I can always successfully run a single job. The following jobs will hang waiting for a pod to provision. Logs contain:
{code:java} WARNING: Error in provisioning; [snip] Caused by: io.fabric8.kubernetes.client.KubernetesClientException [snip] Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:CONTROLLER_NAMESPACE:CONTROLLER_SERVICEACCOUNT" cannot create resource "pods" in API group "" in the namespace "AGENT_NAMESPACE"{code} Things get a bit random at this point. A pod may start successfully but then an error is logged that the controller SA cannot *get* pods while the job is running. If I wait long enough a pending job might finally get through after 9-10 minutes of waiting and completes. The following job will have errors again. I've tried hard coding the service account to the pod template, but this has not helped. Problem combo: kubernetes-plugin: 3802.vb_b_600831fcb_3 kubernetes-client-api-plugin: 6.3.1-206.v76d3b_6b_14db_b Kubernetes-credentials-plugin: 0.10.0 After reverting to previous combination things work fine: kubernetes-plugin: 3743.v1fa_4c724c3b_7 kubernetes-client-api-plugin: 5.12.2-193.v26a_6078f65a_9 Kubernetes-credentials-plugin: 0.9.0 |
Markus
made changes -
| Description |
Original:
I have separate namespaces for the Jenkins controller and agent pods in the same Kubernetes cluster. When I start Jenkins, I can always successfully run a single job. The following jobs will hang waiting for a pod to provision. Logs contain:
{code:java} WARNING: Error in provisioning; [snip] Caused by: io.fabric8.kubernetes.client.KubernetesClientException [snip] Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:CONTROLLER_NAMESPACE:CONTROLLER_SERVICEACCOUNT" cannot create resource "pods" in API group "" in the namespace "AGENT_NAMESPACE"{code} Things get a bit random at this point. A pod may start successfully but then an error is logged that the controller SA cannot *get* pods while the job is running. If I wait long enough a pending job might finally get through after 9-10 minutes of waiting and completes. The following job will have errors again. I've tried hard coding the service account to the pod template, but this has not helped. Problem combo: kubernetes-plugin: 3802.vb_b_600831fcb_3 kubernetes-client-api-plugin: 6.3.1-206.v76d3b_6b_14db_b Kubernetes-credentials-plugin: 0.10.0 After reverting to previous combination things work fine: kubernetes-plugin: 3743.v1fa_4c724c3b_7 kubernetes-client-api-plugin: 5.12.2-193.v26a_6078f65a_9 Kubernetes-credentials-plugin: 0.9.0 |
New:
I have separate namespaces for the Jenkins controller and agent pods in the same Kubernetes cluster. When I start Jenkins, I can always successfully run a single job. The following jobs will hang waiting for a pod to provision. Logs contain:
{code:java} WARNING: Error in provisioning; [snip] Caused by: io.fabric8.kubernetes.client.KubernetesClientException [snip] Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:CONTROLLER_NAMESPACE:CONTROLLER_SERVICEACCOUNT" cannot create resource "pods" in API group "" in the namespace "AGENT_NAMESPACE"{code} The plugin is using the controller service account in the controller namespace instead of the configured agent namespace and service account from the credential token. Things get a bit random at this point. A pod may start successfully but then an error is logged that the controller SA cannot *get* pods while the job is running. If I wait long enough a pending job might finally get through after 9-10 minutes of waiting and complete. The following job will have errors again. I've tried hard coding the service account to the pod template, but this has not helped. Problem combo: kubernetes-plugin: 3802.vb_b_600831fcb_3 kubernetes-client-api-plugin: 6.3.1-206.v76d3b_6b_14db_b Kubernetes-credentials-plugin: 0.10.0 After reverting to previous combination things work fine: kubernetes-plugin: 3743.v1fa_4c724c3b_7 kubernetes-client-api-plugin: 5.12.2-193.v26a_6078f65a_9 Kubernetes-credentials-plugin: 0.9.0 |
Markus
made changes -
| Description |
Original:
I have separate namespaces for the Jenkins controller and agent pods in the same Kubernetes cluster. When I start Jenkins, I can always successfully run a single job. The following jobs will hang waiting for a pod to provision. Logs contain:
{code:java} WARNING: Error in provisioning; [snip] Caused by: io.fabric8.kubernetes.client.KubernetesClientException [snip] Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:CONTROLLER_NAMESPACE:CONTROLLER_SERVICEACCOUNT" cannot create resource "pods" in API group "" in the namespace "AGENT_NAMESPACE"{code} The plugin is using the controller service account in the controller namespace instead of the configured agent namespace and service account from the credential token. Things get a bit random at this point. A pod may start successfully but then an error is logged that the controller SA cannot *get* pods while the job is running. If I wait long enough a pending job might finally get through after 9-10 minutes of waiting and complete. The following job will have errors again. I've tried hard coding the service account to the pod template, but this has not helped. Problem combo: kubernetes-plugin: 3802.vb_b_600831fcb_3 kubernetes-client-api-plugin: 6.3.1-206.v76d3b_6b_14db_b Kubernetes-credentials-plugin: 0.10.0 After reverting to previous combination things work fine: kubernetes-plugin: 3743.v1fa_4c724c3b_7 kubernetes-client-api-plugin: 5.12.2-193.v26a_6078f65a_9 Kubernetes-credentials-plugin: 0.9.0 |
New:
I have separate namespaces for the Jenkins controller and agent pods in the same Kubernetes cluster. When I start Jenkins, I can always successfully run a single job. The following jobs will hang waiting for a pod to provision. Logs contain:
{code:java} WARNING: Error in provisioning; [snip] Caused by: io.fabric8.kubernetes.client.KubernetesClientException [snip] Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:CONTROLLER_NAMESPACE:CONTROLLER_SERVICEACCOUNT" cannot create resource "pods" in API group "" in the namespace "AGENT_NAMESPACE"{code} The plugin is using the controller service account in the controller namespace instead of the service account from the credential token. Things get a bit random at this point. A pod may start successfully but then an error is logged that the controller SA cannot *get* pods while the job is running. If I wait long enough a pending job might finally get through after 9-10 minutes of waiting and complete. The following job will have errors again. I've tried hard coding the service account to the pod template, but this has not helped. Problem combo: kubernetes-plugin: 3802.vb_b_600831fcb_3 kubernetes-client-api-plugin: 6.3.1-206.v76d3b_6b_14db_b Kubernetes-credentials-plugin: 0.10.0 After reverting to previous combination things work fine: kubernetes-plugin: 3743.v1fa_4c724c3b_7 kubernetes-client-api-plugin: 5.12.2-193.v26a_6078f65a_9 Kubernetes-credentials-plugin: 0.9.0 |
Jonathan
added a comment - Hi, I have exactly the same error but with a different output (the only difference is that my agents are in different clusters). Failed to start websocket connection: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://api-aks/api/v1/namespaces/jenkins-agents/pods/name-pod. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized. After this error I have to wait 10 minutes to run again. For me the rollback solution doesn't work because I am back to the issue https://issues.jenkins.io/browse/JENKINS-70405
Please can someone help us, we have everything blocked.
Jonathan
added a comment - Hi, I have exactly the same error but with a different output (the only difference is that my agents are in different clusters). Failed to start websocket connection: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://api-aks/api/v1/namespaces/jenkins-agents/pods/name-pod . Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized. After this error I have to wait 10 minutes to run again. For me the rollback solution doesn't work because I am back to the issue https://issues.jenkins.io/browse/JENKINS-70405
Please can someone help us, we have everything blocked. Jonathan
made changes -
| Priority | Original: Major [ 3 ] | New: Blocker [ 1 ] |
maurice ampt
added a comment - We (Rogier and I) have the same as Jonathan, we have also experienced various slightly different messages and also cannot rollback to the versions mentioned in the OP due to the same previous issue (JENKINS-70405).
An error we got mid run when the job entered a
container('...')
step:
Failed to start websocket connection: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://<EKS cluster>/api/v1/namespaces/<namespace>/pods/<pod-name>. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "<pod-name>" is forbidden: User "<serviceaccount>" cannot get resource "pods" in API group "" in the namespace "<namespace>".
An error when attempting to create the pod:
ERROR: Unable to create pod Kubernetes <pod-name>.
Failure executing: POST at: https://<EKS-cluster>/api/v1/namespaces/<namespace>/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "<serviceaccount>" cannot create resource "pods" in API group "" in the namespace "<namespace>".
maurice ampt
added a comment - We (Rogier and I) have the same as Jonathan, we have also experienced various slightly different messages and also cannot rollback to the versions mentioned in the OP due to the same previous issue ( JENKINS-70405 ).
An error we got mid run when the job entered a
container( '...' )
step:
Failed to start websocket connection: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https: //<EKS cluster>/api/v1/namespaces/<namespace>/pods/<pod-name>. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "<pod-name>" is forbidden: User "<serviceaccount>" cannot get resource "pods" in API group "" in the namespace " <namespace>".
An error when attempting to create the pod:
ERROR: Unable to create pod Kubernetes <pod-name>.
Failure executing: POST at: https: //<EKS-cluster>/api/v1/namespaces/<namespace>/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "<serviceaccount>" cannot create resource "pods" in API group "" in the namespace " <namespace>".
We have the exact same issue after upgrading to 3802.vb_b_600831fcb_3