-
Bug
-
Resolution: Fixed
-
Blocker
-
None
-
-
kubernetes 3900.va_dce992317b_4
I have separate namespaces for the Jenkins controller and agent pods in the same Kubernetes cluster. When I start Jenkins, I can always successfully run a single job. The following jobs will hang waiting for a pod to provision. Logs contain:
WARNING: Error in provisioning; [snip] Caused by: io.fabric8.kubernetes.client.KubernetesClientException [snip] Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:CONTROLLER_NAMESPACE:CONTROLLER_SERVICEACCOUNT" cannot create resource "pods" in API group "" in the namespace "AGENT_NAMESPACE"
The plugin is using the controller service account in the controller namespace instead of the service account from the credential token.
Things get a bit random at this point. A pod may start successfully but then an error is logged that the controller SA cannot get pods while the job is running. If I wait long enough a pending job might finally get through after 9-10 minutes of waiting and complete. The following job will have errors again.
I've tried hard coding the service account to the pod template, but this has not helped.
Problem combo:
kubernetes-plugin: 3802.vb_b_600831fcb_3
kubernetes-client-api-plugin: 6.3.1-206.v76d3b_6b_14db_b
Kubernetes-credentials-plugin: 0.10.0
After reverting to previous combination things work fine:
kubernetes-plugin: 3743.v1fa_4c724c3b_7
kubernetes-client-api-plugin: 5.12.2-193.v26a_6078f65a_9
Kubernetes-credentials-plugin: 0.9.0
- duplicates
-
-
- Resolved
-
- is duplicated by
-
-
- Reopened
-
-
-
- Closed
-
- links to
[JENKINS-70416] Kubernetes plugin uses controller service account instead of configured credentials
Markus
created issue -
Markus
made changes -
| Description |
Original:
I have separate namespaces for the Jenkins controller and agent pods in the same Kubernetes cluster. When I start Jenkins, I can always successfully run a single job. After this things get a bit random. Logs contain:
{code:java} WARNING: Error in provisioning; [snip] Caused by: io.fabric8.kubernetes.client.KubernetesClientException [snip] Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:CONTROLLER_NAMESPACE:CONTROLLER_SERVICEACCOUNT" cannot create resource "pods" in API group "" in the namespace "AGENT_NAMESPACE"{code} As said, things get a bit random. A pod may start successfully but then an error is logged that the controller SA cannot *get* pods while the job is running. If I wait long enough a pending job might finally get through after 9-10 minutes of waiting and completes. The following job will have errors again. I've tried hard coding the service account to the pod template, but this has not helped. Problem combo: kubernetes-plugin: 3802.vb_b_600831fcb_3 kubernetes-client-api-plugin: 6.3.1-206.v76d3b_6b_14db_b Kubernetes-credentials-plugin: 0.10.0 After reverting to previous combination things work fine: kubernetes-plugin: 3743.v1fa_4c724c3b_7 kubernetes-client-api-plugin: 5.12.2-193.v26a_6078f65a_9 Kubernetes-credentials-plugin: 0.9.0 Updating plugins one by one suggests it's specifically the kubernetes plugin update, not the credentials plugin that causes the error. |
New:
I have separate namespaces for the Jenkins controller and agent pods in the same Kubernetes cluster. When I start Jenkins, I can always successfully run a single job. After this things get a bit random. Logs contain:
{code:java} WARNING: Error in provisioning; [snip] Caused by: io.fabric8.kubernetes.client.KubernetesClientException [snip] Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:CONTROLLER_NAMESPACE:CONTROLLER_SERVICEACCOUNT" cannot create resource "pods" in API group "" in the namespace "AGENT_NAMESPACE"{code} As said, things get a bit random. A pod may start successfully but then an error is logged that the controller SA cannot *get* pods while the job is running. If I wait long enough a pending job might finally get through after 9-10 minutes of waiting and completes. The following job will have errors again. I've tried hard coding the service account to the pod template, but this has not helped. Problem combo: kubernetes-plugin: 3802.vb_b_600831fcb_3 kubernetes-client-api-plugin: 6.3.1-206.v76d3b_6b_14db_b Kubernetes-credentials-plugin: 0.10.0 After reverting to previous combination things work fine: kubernetes-plugin: 3743.v1fa_4c724c3b_7 kubernetes-client-api-plugin: 5.12.2-193.v26a_6078f65a_9 Kubernetes-credentials-plugin: 0.9.0 |
Markus
made changes -
| Description |
Original:
I have separate namespaces for the Jenkins controller and agent pods in the same Kubernetes cluster. When I start Jenkins, I can always successfully run a single job. After this things get a bit random. Logs contain:
{code:java} WARNING: Error in provisioning; [snip] Caused by: io.fabric8.kubernetes.client.KubernetesClientException [snip] Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:CONTROLLER_NAMESPACE:CONTROLLER_SERVICEACCOUNT" cannot create resource "pods" in API group "" in the namespace "AGENT_NAMESPACE"{code} As said, things get a bit random. A pod may start successfully but then an error is logged that the controller SA cannot *get* pods while the job is running. If I wait long enough a pending job might finally get through after 9-10 minutes of waiting and completes. The following job will have errors again. I've tried hard coding the service account to the pod template, but this has not helped. Problem combo: kubernetes-plugin: 3802.vb_b_600831fcb_3 kubernetes-client-api-plugin: 6.3.1-206.v76d3b_6b_14db_b Kubernetes-credentials-plugin: 0.10.0 After reverting to previous combination things work fine: kubernetes-plugin: 3743.v1fa_4c724c3b_7 kubernetes-client-api-plugin: 5.12.2-193.v26a_6078f65a_9 Kubernetes-credentials-plugin: 0.9.0 |
New:
I have separate namespaces for the Jenkins controller and agent pods in the same Kubernetes cluster. When I start Jenkins, I can always successfully run a single job. The following jobs will hang waiting for a pod to provision. Logs contain:
{code:java} WARNING: Error in provisioning; [snip] Caused by: io.fabric8.kubernetes.client.KubernetesClientException [snip] Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:CONTROLLER_NAMESPACE:CONTROLLER_SERVICEACCOUNT" cannot create resource "pods" in API group "" in the namespace "AGENT_NAMESPACE"{code} Things get a bit random at this point. A pod may start successfully but then an error is logged that the controller SA cannot *get* pods while the job is running. If I wait long enough a pending job might finally get through after 9-10 minutes of waiting and completes. The following job will have errors again. I've tried hard coding the service account to the pod template, but this has not helped. Problem combo: kubernetes-plugin: 3802.vb_b_600831fcb_3 kubernetes-client-api-plugin: 6.3.1-206.v76d3b_6b_14db_b Kubernetes-credentials-plugin: 0.10.0 After reverting to previous combination things work fine: kubernetes-plugin: 3743.v1fa_4c724c3b_7 kubernetes-client-api-plugin: 5.12.2-193.v26a_6078f65a_9 Kubernetes-credentials-plugin: 0.9.0 |
Markus
made changes -
| Description |
Original:
I have separate namespaces for the Jenkins controller and agent pods in the same Kubernetes cluster. When I start Jenkins, I can always successfully run a single job. The following jobs will hang waiting for a pod to provision. Logs contain:
{code:java} WARNING: Error in provisioning; [snip] Caused by: io.fabric8.kubernetes.client.KubernetesClientException [snip] Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:CONTROLLER_NAMESPACE:CONTROLLER_SERVICEACCOUNT" cannot create resource "pods" in API group "" in the namespace "AGENT_NAMESPACE"{code} Things get a bit random at this point. A pod may start successfully but then an error is logged that the controller SA cannot *get* pods while the job is running. If I wait long enough a pending job might finally get through after 9-10 minutes of waiting and completes. The following job will have errors again. I've tried hard coding the service account to the pod template, but this has not helped. Problem combo: kubernetes-plugin: 3802.vb_b_600831fcb_3 kubernetes-client-api-plugin: 6.3.1-206.v76d3b_6b_14db_b Kubernetes-credentials-plugin: 0.10.0 After reverting to previous combination things work fine: kubernetes-plugin: 3743.v1fa_4c724c3b_7 kubernetes-client-api-plugin: 5.12.2-193.v26a_6078f65a_9 Kubernetes-credentials-plugin: 0.9.0 |
New:
I have separate namespaces for the Jenkins controller and agent pods in the same Kubernetes cluster. When I start Jenkins, I can always successfully run a single job. The following jobs will hang waiting for a pod to provision. Logs contain:
{code:java} WARNING: Error in provisioning; [snip] Caused by: io.fabric8.kubernetes.client.KubernetesClientException [snip] Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:CONTROLLER_NAMESPACE:CONTROLLER_SERVICEACCOUNT" cannot create resource "pods" in API group "" in the namespace "AGENT_NAMESPACE"{code} The plugin is using the controller service account in the controller namespace instead of the configured agent namespace and service account from the credential token. Things get a bit random at this point. A pod may start successfully but then an error is logged that the controller SA cannot *get* pods while the job is running. If I wait long enough a pending job might finally get through after 9-10 minutes of waiting and complete. The following job will have errors again. I've tried hard coding the service account to the pod template, but this has not helped. Problem combo: kubernetes-plugin: 3802.vb_b_600831fcb_3 kubernetes-client-api-plugin: 6.3.1-206.v76d3b_6b_14db_b Kubernetes-credentials-plugin: 0.10.0 After reverting to previous combination things work fine: kubernetes-plugin: 3743.v1fa_4c724c3b_7 kubernetes-client-api-plugin: 5.12.2-193.v26a_6078f65a_9 Kubernetes-credentials-plugin: 0.9.0 |
Markus
made changes -
| Description |
Original:
I have separate namespaces for the Jenkins controller and agent pods in the same Kubernetes cluster. When I start Jenkins, I can always successfully run a single job. The following jobs will hang waiting for a pod to provision. Logs contain:
{code:java} WARNING: Error in provisioning; [snip] Caused by: io.fabric8.kubernetes.client.KubernetesClientException [snip] Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:CONTROLLER_NAMESPACE:CONTROLLER_SERVICEACCOUNT" cannot create resource "pods" in API group "" in the namespace "AGENT_NAMESPACE"{code} The plugin is using the controller service account in the controller namespace instead of the configured agent namespace and service account from the credential token. Things get a bit random at this point. A pod may start successfully but then an error is logged that the controller SA cannot *get* pods while the job is running. If I wait long enough a pending job might finally get through after 9-10 minutes of waiting and complete. The following job will have errors again. I've tried hard coding the service account to the pod template, but this has not helped. Problem combo: kubernetes-plugin: 3802.vb_b_600831fcb_3 kubernetes-client-api-plugin: 6.3.1-206.v76d3b_6b_14db_b Kubernetes-credentials-plugin: 0.10.0 After reverting to previous combination things work fine: kubernetes-plugin: 3743.v1fa_4c724c3b_7 kubernetes-client-api-plugin: 5.12.2-193.v26a_6078f65a_9 Kubernetes-credentials-plugin: 0.9.0 |
New:
I have separate namespaces for the Jenkins controller and agent pods in the same Kubernetes cluster. When I start Jenkins, I can always successfully run a single job. The following jobs will hang waiting for a pod to provision. Logs contain:
{code:java} WARNING: Error in provisioning; [snip] Caused by: io.fabric8.kubernetes.client.KubernetesClientException [snip] Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:CONTROLLER_NAMESPACE:CONTROLLER_SERVICEACCOUNT" cannot create resource "pods" in API group "" in the namespace "AGENT_NAMESPACE"{code} The plugin is using the controller service account in the controller namespace instead of the service account from the credential token. Things get a bit random at this point. A pod may start successfully but then an error is logged that the controller SA cannot *get* pods while the job is running. If I wait long enough a pending job might finally get through after 9-10 minutes of waiting and complete. The following job will have errors again. I've tried hard coding the service account to the pod template, but this has not helped. Problem combo: kubernetes-plugin: 3802.vb_b_600831fcb_3 kubernetes-client-api-plugin: 6.3.1-206.v76d3b_6b_14db_b Kubernetes-credentials-plugin: 0.10.0 After reverting to previous combination things work fine: kubernetes-plugin: 3743.v1fa_4c724c3b_7 kubernetes-client-api-plugin: 5.12.2-193.v26a_6078f65a_9 Kubernetes-credentials-plugin: 0.9.0 |
Jonathan
made changes -
| Priority | Original: Major [ 3 ] | New: Blocker [ 1 ] |
| Link |
New:
This issue is duplicated by |
| Link | New: This issue is duplicated by JENKINS-70436 [ JENKINS-70436 ] |
| Link |
New:
This issue duplicates |
| Resolution | New: Duplicate [ 3 ] | |
| Status | Original: Open [ 1 ] | New: Closed [ 6 ] |
Allan BURDAJEWICZ
made changes -
| Assignee | New: Allan BURDAJEWICZ [ allan_burdajewicz ] | |
| Resolution | Original: Duplicate [ 3 ] | |
| Status | Original: Closed [ 6 ] | New: Reopened [ 4 ] |
