Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-70416

Kubernetes plugin uses controller service account instead of configured credentials

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • kubernetes-plugin
    • None
    • kubernetes 3900.va_dce992317b_4

      I have separate namespaces for the Jenkins controller and agent pods in the same Kubernetes cluster. When I start Jenkins, I can always successfully run a single job. The following jobs will hang waiting for a pod to provision. Logs contain:

      WARNING: Error in provisioning; [snip]
Caused by: io.fabric8.kubernetes.client.KubernetesClientException [snip] Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:CONTROLLER_NAMESPACE:CONTROLLER_SERVICEACCOUNT" cannot create resource "pods" in API group "" in the namespace "AGENT_NAMESPACE"

      The plugin is using the controller service account in the controller namespace instead of the service account from the credential token. 

      Things get a bit random at this point. A pod may start successfully but then an error is logged that the controller SA cannot get pods while the job is running. If I wait long enough a pending job might finally get through after 9-10 minutes of waiting and complete. The following job will have errors again.

      I've tried hard coding the service account to the pod template, but this has not helped.

      Problem combo:

      kubernetes-plugin: 3802.vb_b_600831fcb_3
      kubernetes-client-api-plugin: 6.3.1-206.v76d3b_6b_14db_b
      Kubernetes-credentials-plugin: 0.10.0

      After reverting to previous combination things work fine:

      kubernetes-plugin: 3743.v1fa_4c724c3b_7
      kubernetes-client-api-plugin: 5.12.2-193.v26a_6078f65a_9
      Kubernetes-credentials-plugin: 0.9.0

          [JENKINS-70416] Kubernetes plugin uses controller service account instead of configured credentials

          Jonathan added a comment - - edited

          we are in AKS with 1.24.6, and we have the same error with JDK 11 and 17, actually, we are in 17.

          Jonathan added a comment - - edited we are in AKS with 1.24.6, and we have the same error with JDK 11 and 17, actually, we are in 17.

          maurice ampt added a comment -

          s7an_it EKS 1.21, Jenkins java.version: 11.0.17

          maurice ampt added a comment - s7an_it EKS 1.21, Jenkins java.version: 11.0.17

          Jonathan added a comment -

          Any news? or any way to be able to use jenkins we are blocked.

          Jonathan added a comment - Any news? or any way to be able to use jenkins we are blocked.

          Dmytro added a comment - - edited

          Java 11.07, jenkins version 2.375.2 (latest jenkins lts docker image). Linode Kubernetes (kubernetes versions 1.23 and 1.24 on two different clusters). Same issue when running slave pods in remote cluster. On original cluster without configured auth everything's OK.

          Dmytro added a comment - - edited Java 11.07, jenkins version 2.375.2 (latest jenkins lts docker image). Linode Kubernetes (kubernetes versions 1.23 and 1.24 on two different clusters). Same issue when running slave pods in remote cluster. On original cluster without configured auth everything's OK.

          Miguel added a comment -

          Hi, we have the same problem with AKS 1.25.4, java jdk17, is there any fixed?

          Miguel added a comment - Hi, we have the same problem with AKS 1.25.4, java jdk17, is there any fixed?

          Pablo added a comment -

          Hello. We have been blocked for 1 week with this error. Can someone help us?

          Many thanks

          Pablo added a comment - Hello. We have been blocked for 1 week with this error. Can someone help us? Many thanks

          Jonathan added a comment -

          we have found the latest stable version for us. To fix this temporarily until they provide a stable version it is necessary to revert to previous versions of the following plugins, using the advanced settings in plugins:

          https://updates.jenkins-ci.org/download/plugins/kubernetes/3743.v1fa_4c724c3b_7/kubernetes.hpi
          https://updates.jenkins-ci.org/download/plugins/kubernetes-credentials/0.9.0/kubernetes-credentials.hpi
          https://updates.jenkins-ci.org/download/plugins/kubernetes-client-api/5.12.2-193.v26a_6078f65a_9/kubernetes-client-api.hpi

          Jonathan added a comment - we have found the latest stable version for us. To fix this temporarily until they provide a stable version it is necessary to revert to previous versions of the following plugins, using the advanced settings in plugins: https://updates.jenkins-ci.org/download/plugins/kubernetes/3743.v1fa_4c724c3b_7/kubernetes.hpi https://updates.jenkins-ci.org/download/plugins/kubernetes-credentials/0.9.0/kubernetes-credentials.hpi https://updates.jenkins-ci.org/download/plugins/kubernetes-client-api/5.12.2-193.v26a_6078f65a_9/kubernetes-client-api.hpi

          Vincent Latombe added a comment -

          Dupe of JENKINS-70493. This is fixed with the latest version.

          Vincent Latombe added a comment - Dupe of JENKINS-70493 . This is fixed with the latest version.

          Allan BURDAJEWICZ added a comment - - edited

          vlatombe I think this is still a problem. I reproduced this issue today. Agents in separate namespaces but in the same cluster, and using one service account per namespace. For each namespace you create a Kubernetes Cloud and a specific Secret Text credentials for the namespaced SA. But you leave Kubernetes URl and Certificate empty since it should not be needed as we are in the same cluster.

          In https://issues.jenkins.io/browse/JENKINS-70493 we fixed the problem with a conditional that if the Kubernetes URL is provided, then we do not use the autoconfigure. That fix the problem for remote cluster authentication. Here the scenario is a multi-namespace in-cluster. So you should not need to provide the Kubernetes URL or Certificate, but only the auth to connect to other namespaces. I think that in such a scenario, we need to rely on the autoconfigure() in the ConfigBuilder first - to kind of auto-populate what we can - and then deactivate the auto-configure if the auth is provided. So that we do not go through the refresh https://github.com/fabric8io/kubernetes-client/blob/v6.4.1/kubernetes-client-api/src/main/java/io/fabric8/kubernetes/client/Config.java#L639. WDYT ?

          We leverage the autoconfigure flag to populate the Config in its constructor, but then turn it off if auth is provided to prevent the automated refresh that we do not want in this case.

          Allan BURDAJEWICZ added a comment - - edited vlatombe I think this is still a problem. I reproduced this issue today. Agents in separate namespaces but in the same cluster, and using one service account per namespace . For each namespace you create a Kubernetes Cloud and a specific Secret Text credentials for the namespaced SA. But you leave Kubernetes URl and Certificate empty since it should not be needed as we are in the same cluster. In https://issues.jenkins.io/browse/JENKINS-70493 we fixed the problem with a conditional that if the Kubernetes URL is provided, then we do not use the autoconfigure. That fix the problem for remote cluster authentication. Here the scenario is a multi-namespace in-cluster. So you should not need to provide the Kubernetes URL or Certificate, but only the auth to connect to other namespaces. I think that in such a scenario, we need to rely on the autoconfigure() in the ConfigBuilder first - to kind of auto-populate what we can - and then deactivate the auto-configure if the auth is provided. So that we do not go through the refresh https://github.com/fabric8io/kubernetes-client/blob/v6.4.1/kubernetes-client-api/src/main/java/io/fabric8/kubernetes/client/Config.java#L639 . WDYT ? We leverage the autoconfigure flag to populate the Config in its constructor, but then turn it off if auth is provided to prevent the automated refresh that we do not want in this case.

          Allan BURDAJEWICZ added a comment -

          Proposed https://github.com/jenkinsci/kubernetes-plugin/pull/1337

          Allan BURDAJEWICZ added a comment - Proposed https://github.com/jenkinsci/kubernetes-plugin/pull/1337

            allan_burdajewicz Allan BURDAJEWICZ
            markusr Markus
            Votes:
            10 Vote for this issue
            Watchers:
            17 Start watching this issue

              Created:
              Updated:
              Resolved: