Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-57352

Test connection fails and also unable to get K8s pods via Jenkins using service account

      PS: Jenkins is deployed in K8s and not using openshift [do not intend to]

      Provisioning of K8s pods from Kubernetes Jenkins plugin using the my cloud credentials is failing, hence created service account using below command:

      kubectl  -n mynamespace create serviceaccount jenkins
      

      Then after selecting service account authentication credentials tried Test Connection in K8s-plugin cloud configuration page, but connection failed with below error:

      Error testing connection https://api.k8s2.apac.cloud.net:6443: Failure executing: GET at: https://api.k8s2.apac.cloud.net:6443/api/v1/namespaces/mynamespace/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:mynamespace:default" cannot list resource "pods" in API group "" in the namespace "mynamespace": Unexpected user-id: system:serviceaccount:mynamespace:default.
      

      Also when jenkins triggered to run in labeled k8s pods, fails with below error:

      Failed to count the # of live instances on Kubernetes
      io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://api.k8s2.apac.cloud.net:6443/api/v1/namespaces/mynamespace/pods?labelSelector=jenkins%3Dslave. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "mynamespace": Unexpected user-id: system:anonymous.
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:472)
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:409)
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:381)
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:328)
      	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:584)
      	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:49)
      	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:493)
      	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:448)
      	at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715)
      	at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320)
      	at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61)
      	at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809)
      	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
      	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

          [JENKINS-57352] Test connection fails and also unable to get K8s pods via Jenkins using service account

          looks like it is picking the default service account, but I've never seen the "Unexpected user-id" error
          How do you configure the service account in jenkins?

          Carlos Sanchez added a comment - looks like it is picking the default service account, but I've never seen the "Unexpected user-id" error How do you configure the service account in jenkins?

          Harsha GV added a comment -

          Using secret text by adding token string of service account

          Harsha GV added a comment - Using secret text by adding token string of service account

          what kubernetes version? distro? cloud?

          Carlos Sanchez added a comment - what kubernetes version? distro? cloud?

          Harsha GV added a comment -

          K8s:
          Client v1.11.5
          Server v1.13.5

          Jenkins deployed in K8s.

          Also could you please point me to any documentation to get the minimum roles and rolebindings needs to be assigned to a Jenkins service account - used by k8s plugin to to dynamically provision a Jenkins agent, run a single build, then tear-down that agent?

          Harsha GV added a comment - K8s: Client v1.11.5 Server v1.13.5 Jenkins deployed in K8s. Also could you please point me to any documentation to get the minimum roles and rolebindings needs to be assigned to a Jenkins service account - used by k8s plugin to to dynamically provision a Jenkins agent, run a single build, then tear-down that agent?

          Carlos Sanchez added a comment - what kubernetes ? EKS? GKE?...? Roles are here https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/kubernetes/service-account.yml

          Harsha GV added a comment - - edited

          Using GKE kubernetes.

          Service account is created and its roles were applied as defined in the https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/kubernetes/service-account.yml.

          But test connection still throws the error:

          Error testing connection https://api.k8s2.apac.cloud.net:6443: Failure executing: GET at: https://api.k8s2.apac.cloud.net:6443/api/v1/namespaces/mynamespace/pods. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.
          

          Harsha GV added a comment - - edited Using GKE kubernetes. Service account is created and its roles were applied as defined in the https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/kubernetes/service-account.yml . But test connection still throws the error: Error testing connection https: //api.k8s2.apac.cloud.net:6443: Failure executing: GET at: https://api.k8s2.apac.cloud.net:6443/api/v1/namespaces/mynamespace/pods. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.

          Jesse Glick added a comment -

          Probably either a cluster or Jenkins credentials misconfiguration. Unclear how to reproduce from scratch.

          Jesse Glick added a comment - Probably either a cluster or Jenkins credentials misconfiguration. Unclear how to reproduce from scratch.

          I was able to reproduce the issue by using an older version of Kubernetes-plugin running on dedicated EC2 instance and trying to connect to the EKS cluster.

           

          Kubernetes Plugin: 1.13.5

          EKS: 1.13

           

          ```WARNING: Failed to count the # of live instances on Kubernetes
          io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://XXXXX.sk1.eu-west-1.eks.amazonaws.com/api/v1/namespaces/kube-system/pods?labelSelector=jenkins%3Dslave. Message: pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "kube-system". Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=null, kind=pods, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "kube-system", metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).```

          junaid mukhtar added a comment - I was able to reproduce the issue by using an older version of Kubernetes-plugin running on dedicated EC2 instance and trying to connect to the EKS cluster.   Kubernetes Plugin: 1.13.5 EKS: 1.13   ```WARNING: Failed to count the # of live instances on Kubernetes io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://XXXXX.sk1.eu-west-1.eks.amazonaws.com/api/v1/namespaces/kube-system/pods?labelSelector=jenkins%3Dslave . Message: pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "kube-system". Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=null, kind=pods, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "kube-system", metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).```

            csanchez Carlos Sanchez
            bluegriffin Harsha GV
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: