Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62646

Failed to initialize Kubernetes secret provider

      Hi,

      We've been seeing the exception below for a while now. Since upgrading to Jenkins 2.240 and Kubernetes Plugin 1.26 it has started to add a permanent notification every N minutes. When I logged in today, there were about 9480 notifications and it was keeping on adding new notifications. Those won't go away until you restart Jenkins. 

      Failed to initialize Kubernetes secret providerio.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.43.0.1/api/v1/namespaces/jenkins/secrets?labelSelector=jenkins.io%2Fcredentials-type. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. secrets is forbidden: User "system:serviceaccount:jenkins:default" cannot list resource "secrets" in API group "" in the namespace "jenkins".
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:505)
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:412)
      	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:151)
      	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:621)
      	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:70)
      	at com.cloudbees.jenkins.plugins.kubernetes_credentials_provider.KubernetesCredentialProvider.startWatchingForSecrets(KubernetesCredentialProvider.java:115)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
      	at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
      	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
      	at jenkins.model.Jenkins$5.runTask(Jenkins.java:1131)
      	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
      	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
      	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

      I'm not sure why is it complaining, since executing a curl request to the above endpoint using the jenkin's user token gives a successful result (it simply has no secrets configured in the namespace):

      $  curl -k "https://10.43.0.1/api/v1/namespaces/jenkins/secrets?labelSelector=jenkins.io%2Fcredentials-type" -H "Authorization: Bearer $JENKINS_USER_TOKEN"
      {
        "kind": "SecretList",
        "apiVersion": "v1",
        "metadata": {
          "selfLink": "/api/v1/namespaces/jenkins/secrets",
          "resourceVersion": "63369926"
        },
        "items": []
      }
      

       

          [JENKINS-62646] Failed to initialize Kubernetes secret provider

          LUCIANO BEJA added a comment -

          I installed it because it is mandatory for the Jenkins integration plugin with Kubernetes.

           

          LUCIANO BEJA added a comment - I installed it because it is mandatory for the Jenkins integration plugin with Kubernetes.  

          James Nord added a comment - - edited

          > I installed it because it is mandatory for the Jenkins integration plugin with Kubernetes.

          it should not be, are you sure? I have several instances using k8s without this plugin

          James Nord added a comment - - edited > I installed it because it is mandatory for the Jenkins integration plugin with Kubernetes. it should not be, are you sure? I have several instances using k8s without this plugin

          LUCIANO BEJA added a comment - - edited

          See the image I attached. I don't have the option to uninstall it.

          LUCIANO BEJA added a comment - - edited See the image I attached. I don't have the option to uninstall it.

          James Nord added a comment -

          'kubernetes credentials' is not 'kubernetes credential provider'. but also the reason why something is not uninstallable is shown in a tooltip, so please make sure any screenshots contain those details

          James Nord added a comment - 'kubernetes credentials' is not 'kubernetes credential provider'. but also the reason why something is not uninstallable is shown in a tooltip, so please make sure any screenshots contain those details

          Steve Todorov added a comment -

          Hi teilo,

          Apologies for my late reply.

          I believe this was auto-installed by the kubernetes-plugin quite a while ago. We are planning to use it for some deployment jobs, but haven't gotten to it yet.

          Nevertheless I believe this plugin should work with an empty list of credentials.

          P.S Sorry for labeling the issue wrong and thanks for fixing it for me!

          Steve Todorov added a comment - Hi teilo , Apologies for my late reply. I believe this was auto-installed by the kubernetes-plugin quite a while ago. We are planning to use it for some deployment jobs, but haven't gotten to it yet. Nevertheless I believe this plugin should work with an empty list of credentials. P.S Sorry for labeling the issue wrong and thanks for fixing it for me!

          James Nord added a comment -

          for clarity, the plugin does work when no secrets are configured.

          the service account is missing permissions to list secrets.
          what is strange is if you restart ienki s the exception goes away for some time.

          this implies something is funky with the service account,. the token the service has should not expire for the lifetime of the service iiuc.

          without detailed logs from kubectl running in the same pod and this plugin when it happens it there s hard to reason the cause. for those that do have this issue knowing the following may help :

          James Nord added a comment - for clarity, the plugin does work when no secrets are configured. the service account is missing permissions to list secrets. what is strange is if you restart ienki s the exception goes away for some time. this implies something is funky with the service account,. the token the service has should not expire for the lifetime of the service iiuc. without detailed logs from kubectl running in the same pod and this plugin when it happens it there s hard to reason the cause. for those that do have this issue knowing the following may help : how long after a restart does it take to occur what k8s distro and version are you using what k8s authorizer and config do you use ( https://kubernetes.io/docs/reference/access-authn-authz/authorization/ )

          LUCIANO BEJA added a comment -

          Hi James, sorry for the delay in responding. You're right, I happened to have already removed the credential provider and I didn't even notice. Sorry and thank you

          LUCIANO BEJA added a comment - Hi James, sorry for the delay in responding. You're right, I happened to have already removed the credential provider and I didn't even notice. Sorry and thank you

          Steve Todorov added a comment - - edited

          > the service account is missing permissions to list secrets.

          teilo the service account has access as I've mentioned in the issue description - the secret list is simply empty, because there are none configured in k8s. If I configure one it lists using the command in the description.

          Regarding your questions:

          • how long after a restart does it take to occur - about 30 seconds after a Jenkins restart.
          • kubectl version
          Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:47:41Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
          
          Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
          

          After looking at the log I posted above, I actually noticed the plugin is attempting to use the wrong service account:

          system:serviceaccount:jenkins:default

           

          The service account I have Jenkins running with is actually system:serviceaccount:jenkins:jenkins, but I don't see anywhere an option to configure this for the plugin?

           

           

           

          Steve Todorov added a comment - - edited > the service account is missing permissions to list secrets. teilo the service account has access as I've mentioned in the issue description - the secret list is simply empty, because there are none configured in k8s. If I configure one it lists using the command in the description. Regarding your questions: how long after a restart does it take to occur - about 30 seconds after a Jenkins restart. kubectl version Client Version: version.Info{Major: "1" , Minor: "18" , GitVersion: "v1.18.5" , GitCommit: "e6503f8d8f769ace2f338794c914a96fc335df0f" , GitTreeState: "clean" , BuildDate: "2020-06-26T03:47:41Z" , GoVersion: "go1.13.9" , Compiler : "gc" , Platform: "linux/amd64" } Server Version: version.Info{Major: "1" , Minor: "18" , GitVersion: "v1.18.3" , GitCommit: "2e7996e3e2712684bc73f0dec0200d64eec7fe40" , GitTreeState: "clean" , BuildDate: "2020-05-20T12:43:34Z" , GoVersion: "go1.13.9" , Compiler : "gc" , Platform: "linux/amd64" } After looking at the log I posted above, I actually noticed the plugin is attempting to use the wrong service account: system:serviceaccount:jenkins: default   The service account I have Jenkins running with is actually system:serviceaccount:jenkins:jenkins , but I don't see anywhere an option to configure this for the plugin?      

          James Nord added a comment -

          in the pod can you check /var/run/secrets/kubernetes.io/serviceaccount/token is the same token that you expect it to be. and try the curl command with that token as the contents

          this is all handled by kubernetes and we just pick up what should be there.

          other than that have you configured any environment for the Jenkins container?

          James Nord added a comment - in the pod can you check /var/run/secrets/kubernetes.io/serviceaccount/token is the same token that you expect it to be. and try the curl command with that token as the contents this is all handled by kubernetes and we just pick up what should be there. other than that have you configured any environment for the Jenkins container?

          Steve Todorov added a comment -

          Thanks for the tips! This pointed me into the right direction. Our `deployment` was using the `default` service account which was exactly what was exposed to the `pod` hence the exception. After I fixed that - the error message is no longer showing up.

           

          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: jenkins
            namespace: jenkins
            labels:
              workload.user.cattle.io/workloadselector: deployment-jenkins-jenkins
          spec:
            progressDeadlineSeconds: 600
            replicas: 1
            revisionHistoryLimit: 10
            selector:
              matchLabels:
                workload.user.cattle.io/workloadselector: deployment-jenkins-jenkins
            strategy:
              rollingUpdate:
                maxSurge: 0
                maxUnavailable: 1
              type: RollingUpdate
            template:
              metadata:
                labels:
                  workload.user.cattle.io/workloadselector: deployment-jenkins-jenkins
              spec:
                # https://issues.jenkins-ci.org/browse/JENKINS-62646
                serviceAccountName: jenkins
                automountServiceAccountToken: true
                containers:
                  - name: jenkins
                    image: jenkins/jenkins:2.243-alpine
                    imagePullPolicy: Always
          

          lucianobeja maybe check if you have the `serviceAccountName` configured in your deployment as well.

           

          Steve Todorov added a comment - Thanks for the tips! This pointed me into the right direction. Our `deployment` was using the `default` service account which was exactly what was exposed to the `pod` hence the exception. After I fixed that - the error message is no longer showing up.   apiVersion: apps/v1 kind: Deployment metadata: name: jenkins namespace: jenkins labels: workload.user.cattle.io/workloadselector: deployment-jenkins-jenkins spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: workload.user.cattle.io/workloadselector: deployment-jenkins-jenkins strategy: rollingUpdate: maxSurge: 0 maxUnavailable: 1 type: RollingUpdate template: metadata: labels: workload.user.cattle.io/workloadselector: deployment-jenkins-jenkins spec: # https: //issues.jenkins-ci.org/browse/JENKINS-62646 serviceAccountName: jenkins automountServiceAccountToken: true containers: - name: jenkins image: jenkins/jenkins:2.243-alpine imagePullPolicy: Always lucianobeja maybe check if you have the `serviceAccountName` configured in your deployment as well.  

            teilo James Nord
            stodorov Steve Todorov
            Votes:
            5 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: