If you have a shared cloud which is using its own set of credentials and you update the secret value (ex. service account token) used for authentication you will then need to restart the controller because the underlying k8s client is still using the old credential value.

Steps to Reproduce

In the end i found a pretty easy way to create a test bed for this issue:

spin up a cluster somewhere I used gke.

Create a new namespace and service account + token that will be used to host the cloud agents. You can do that with the following commands (assuming you are connected to your cluster):`

kubectl create ns agents
kubens agents
kubectl cluster-info
kubectl apply -f service_account.yaml
kubectl apply -f cluster-role-binding.yaml
kubectl apply -f service_account_secret.yaml
kubectl create token jenkins -n agents

Now, you can use the command

kubectl get secret seco-secret -o yaml
kubectl cluster-info

This will give you the server cert, secret token, and cluster endpoint information needed to configure the cloud. The cert and token are base64 encoded and need to be decoded before putting them into the cloud config. The first line of the kubectl cluster-info command output should look similar to Kubernetes control plane is running at https://34.148.228.118 and is the value you should use for the api endpoint defined in the cloud.

In your controller, create a new kubernetes cloud using the above info to configure it. You will need to create a new secret text credential and select it in the cloud config.

If you do the above correctly then you should be able to use the test button to verify that it all works. From there you can try to reproduce the issue.

Now go to manage jenkins-> credentials and update the secret value for the credential you created so that the secret is some bogus value. Now try to run a test build in your clodu, it will succeed even though you have broken the credentials. If you go back to the cloud config and try the “test” button it will correctly fail, but you will still be able to run builds in your cloud.

You have to restart the server in order to forcec a new k8s client to be instantiated with the current (broken) credentials.