-
Bug
-
Resolution: Fixed
-
Minor
-
None
-
Jenkins ver. 2.164.3
Kubernetes Credentials Plugin 0.4.0
-
-
kubernetes-credentials-provider-0.13
We have observed that sometimes Jenkins does not pick up new secrets after they have been added to kubernetes.
We do not observe any seemingly related errors in the jenkins log and are forced to restart jenkins in order for it to pick up the new secrets.
Sometimes it does find the new secrets but not always and there is not a clear indicator of when it will or will not find new secrets and display them in the credential store UI.
Is there any way to manually have it rescan for secrets?
[JENKINS-58257] New secrets written to kubernetes store are not always loaded by Jenkins
We have experienced this as well. Most likely the issue is the watch task closes due to some error like:
```
2019-08-25 06:20:05.874+0000 [id=43] INFO c.c.j.p.k.KubernetesCredentialProvider#onClose: onClose.
io.fabric8.kubernetes.client.KubernetesClientException: too old resource version: 130846525 (135483452)
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:254)
at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
The plugin does not attempt to recover from a closed watch connection. I've submitted a PR to restart the watcher if the onClose method is called with a client exception.
https://github.com/jenkinsci/kubernetes-credentials-provider-plugin/pull/31
Kyle Cronin
added a comment - - edited We have experienced this as well. Most likely the issue is the watch task closes due to some error like:
```
2019-08-25 06:20:05.874+0000 [id=43] INFO c.c.j.p.k.KubernetesCredentialProvider#onClose: onClose.
io.fabric8.kubernetes.client.KubernetesClientException: too old resource version: 130846525 (135483452)
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:254)
at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
The plugin does not attempt to recover from a closed watch connection. I've submitted a PR to restart the watcher if the onClose method is called with a client exception.
https://github.com/jenkinsci/kubernetes-credentials-provider-plugin/pull/31 ran into this as well, workaround needs a Singleton, wont work with Singletom. 
import com.cloudbees.jenkins.plugins.kubernetes_credentials_provider.KubernetesCredentialProvider
def kp = ExtensionList.lookupSingleton(KubernetesCredentialProvider.class)
kp.stopWatchingForSecrets()
kp.startWatchingForSecrets()
jugglefish this should have been fixed in kubernetes-credentials-provider-0.13
if it is not working for you please create a new ticket with all the logs from the provider
when this happens can you attache a stackdump of your jenkins instance and the jenkins logs please.
as a workaround you should be able to run the following in the groovy console
import com.cloudbees.jenkins.plugins.kubernetes_credentials_provider.KubernetesCredentialProvider def kp = ExtensionList.lookupSingletom(KubernetesCredentialProvider.class) kp.stopWatchingForSecrets() kp.startWatchingForSecrets()