-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins 2.483
GCP Secrets Manager Credentials Provider 0.3.1
We are trying to swap to using this plugin to load secrets from GCP but we're experiencing an issue. When we boot up our jenkins server, the secrets from the GCP provider don't show up for around 5-10 minutes after the jenkins server is started. This causes any jobs that start that are requiring secrets to fail. I've tried enabling debug logs for the plugin but I am not seeing anything happening until the secrets are finally loaded, then it all looks normal:
2024-11-11 20:56:39.118+0000 [id=422] INFO i.j.p.c.g.s.CredentialsSupplier#get: Using filter "labels.jenkins-credentials-type:*" to list secrets for xxxxx
I noticed that it looks like the secrets are cached for 5 minutes. Is it possible that the initial lookup is failing or somehow caching an empty list of secrets? I am not seeing any logs to indicate that it's failing, but I wonder if there is maybe some sort of plugin initialization ordering issue that might be affecting the load on startup.
Would appreciate any help with troubleshooting this.
[JENKINS-74838] Delay in Loading Secrets from GCP
Mason
added a comment - Ran into some more issues with adding permanent nodes to jcasc triggering the same issue.
I was able to scrape together enough java to submit a patch for this. We are currently running this patched version and it seems to be working well. https://github.com/jenkinsci/gcp-secrets-manager-credentials-provider-plugin/pull/221
Mason
added a comment - Ran into some more issues with adding permanent nodes to jcasc triggering the same issue.
I was able to scrape together enough java to submit a patch for this. We are currently running this patched version and it seems to be working well. https://github.com/jenkinsci/gcp-secrets-manager-credentials-provider-plugin/pull/221 Thanks for the investigation msedlik. After review I can get this merged into upstream and available in the next published version.
Ok I think we figured this out -
We have a groovy init script that's run by jcasc that populates a service account api token from the credentials store. This was seemingly getting run before the gcp secrets manager configs were populated. Fetching that secret was making us hit the case where the plugin returns an empty list of secrets when the project is not configured, and that was getting cached for 5 minutes. I was able to modify our init script to specifically query the system credential provider, which prevents the gcp provider from loading anything before its config is populated.
It might be nice to modify this plugin to not cache the empty list returned when the project is not set, but I am not familiar enough with Java to PR that myself.