[JENKINS-68095] The application default credentials not working in plugin gcp-secrets-manager with git checkout

Type: Bug
Resolution: Unresolved
Priority: Major
Component/s: gcp-secrets-manager-credentials-provider-plugin
Labels:
None

Similar Issues:
Powered by SuggestiMate

Show

I have configured the Jenkins master in GCP with applications default credentials, so I don't have configured anywhere GOOGLE_APPLICATION_CREDENTIALS and when credentials are being used withCredentials method, then it's working properly

node('some-non-gcp-agent') {
    withCredentials(bindings: [sshUserPrivateKey(credentialsId: 'some-ssh-key', keyFileVariable: 'SSH_KEY_FOR_ABC')]) {
        echo env.SSH_KEY_FOR_ABC
    }
}

outputs:

...
[Pipeline] {
    [Pipeline] withCredentials
    Masking supported pattern matches of $SSH_KEY_FOR_ABC
    [Pipeline] {
        [Pipeline] echo
        ****
        [Pipeline] }
    [Pipeline] // withCredentials
    [Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

But when it's used with git checkout like this:

node('some-non-gcp-agent') {
    git url: 'git@git.example.com:some/repo.git', credentialsId: 'some-ssh-key'
}

it throws an exception:

Running on some-non-gcp-agent in /home/jenkins-slave/workspace/test-pipeline
[Pipeline] {
    [Pipeline] git
    Selected Git installation does not exist. Using Default
    The recommended git tool is: NONE
    using credential some-ssh-key
    Cloning the remote Git repository
    Cloning repository git@git.example.com:some/repo.git
        > git init /home/jenkins-slave/workspace/test-pipeline # timeout=10
    Fetching upstream changes from git@git.example.com:some/repo.git
        > git --version # timeout=10
        > git --version # 'git version 1.8.3.1'
    using GIT_SSH to set credentials some-ssh-key

    [Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
at com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentials(DefaultCredentialsProvider.java:134)
at com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:119)
at com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:91)
at com.google.api.gax.core.GoogleCredentialsProvider.getCredentials(GoogleCredentialsProvider.java:67)
at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:145)
at com.google.cloud.secretmanager.v1.stub.GrpcSecretManagerServiceStub.create(GrpcSecretManagerServiceStub.java:237)
at com.google.cloud.secretmanager.v1.stub.SecretManagerServiceStubSettings.createStub(SecretManagerServiceStubSettings.java:226)
at com.google.cloud.secretmanager.v1.SecretManagerServiceClient.<init>(SecretManagerServiceClient.java:154)
at com.google.cloud.secretmanager.v1.SecretManagerServiceClient.create(SecretManagerServiceClient.java:135)
at com.google.cloud.secretmanager.v1.SecretManagerServiceClient.create(SecretManagerServiceClient.java:126)
at io.jenkins.plugins.credentials.gcp.secretsmanager.GcpSecretGetter.getPayload(GcpSecretGetter.java:35)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to some-non-gcp-agent
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1797)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
at hudson.remoting.Channel.call(Channel.java:1001)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:143)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:129)
at com.sun.proxy.$Proxy136.execute(Unknown Source)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1224)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1302)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused: com.cloudbees.plugins.credentials.CredentialsUnavailableException: Property 'secret' is currently unavailable, reason: Could not retrieve the credentials from GCP Secrets Manager
at io.jenkins.plugins.credentials.gcp.secretsmanager.GcpSecretGetter.getPayload(GcpSecretGetter.java:47)
at io.jenkins.plugins.credentials.gcp.secretsmanager.GcpSecretGetter.getSecretString(GcpSecretGetter.java:26)
at io.jenkins.plugins.credentials.gcp.secretsmanager.CredentialsFactory$SecretSupplier.get(CredentialsFactory.java:77)
at io.jenkins.plugins.credentials.gcp.secretsmanager.CredentialsFactory$SecretSupplier.get(CredentialsFactory.java:63)
at io.jenkins.plugins.credentials.gcp.secretsmanager.GcpSshUserPrivateKey.getPrivateKeys(GcpSshUserPrivateKey.java:29)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.createSshKeyFile(CliGitAPIImpl.java:2279)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2022)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:84)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:618)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:847)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:158)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:151)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:376)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
at java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:829)
Finished: FAILURE

Michael Daugherty added a comment - 2022-10-08 04:49 - edited

I'll need some more information about your Jenkins setup to help debug this issue.

Are the default credentials available on both the controller and the agents?

One test we can do is run:

curl "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email" -H "Metadata-Flavor: Google"

on both the agent and the controller to ensure it is the email expected.

Michael Daugherty added a comment - 2022-10-08 04:49 - edited I'll need some more information about your Jenkins setup to help debug this issue. Are the default credentials available on both the controller and the agents? One test we can do is run: curl "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email" -H "Metadata-Flavor: Google" on both the agent and the controller to ensure it is the email expected.

Mateusz Janczuk added a comment - 2022-10-08 06:57 - edited

I can confirm that service account email will be not available on agents.

Agents usually are using different service accounts than a master (main) instance because they have different responsibilities/scopes. We also have agents used outside google so they may not have any google accounts.

An interesting part for me is why it's working with:

withCredentials(bindings: [sshUserPrivateKey(credentialsId: 'some-ssh-key'

but it's not working on the same non-gcp agent:

 git url: 'git@git.example.com:some/repo.git', credentialsId: 'some-ssh-key'

Because in both cases agent has no access to credentials, only the master has them.

Mateusz Janczuk added a comment - 2022-10-08 06:57 - edited I can confirm that service account email will be not available on agents. Agents usually are using different service accounts than a master (main) instance because they have different responsibilities/scopes. We also have agents used outside google so they may not have any google accounts. An interesting part for me is why it's working with: withCredentials(bindings: [sshUserPrivateKey(credentialsId: 'some-ssh-key' but it's not working on the same non-gcp agent: git url: 'git@git.example.com:some/repo.git' , credentialsId: 'some-ssh-key' Because in both cases agent has no access to credentials, only the master has them.

Jenkins

Details

Description

Attachments

Activity

Collapse comment: Michael Daugherty added a comment - 2022-10-08 04:49, Edited by Michael Daugherty - 2022-10-08 04:57

Expand comment: Michael Daugherty added a comment - 2022-10-08 04:49, Edited by Michael Daugherty - 2022-10-08 04:57

Collapse comment: Mateusz Janczuk added a comment - 2022-10-08 06:57, Edited by Mateusz Janczuk - 2022-10-08 06:58

Expand comment: Mateusz Janczuk added a comment - 2022-10-08 06:57, Edited by Mateusz Janczuk - 2022-10-08 06:58

People

Dates