[JENKINS-66759] AWS IRSA not working on version 1.12.69

Uladzimir Kalinouski created issue - 2021-09-29 15:40

Uladzimir Kalinouski made changes - 2021-09-29 15:41

Description

Original: Environment with AWS IAM Role assigned to K8s Service Account assigned to Jenkins Pod.

Plugins *Amazon EC2* and *AWS Secrets Manager Credentials Provider* are unable to authenticate with *aws-java-sdk* version 1.12.69, everything works fine on 1.12.68.

Inside container *_curl [http://169.254.169.254/latest/meta-data/iam/info]_* gives correct *__* IAM Role on both versions.

New: Environment with AWS IAM Role assigned to K8s Service Account assigned to Jenkins Pod.

Plugins *Amazon EC2* and *AWS Secrets Manager Credentials Provider* are unable to authenticate with *aws-java-sdk* version 1.12.69, everything works fine on 1.12.68, downgrade helps.

Inside container *_curl [http://169.254.169.254/latest/meta-data/iam/info]_* gives correct *__* IAM Role on both versions.

Vincent Latombe added a comment - 2021-09-30 08:12 - edited

Hi, thank you for the report. ~~However I'm unable to reproduce on my side. Could you provide some logs?~~

I can reproduce now.

Vincent Latombe added a comment - 2021-09-30 08:12 - edited Hi, thank you for the report. However I'm unable to reproduce on my side. Could you provide some logs? I can reproduce now.

Uladzimir Kalinouski added a comment - 2021-09-30 11:30

Sure.

EC2

jenkins com.amazonaws.services.ec2.model.AmazonEC2Exception: You are not authorized to perform this operation. (Service: AmazonEC2; Status Code: 403; Error Code: UnauthorizedOperation; Request ID: bbfXcXXd-XXXX-XXXX-XXXf-XXdc245ead81; Proxy: null)jenkins com.amazonaws.services.ec2.model.AmazonEC2Exception: You are not authorized to perform this operation. (Service: AmazonEC2; Status Code: 403; Error Code: UnauthorizedOperation; Request ID: bbfXcXXd-XXXX-XXXX-XXXf-XXdc245ead81; Proxy: null)jenkins  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1862)jenkins  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1415)jenkins  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384)jenkins  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1154)jenkins  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:811)jenkins  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)jenkins  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)jenkins  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)jenkins  at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)jenkins  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)jenkins  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)jenkins  at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:30343)jenkins  at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:30310)jenkins  at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:30299)jenkins  at com.amazonaws.services.ec2.AmazonEC2Client.executeDescribeImages(AmazonEC2Client.java:13670)jenkins  at com.amazonaws.services.ec2.AmazonEC2Client.describeImages(AmazonEC2Client.java:13638)jenkins  at hudson.plugins.ec2.SlaveTemplate.getImage(SlaveTemplate.java:1332)jenkins  at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:886)jenkins  at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:714)jenkins  at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:740)jenkins  at hudson.slaves.Cloud.provision(Cloud.java:201)jenkins  at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:730)jenkins  at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:335)jenkins  at hudson.slaves.NodeProvisioner.access$900(NodeProvisioner.java:65)jenkins  at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:824)jenkins  at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:91)jenkins  at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)jenkins  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)jenkins  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)jenkins  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)jenkins  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)jenkins  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)jenkins  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)jenkins  at java.lang.Thread.run(Thread.java:748)

AWS Secrets Manager Credentials Provider

jenkins 2021-09-30 11:24:30.541+0000 [id=6166]	WARNING	i.j.p.c.s.AwsCredentialsProvider#getCredentials: Could not list credentials in Secrets Manager: message=[User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/productionXXXXXXXXXXXXXXXXXXXXXXXXXX/i-0aXXXXXXXXXdXdXeX is not authorized to perform: secretsmanager:ListSecrets because no identity-based policy allows the secretsmanager:ListSecrets action (Service: AWSSecretsManager; Status Code: 400; Error Code: AccessDeniedException; Request ID: eXcXXdXb-fbdb-XXbX-aXXc-XeXcXfXXebfX; Proxy: null)]

List of plugins:

ace-editor: 1.1
ansible: 1.1
ansicolor: 1.0.0
ant: 1.11
antisamy-markup-formatter: 2.1
apache-httpcomponents-client-4-api: 4.5.13-1.0
artifactory: 3.13.2
authentication-tokens: 1.4
authorize-project: 1.4.0
aws-credentials: 1.30
aws-java-sdk-cloudformation: 1.12.69
aws-java-sdk-codebuild: 1.12.69
aws-java-sdk-core: 1.12.69
aws-java-sdk-ec2: 1.12.69
aws-java-sdk-ecr: 1.12.69
aws-java-sdk-ecs: 1.12.69
aws-java-sdk-elasticbeanstalk: 1.12.69
aws-java-sdk-iam: 1.12.69
aws-java-sdk-jmespath: 1.12.69
aws-java-sdk-kms: 1.12.69
aws-java-sdk-logs: 1.12.69
aws-java-sdk-s3: 1.12.69
aws-java-sdk-ssm: 1.12.69
aws-java-sdk-sts: 1.12.69
aws-java-sdk: 1.12.69
aws-parameter-store: 1.2.2
aws-secrets-manager-credentials-provider: 0.5.4
azure-ad: 184.v44f04b65bdd5
azure-commons: 1.1.3
azure-sdk: 48.veb7555463c4b
bitbucket: 1.1.29
blackduck-detect: 7.0.0
block-queued-job: 0.2.0
bootstrap4-api: 4.6.0-3
bootstrap5-api: 5.1.1-1
bouncycastle-api: 2.25
branch-api: 2.7.0
build-timeout: 1.20
caffeine-api: 2.9.2-29.v717aac953ff3
checks-api: 1.7.2
cloudbees-bitbucket-branch-source: 2.9.11
cloudbees-folder: 6.16
command-launcher: 1.6
config-file-provider: 3.8.1
configuration-as-code: 1.53
credentials-binding: 1.27
credentials: 2.6.1
display-url-api: 2.3.5
docker-commons: 1.17
docker-workflow: 1.26
durable-task: 1.39
ec2: 1.64
echarts-api: 5.1.2-11
envinject-api: 1.7
envinject: 2.4.0
external-monitor-job: 1.7
font-awesome-api: 5.15.4-1
generic-webhook-trigger: 1.77
git-client: 3.9.0
git-server: 1.10
git: 4.8.2
google-oauth-plugin: 1.0.6
gradle: 1.37.1
greenballs: 1.15.1
handlebars: 3.0.8
handy-uri-templates-2-api: 2.1.8-1.0
ivy: 2.1
jackson2-api: 2.12.4
javadoc: 1.6
jdk-tool: 1.5
jira: 3.6
job-dsl: 1.77
job-restrictions: 0.8
jobConfigHistory: 2.28.1
jquery-detached: 1.2.1
jquery3-api: 3.6.0-2
jsch: 0.1.55.2
junit: 1.53
kubernetes-client-api: 5.4.1
kubernetes-credentials: 0.9.0
kubernetes: 1.30.1
ldap: 2.7
lockable-resources: 2.11
mailer: 1.34
mapdb-api: 1.0.9.0
matrix-auth: 2.6.8
matrix-project: 1.19
maven-plugin: 3.13
mercurial: 2.15
metrics: 4.0.2.8
momentjs: 1.1.1
node-iterator-api: 1.5.1
oauth-credentials: 0.4
pam-auth: 1.6
pipeline-build-step: 2.15
pipeline-graph-analysis: 1.11
pipeline-input-step: 2.12
pipeline-milestone-step: 1.3.2
pipeline-model-api: 1.9.2
pipeline-model-definition: 1.9.2
pipeline-model-extensions: 1.9.2
pipeline-rest-api: 2.19
pipeline-stage-step: 2.5
pipeline-stage-tags-metadata: 1.9.2
pipeline-stage-view: 2.19
pipeline-utility-steps: 2.10.0
plain-credentials: 1.7
plugin-util-api: 2.5.0
popper-api: 1.16.1-2
popper2-api: 2.10.1-1
purge-job-history: 1.6
pyenv-pipeline: 2.1.2
pyenv: 0.0.7
resource-disposer: 0.16
ruby-runtime: 0.12
run-condition: 1.5
saml: 2.0.8
scm-api: 2.6.5
script-security: 1.78
slack: 2.48
snakeyaml-api: 1.29.1
ssh-agent: 1.23
ssh-credentials: 1.19
sshd: 3.1.0
stashNotifier: 1.20
structs: 1.23
subversion: 2.14.5
sumologic-publisher: 2.2.1
token-macro: 266.v44a80cf277fd
trilead-api: 1.0.13
variant: 1.4
windows-slaves: 1.8
workflow-aggregator: 2.6
workflow-api: 2.46
workflow-basic-steps: 2.24
workflow-cps-global-lib: 2.21
workflow-cps: 2.94
workflow-durable-task-step: 2.40
workflow-job: 2.41
workflow-multibranch: 2.26
workflow-scm-step: 2.13
workflow-step-api: 2.24
workflow-support: 3.8
ws-cleanup: 0.39

Uladzimir Kalinouski added a comment - 2021-09-30 11:30 Sure. EC2 jenkins com.amazonaws.services.ec2.model.AmazonEC2Exception: You are not authorized to perform this operation. (Service: AmazonEC2; Status Code: 403; Error Code: UnauthorizedOperation; Request ID: bbfXcXXd-XXXX-XXXX-XXXf-XXdc245ead81; Proxy: null )jenkins com.amazonaws.services.ec2.model.AmazonEC2Exception: You are not authorized to perform this operation. (Service: AmazonEC2; Status Code: 403; Error Code: UnauthorizedOperation; Request ID: bbfXcXXd-XXXX-XXXX-XXXf-XXdc245ead81; Proxy: null )jenkins at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1862)jenkins at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1415)jenkins at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384)jenkins at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1154)jenkins at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:811)jenkins at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)jenkins at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)jenkins at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)jenkins at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)jenkins at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)jenkins at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)jenkins at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:30343)jenkins at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:30310)jenkins at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:30299)jenkins at com.amazonaws.services.ec2.AmazonEC2Client.executeDescribeImages(AmazonEC2Client.java:13670)jenkins at com.amazonaws.services.ec2.AmazonEC2Client.describeImages(AmazonEC2Client.java:13638)jenkins at hudson.plugins.ec2.SlaveTemplate.getImage(SlaveTemplate.java:1332)jenkins at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:886)jenkins at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:714)jenkins at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:740)jenkins at hudson.slaves.Cloud.provision(Cloud.java:201)jenkins at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:730)jenkins at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:335)jenkins at hudson.slaves.NodeProvisioner.access$900(NodeProvisioner.java:65)jenkins at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:824)jenkins at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:91)jenkins at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)jenkins at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)jenkins at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)jenkins at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)jenkins at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)jenkins at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)jenkins at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)jenkins at java.lang. Thread .run( Thread .java:748) AWS Secrets Manager Credentials Provider jenkins 2021-09-30 11:24:30.541+0000 [id=6166] WARNING i.j.p.c.s.AwsCredentialsProvider#getCredentials: Could not list credentials in Secrets Manager: message=[User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/productionXXXXXXXXXXXXXXXXXXXXXXXXXX/i-0aXXXXXXXXXdXdXeX is not authorized to perform: secretsmanager:ListSecrets because no identity-based policy allows the secretsmanager:ListSecrets action (Service: AWSSecretsManager; Status Code: 400; Error Code: AccessDeniedException; Request ID: eXcXXdXb-fbdb-XXbX-aXXc-XeXcXfXXebfX; Proxy: null )] List of plugins: ace-editor: 1.1 ansible: 1.1 ansicolor: 1.0.0 ant: 1.11 antisamy-markup-formatter: 2.1 apache-httpcomponents-client-4-api: 4.5.13-1.0 artifactory: 3.13.2 authentication-tokens: 1.4 authorize-project: 1.4.0 aws-credentials: 1.30 aws-java-sdk-cloudformation: 1.12.69 aws-java-sdk-codebuild: 1.12.69 aws-java-sdk-core: 1.12.69 aws-java-sdk-ec2: 1.12.69 aws-java-sdk-ecr: 1.12.69 aws-java-sdk-ecs: 1.12.69 aws-java-sdk-elasticbeanstalk: 1.12.69 aws-java-sdk-iam: 1.12.69 aws-java-sdk-jmespath: 1.12.69 aws-java-sdk-kms: 1.12.69 aws-java-sdk-logs: 1.12.69 aws-java-sdk-s3: 1.12.69 aws-java-sdk-ssm: 1.12.69 aws-java-sdk-sts: 1.12.69 aws-java-sdk: 1.12.69 aws-parameter-store: 1.2.2 aws-secrets-manager-credentials-provider: 0.5.4 azure-ad: 184.v44f04b65bdd5 azure-commons: 1.1.3 azure-sdk: 48.veb7555463c4b bitbucket: 1.1.29 blackduck-detect: 7.0.0 block-queued-job: 0.2.0 bootstrap4-api: 4.6.0-3 bootstrap5-api: 5.1.1-1 bouncycastle-api: 2.25 branch-api: 2.7.0 build-timeout: 1.20 caffeine-api: 2.9.2-29.v717aac953ff3 checks-api: 1.7.2 cloudbees-bitbucket-branch-source: 2.9.11 cloudbees-folder: 6.16 command-launcher: 1.6 config-file-provider: 3.8.1 configuration-as-code: 1.53 credentials-binding: 1.27 credentials: 2.6.1 display-url-api: 2.3.5 docker-commons: 1.17 docker-workflow: 1.26 durable-task: 1.39 ec2: 1.64 echarts-api: 5.1.2-11 envinject-api: 1.7 envinject: 2.4.0 external-monitor-job: 1.7 font-awesome-api: 5.15.4-1 generic -webhook-trigger: 1.77 git-client: 3.9.0 git-server: 1.10 git: 4.8.2 google-oauth-plugin: 1.0.6 gradle: 1.37.1 greenballs: 1.15.1 handlebars: 3.0.8 handy-uri-templates-2-api: 2.1.8-1.0 ivy: 2.1 jackson2-api: 2.12.4 javadoc: 1.6 jdk-tool: 1.5 jira: 3.6 job-dsl: 1.77 job-restrictions: 0.8 jobConfigHistory: 2.28.1 jquery-detached: 1.2.1 jquery3-api: 3.6.0-2 jsch: 0.1.55.2 junit: 1.53 kubernetes-client-api: 5.4.1 kubernetes-credentials: 0.9.0 kubernetes: 1.30.1 ldap: 2.7 lockable-resources: 2.11 mailer: 1.34 mapdb-api: 1.0.9.0 matrix-auth: 2.6.8 matrix-project: 1.19 maven-plugin: 3.13 mercurial: 2.15 metrics: 4.0.2.8 momentjs: 1.1.1 node-iterator-api: 1.5.1 oauth-credentials: 0.4 pam-auth: 1.6 pipeline-build-step: 2.15 pipeline-graph-analysis: 1.11 pipeline-input-step: 2.12 pipeline-milestone-step: 1.3.2 pipeline-model-api: 1.9.2 pipeline-model-definition: 1.9.2 pipeline-model-extensions: 1.9.2 pipeline- rest -api: 2.19 pipeline-stage-step: 2.5 pipeline-stage-tags-metadata: 1.9.2 pipeline-stage-view: 2.19 pipeline-utility-steps: 2.10.0 plain-credentials: 1.7 plugin-util-api: 2.5.0 popper-api: 1.16.1-2 popper2-api: 2.10.1-1 purge-job-history: 1.6 pyenv-pipeline: 2.1.2 pyenv: 0.0.7 resource-disposer: 0.16 ruby-runtime: 0.12 run-condition: 1.5 saml: 2.0.8 scm-api: 2.6.5 script-security: 1.78 slack: 2.48 snakeyaml-api: 1.29.1 ssh-agent: 1.23 ssh-credentials: 1.19 sshd: 3.1.0 stashNotifier: 1.20 structs: 1.23 subversion: 2.14.5 sumologic-publisher: 2.2.1 token-macro: 266.v44a80cf277fd trilead-api: 1.0.13 variant: 1.4 windows-slaves: 1.8 workflow-aggregator: 2.6 workflow-api: 2.46 workflow-basic-steps: 2.24 workflow-cps-global-lib: 2.21 workflow-cps: 2.94 workflow-durable-task-step: 2.40 workflow-job: 2.41 workflow-multibranch: 2.26 workflow-scm-step: 2.13 workflow-step-api: 2.24 workflow-support: 3.8 ws-cleanup: 0.39

Vincent Latombe made changes - 2021-09-30 11:40

Status

Original: Open [ 1 ]

New: In Progress [ 3 ]

Vincent Latombe made changes - 2021-09-30 11:40

Status

Original: In Progress [ 3 ]

New: In Review [ 10005 ]

Vincent Latombe made changes - 2021-09-30 11:41

Remote Link

New: This issue links to "PR #540 (Web Link)" [ 26932 ]

Vincent Latombe added a comment - 2021-09-30 12:04

Should be fixed with 1.12.70.

I had to change the plugin organization so the following plugins are removed:

aws-java-sdk-core
aws-java-sdk-kms
aws-java-sdk-s3
aws-java-sdk-sts
aws-java-sdk-jmespath

They have been merged into aws-java-sdk-minimal.

Vincent Latombe added a comment - 2021-09-30 12:04 Should be fixed with 1.12.70. I had to change the plugin organization so the following plugins are removed: aws-java-sdk-core aws-java-sdk-kms aws-java-sdk-s3 aws-java-sdk-sts aws-java-sdk-jmespath They have been merged into aws-java-sdk-minimal.

Vincent Latombe made changes - 2021-09-30 12:04

Released As		New: 1.12.70
Resolution		New: Fixed [ 1 ]
Status	Original: In Review [ 10005 ]	New: Resolved [ 5 ]

Uladzimir Kalinouski added a comment - 2021-09-30 14:33

Tested on one instance, looks like everything is working.

Thank a lot for such quick fix!

Uladzimir Kalinouski added a comment - 2021-09-30 14:33 Tested on one instance, looks like everything is working. Thank a lot for such quick fix!

Jenkins

Details

Description

Attachments

Issue Links

Activity

Collapse comment: Vincent Latombe added a comment - 2021-09-30 08:12, Edited by Vincent Latombe - 2021-09-30 09:44

Expand comment: Vincent Latombe added a comment - 2021-09-30 08:12, Edited by Vincent Latombe - 2021-09-30 09:44

Collapse comment: Uladzimir Kalinouski added a comment - 2021-09-30 11:30

Expand comment: Uladzimir Kalinouski added a comment - 2021-09-30 11:30

Collapse comment: Vincent Latombe added a comment - 2021-09-30 12:04

Expand comment: Vincent Latombe added a comment - 2021-09-30 12:04

Collapse comment: Uladzimir Kalinouski added a comment - 2021-09-30 14:33

Expand comment: Uladzimir Kalinouski added a comment - 2021-09-30 14:33

People

Dates