-
Bug
-
Resolution: Incomplete
-
Blocker
-
None
-
EKS 1.18
Jenkins 2.263.4
AWS
Deploying Jenkins via Helm chart, and configuring with Configuration as code.
Jenkins Master has a service Account created, which is annotated to point to a Role.
Jenkins Agent has a Service Account Created, which is also annotated to a role.
I can see that the Jenkins Master role, is not getting called, so this is almost certainly where the problem is.
Sample Job Im testing with
pipeline {
agent {
kubernetes {
defaultContainer 'jnlp'
yaml """
apiVersion: v1
kind: Pod
metadata:
labels:
component: ci
spec:
serviceAccountName: jenkins-qa-agent
containers:
- name: aws
image: amazon/aws-cli
command:
- cat
tty: true
"""
}
}
stages {
stage('Test') {
steps {
container('aws') {
sh "aws s3 ls s3://s3-us-east-1-jenkins-mgmt-qa"
sh "echo test > test.txt"
sh "echo test > test2.txt"
sh "aws s3 cp test2.txt s3://s3-us-east-1-jenkins-mgmt-qa/artifacts/test2.txt"
archiveArtifacts artifacts: 'test.txt', followSymlinks: false
}
}
}
}
} The result
+ aws s3 cp test2.txt s3://s3-us-east-1-jenkins-mgmt-qa/artifacts/test2.txt Completed 5 Bytes/5 Bytes (33 Bytes/s) with 1 file(s) remaining upload: ./test2.txt to s3://s3-us-east-1-jenkins-mgmt-qa/artifacts/test2.txt Archiving artifacts ERROR: Failed to upload /home/jenkins/agent/workspace/test/test.txt to https://s3-us-east-1-jenkins-mgmt-qa.s3.amazonaws.com/artifacts/test/5/artifacts/test.txt?…, response: 403 Forbidden, body: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>ES9A612YHZRSHJ30</RequestId><HostId>xChB27LSJU1YG66XDMiU7wuLKPM1j30nVZaj+ce3p9g2iYZjqAaShTDub/L8NTYfjPFHfjSxCgo=</HostId></Error> Finished: FAILURE
So you can see the agent has access via its role, the Archive step, fails, because I assume its getting ran from the Master.
Deployment code
serviceAccount: create: true annotations: eks.amazonaws.com/role-arn: arn:aws:iam::29105xxxx:role/eks-epx-mgmt-jenkins-qa-masterserviceAccountAgent: create: true annotations: eks.amazonaws.com/role-arn: arn:aws:iam::29105xxxx:role/eks-epx-mgmt-jenkins-qa-worker
JCasC Code
controller:
JCasC:
globalconfig: |
unclassified:
artifactManager:
artifactManagerFactories:
- jclouds:
provider: s3
aws:
awsCredentials:
region: "us-east-1"
s3:
container: "s3-us-east-1-jenkins-mgmt-qa"
prefix: "artifacts/"
In k8s, I can see that the maser pod, has a service account defined, and it is getting the AWS role information injected into it.
serviceAccount: jenkins-qa
serviceAccountName: jenkins-qa
env:
- name: AWS_DEFAULT_REGION
value: us-east-1
- name: AWS_REGION
value: us-east-1
- name: AWS_ROLE_ARN
value: arn:aws:iam::291053455966:role/eks-epx-mgmt-jenkins-qa-worker
- name: AWS_WEB_IDENTITY_TOKEN_FILE
value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
This leads me to think, that the AWS plugins, is somehow attempting to use the Instance Profile, not the Service Accounts role.
[JENKINS-65080] EKS IRSA Not used for artifact-manager-s3
While artifact upload happens from inside the agent pod, the signature is precalculated by the controller, so it is the role assumed by the controller pod which would be relevant here. Jenkins is not using the AWS CLI for this purpose, it is using the AWS SDK for Java, wrapped in a plugin. Hard to say offhand where the problem lies, but smells like a misconfiguration rather than a product bug.
Yi Gong
added a comment - I can confirm that EKS IRSA works well of this plugin.
The plugin uses the IAM role of the controller.
Yi Gong
added a comment - I can confirm that EKS IRSA works well of this plugin.
The plugin uses the IAM role of the controller. 
Can you confirm which role is being used by the aws container? You can determine this by running `aws sts get-caller-identity`.
Also can you confirm which version of the aws cli is running inside the aws container? Only newer versions of the aws sdk support assuming an identity with a web token.
Please confirm that the version meets the minimum required version as defined on this AWS support page - https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html