Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65080

EKS IRSA Not used for artifact-manager-s3

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Deploying Jenkins via Helm chart, and configuring with Configuration as code.

       

      Jenkins Master has a service Account created, which is annotated to point to a Role.

      Jenkins Agent has a Service Account Created, which is also annotated to a role.

       

      I can see that the Jenkins Master role, is not getting called, so this is almost certainly where the problem is.

      Sample Job Im testing with

       

      pipeline {
        agent {
          kubernetes {
            defaultContainer 'jnlp'
            yaml """
      apiVersion: v1
      kind: Pod
      metadata:
      labels:
        component: ci
      spec:
        serviceAccountName: jenkins-qa-agent
        containers:
        - name: aws
          image: amazon/aws-cli
          command:
          - cat
          tty: true
      """
      }
         }
        stages {
          stage('Test') {
            steps {
              container('aws') {
                sh "aws s3 ls s3://s3-us-east-1-jenkins-mgmt-qa"
                sh "echo test > test.txt"
                sh "echo test > test2.txt"
                sh "aws s3 cp test2.txt s3://s3-us-east-1-jenkins-mgmt-qa/artifacts/test2.txt"
                archiveArtifacts artifacts: 'test.txt', followSymlinks: false
              }
            }
          }
        }
      } The result

       

      + aws s3 cp test2.txt s3://s3-us-east-1-jenkins-mgmt-qa/artifacts/test2.txt
      Completed 5 Bytes/5 Bytes (33 Bytes/s) with 1 file(s) remaining
      upload: ./test2.txt to s3://s3-us-east-1-jenkins-mgmt-qa/artifacts/test2.txt
      
      Archiving artifacts
      ERROR: Failed to upload /home/jenkins/agent/workspace/test/test.txt to https://s3-us-east-1-jenkins-mgmt-qa.s3.amazonaws.com/artifacts/test/5/artifacts/test.txt?…, response: 403 Forbidden, body: <?xml version="1.0" encoding="UTF-8"?>
      <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>ES9A612YHZRSHJ30</RequestId><HostId>xChB27LSJU1YG66XDMiU7wuLKPM1j30nVZaj+ce3p9g2iYZjqAaShTDub/L8NTYfjPFHfjSxCgo=</HostId></Error>
      Finished: FAILURE
      

      So you can see the agent has access via its role, the Archive step, fails, because I assume its getting ran from the Master.

       

      Deployment code

       

      serviceAccount:
        create: true
        annotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::29105xxxx:role/eks-epx-mgmt-jenkins-qa-masterserviceAccountAgent:
        create: true
        annotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::29105xxxx:role/eks-epx-mgmt-jenkins-qa-worker
      
      

       

      JCasC Code

       

      controller:
        JCasC:
            globalconfig: |
              unclassified:
                artifactManager:
                  artifactManagerFactories:
                    - jclouds:
                        provider: s3
              aws:
                awsCredentials:
                  region: "us-east-1"
                s3:
                  container: "s3-us-east-1-jenkins-mgmt-qa"
                  prefix: "artifacts/"

      In k8s, I can see that the maser pod, has a service account defined, and it is getting the AWS role information injected into it.

       

       serviceAccount: jenkins-qa
       serviceAccountName: jenkins-qa
        env:
          - name: AWS_DEFAULT_REGION
            value: us-east-1
          - name: AWS_REGION
            value: us-east-1
          - name: AWS_ROLE_ARN
            value: arn:aws:iam::291053455966:role/eks-epx-mgmt-jenkins-qa-worker
          - name: AWS_WEB_IDENTITY_TOKEN_FILE
            value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token

      This leads me to think, that the AWS plugins, is somehow attempting to use the Instance Profile, not the Service Accounts role.

        Attachments

          Activity

          Hide
          jtancer Jon Tancer added a comment -

          Can you confirm which role is being used by the aws container? You can determine this by running `aws sts get-caller-identity`.

          Also can you confirm which version of the aws cli is running inside the aws container? Only newer versions of the aws sdk support assuming an identity with a web token.

          Please confirm that the version meets the minimum required version as defined on this AWS support page - https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html

          Show
          jtancer Jon Tancer added a comment - Can you confirm which role is being used by the aws container? You can determine this by running `aws sts get-caller-identity`. Also can you confirm which version of the aws cli is running inside the aws container? Only newer versions of the aws sdk support assuming an identity with a web token. Please confirm that the version meets the minimum required version as defined on this AWS support page -  https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            jseiser Justin Seiser
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated: