Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65080

EKS IRSA Not used for artifact-manager-s3

    XMLWordPrintable

Details

    Description

      Deploying Jenkins via Helm chart, and configuring with Configuration as code.

       

      Jenkins Master has a service Account created, which is annotated to point to a Role.

      Jenkins Agent has a Service Account Created, which is also annotated to a role.

       

      I can see that the Jenkins Master role, is not getting called, so this is almost certainly where the problem is.

      Sample Job Im testing with

       

      pipeline {
        agent {
          kubernetes {
            defaultContainer 'jnlp'
            yaml """
      apiVersion: v1
      kind: Pod
      metadata:
      labels:
        component: ci
      spec:
        serviceAccountName: jenkins-qa-agent
        containers:
        - name: aws
          image: amazon/aws-cli
          command:
          - cat
          tty: true
      """
      }
         }
        stages {
          stage('Test') {
            steps {
              container('aws') {
                sh "aws s3 ls s3://s3-us-east-1-jenkins-mgmt-qa"
                sh "echo test > test.txt"
                sh "echo test > test2.txt"
                sh "aws s3 cp test2.txt s3://s3-us-east-1-jenkins-mgmt-qa/artifacts/test2.txt"
                archiveArtifacts artifacts: 'test.txt', followSymlinks: false
              }
            }
          }
        }
      } The result

       

      + aws s3 cp test2.txt s3://s3-us-east-1-jenkins-mgmt-qa/artifacts/test2.txt
      Completed 5 Bytes/5 Bytes (33 Bytes/s) with 1 file(s) remaining
      upload: ./test2.txt to s3://s3-us-east-1-jenkins-mgmt-qa/artifacts/test2.txt
      
      Archiving artifacts
      ERROR: Failed to upload /home/jenkins/agent/workspace/test/test.txt to https://s3-us-east-1-jenkins-mgmt-qa.s3.amazonaws.com/artifacts/test/5/artifacts/test.txt?…, response: 403 Forbidden, body: <?xml version="1.0" encoding="UTF-8"?>
      <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>ES9A612YHZRSHJ30</RequestId><HostId>xChB27LSJU1YG66XDMiU7wuLKPM1j30nVZaj+ce3p9g2iYZjqAaShTDub/L8NTYfjPFHfjSxCgo=</HostId></Error>
      Finished: FAILURE
      

      So you can see the agent has access via its role, the Archive step, fails, because I assume its getting ran from the Master.

       

      Deployment code

       

      serviceAccount:
        create: true
        annotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::29105xxxx:role/eks-epx-mgmt-jenkins-qa-masterserviceAccountAgent:
        create: true
        annotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::29105xxxx:role/eks-epx-mgmt-jenkins-qa-worker
      
      

       

      JCasC Code

       

      controller:
        JCasC:
            globalconfig: |
              unclassified:
                artifactManager:
                  artifactManagerFactories:
                    - jclouds:
                        provider: s3
              aws:
                awsCredentials:
                  region: "us-east-1"
                s3:
                  container: "s3-us-east-1-jenkins-mgmt-qa"
                  prefix: "artifacts/"

      In k8s, I can see that the maser pod, has a service account defined, and it is getting the AWS role information injected into it.

       

       serviceAccount: jenkins-qa
       serviceAccountName: jenkins-qa
        env:
          - name: AWS_DEFAULT_REGION
            value: us-east-1
          - name: AWS_REGION
            value: us-east-1
          - name: AWS_ROLE_ARN
            value: arn:aws:iam::291053455966:role/eks-epx-mgmt-jenkins-qa-worker
          - name: AWS_WEB_IDENTITY_TOKEN_FILE
            value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token

      This leads me to think, that the AWS plugins, is somehow attempting to use the Instance Profile, not the Service Accounts role.

      Attachments

        Activity

          jtancer Jon Tancer added a comment -

          Can you confirm which role is being used by the aws container? You can determine this by running `aws sts get-caller-identity`.

          Also can you confirm which version of the aws cli is running inside the aws container? Only newer versions of the aws sdk support assuming an identity with a web token.

          Please confirm that the version meets the minimum required version as defined on this AWS support page - https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html

          jtancer Jon Tancer added a comment - Can you confirm which role is being used by the aws container? You can determine this by running `aws sts get-caller-identity`. Also can you confirm which version of the aws cli is running inside the aws container? Only newer versions of the aws sdk support assuming an identity with a web token. Please confirm that the version meets the minimum required version as defined on this AWS support page -  https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html
          jglick Jesse Glick added a comment -

          While artifact upload happens from inside the agent pod, the signature is precalculated by the controller, so it is the role assumed by the controller pod which would be relevant here. Jenkins is not using the AWS CLI for this purpose, it is using the AWS SDK for Java, wrapped in a plugin. Hard to say offhand where the problem lies, but smells like a misconfiguration rather than a product bug.

          jglick Jesse Glick added a comment - While artifact upload happens from inside the agent pod, the signature is precalculated by the controller, so it is the role assumed by the controller pod which would be relevant here. Jenkins is not using the AWS CLI for this purpose, it is using the AWS SDK for Java, wrapped in a plugin. Hard to say offhand where the problem lies, but smells like a misconfiguration rather than a product bug.
          topikachu Yi Gong added a comment -

          I can confirm that EKS IRSA works well of this plugin.

          The plugin uses the IAM role of the controller.

          topikachu Yi Gong added a comment - I can confirm that EKS IRSA works well of this plugin. The plugin uses the IAM role of the controller.
          jglick Jesse Glick added a comment -

          Likely a misconfiguration.

          jglick Jesse Glick added a comment - Likely a misconfiguration.

          People

            Unassigned Unassigned
            jseiser Justin Seiser
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: