Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65080

EKS IRSA Not used for artifact-manager-s3

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Deploying Jenkins via Helm chart, and configuring with Configuration as code.

       

      Jenkins Master has a service Account created, which is annotated to point to a Role.

      Jenkins Agent has a Service Account Created, which is also annotated to a role.

       

      I can see that the Jenkins Master role, is not getting called, so this is almost certainly where the problem is.

      Sample Job Im testing with

       

      pipeline {
        agent {
          kubernetes {
            defaultContainer 'jnlp'
            yaml """
      apiVersion: v1
      kind: Pod
      metadata:
      labels:
        component: ci
      spec:
        serviceAccountName: jenkins-qa-agent
        containers:
        - name: aws
          image: amazon/aws-cli
          command:
          - cat
          tty: true
      """
      }
         }
        stages {
          stage('Test') {
            steps {
              container('aws') {
                sh "aws s3 ls s3://s3-us-east-1-jenkins-mgmt-qa"
                sh "echo test > test.txt"
                sh "echo test > test2.txt"
                sh "aws s3 cp test2.txt s3://s3-us-east-1-jenkins-mgmt-qa/artifacts/test2.txt"
                archiveArtifacts artifacts: 'test.txt', followSymlinks: false
              }
            }
          }
        }
      } The result

       

      + aws s3 cp test2.txt s3://s3-us-east-1-jenkins-mgmt-qa/artifacts/test2.txt
      Completed 5 Bytes/5 Bytes (33 Bytes/s) with 1 file(s) remaining
      upload: ./test2.txt to s3://s3-us-east-1-jenkins-mgmt-qa/artifacts/test2.txt
      
      Archiving artifacts
      ERROR: Failed to upload /home/jenkins/agent/workspace/test/test.txt to https://s3-us-east-1-jenkins-mgmt-qa.s3.amazonaws.com/artifacts/test/5/artifacts/test.txt?…, response: 403 Forbidden, body: <?xml version="1.0" encoding="UTF-8"?>
      <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>ES9A612YHZRSHJ30</RequestId><HostId>xChB27LSJU1YG66XDMiU7wuLKPM1j30nVZaj+ce3p9g2iYZjqAaShTDub/L8NTYfjPFHfjSxCgo=</HostId></Error>
      Finished: FAILURE
      

      So you can see the agent has access via its role, the Archive step, fails, because I assume its getting ran from the Master.

       

      Deployment code

       

      serviceAccount:
        create: true
        annotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::29105xxxx:role/eks-epx-mgmt-jenkins-qa-masterserviceAccountAgent:
        create: true
        annotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::29105xxxx:role/eks-epx-mgmt-jenkins-qa-worker
      
      

       

      JCasC Code

       

      controller:
        JCasC:
            globalconfig: |
              unclassified:
                artifactManager:
                  artifactManagerFactories:
                    - jclouds:
                        provider: s3
              aws:
                awsCredentials:
                  region: "us-east-1"
                s3:
                  container: "s3-us-east-1-jenkins-mgmt-qa"
                  prefix: "artifacts/"

      In k8s, I can see that the maser pod, has a service account defined, and it is getting the AWS role information injected into it.

       

       serviceAccount: jenkins-qa
       serviceAccountName: jenkins-qa
        env:
          - name: AWS_DEFAULT_REGION
            value: us-east-1
          - name: AWS_REGION
            value: us-east-1
          - name: AWS_ROLE_ARN
            value: arn:aws:iam::291053455966:role/eks-epx-mgmt-jenkins-qa-worker
          - name: AWS_WEB_IDENTITY_TOKEN_FILE
            value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token

      This leads me to think, that the AWS plugins, is somehow attempting to use the Instance Profile, not the Service Accounts role.

        Attachments

          Activity

          jseiser Justin Seiser created issue -
          jseiser Justin Seiser made changes -
          Field Original Value New Value
          Description Deploying Jenkins via Helm chart, and configuring with Configuration as code.

           

          Jenkins Master has a service Account created, which is annotated to point to a Role.

          Jenkins Agent has a Service Account Created, which is also annotated to a role.

           

          I can see that the Jenkins Master role, is not getting called, so this is almost certainly where the problem is.

          Sample Job Im testing with

           
          {code:java}
          pipeline {
            agent {
              kubernetes {
                defaultContainer 'jnlp'
                yaml """
          apiVersion: v1
          kind: Pod
          metadata:
          labels:
            component: ci
          spec:
            serviceAccountName: jenkins-qa-agent
            containers:
            - name: aws
              image: amazon/aws-cli
              command:
              - cat
              tty: true
          """
          }
             }
            stages {
              stage('Test') {
                steps {
                  container('aws') {
                    sh "aws s3 ls s3://s3-us-east-1-jenkins-mgmt-qa"
                    sh "echo test > test.txt"
                    sh "echo test > test2.txt"
                    sh "aws s3 cp test2.txt s3://s3-us-east-1-jenkins-mgmt-qa/artifacts/test2.txt"
                    archiveArtifacts artifacts: 'test.txt', followSymlinks: false
                  }
                }
              }
            }
          } The result{code}
           
          {code:java}
          + aws s3 cp test2.txt s3://s3-us-east-1-jenkins-mgmt-qa/artifacts/test2.txt
          Completed 5 Bytes/5 Bytes (33 Bytes/s) with 1 file(s) remaining
          upload: ./test2.txt to s3://s3-us-east-1-jenkins-mgmt-qa/artifacts/test2.txt

          Archiving artifacts
          ERROR: Failed to upload /home/jenkins/agent/workspace/test/test.txt to https://s3-us-east-1-jenkins-mgmt-qa.s3.amazonaws.com/artifacts/test/5/artifacts/test.txt?…, response: 403 Forbidden, body: <?xml version="1.0" encoding="UTF-8"?>
          <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>ES9A612YHZRSHJ30</RequestId><HostId>xChB27LSJU1YG66XDMiU7wuLKPM1j30nVZaj+ce3p9g2iYZjqAaShTDub/L8NTYfjPFHfjSxCgo=</HostId></Error>
          Finished: FAILURE
          {code}
          So you can see the agent has access via its role, the Archive step, fails, because I assume its getting ran from the Master.

           

          Deployment code

           
          {code:java}
          serviceAccount:
            create: true
            annotations:
              eks.amazonaws.com/role-arn: arn:aws:iam::291053455966:role/eks-epx-mgmt-jenkins-qa-masterserviceAccountAgent:
            create: true
            annotations:
              eks.amazonaws.com/role-arn: arn:aws:iam::291053455966:role/eks-epx-mgmt-jenkins-qa-worker

          {code}
           

          JCasC Code

           
          {code:java}
          controller:
            JCasC:
                globalconfig: |
                  unclassified:
                    artifactManager:
                      artifactManagerFactories:
                        - jclouds:
                            provider: s3
                  aws:
                    awsCredentials:
                      region: "us-east-1"
                    s3:
                      container: "s3-us-east-1-jenkins-mgmt-qa"
                      prefix: "artifacts/"{code}
          In k8s, I can see that the maser pod, has a service account defined, and it is getting the AWS role information injected into it.

           
          {code:java}
           serviceAccount: jenkins-qa
           serviceAccountName: jenkins-qa
            env:
              - name: AWS_DEFAULT_REGION
                value: us-east-1
              - name: AWS_REGION
                value: us-east-1
              - name: AWS_ROLE_ARN
                value: arn:aws:iam::291053455966:role/eks-epx-mgmt-jenkins-qa-worker
              - name: AWS_WEB_IDENTITY_TOKEN_FILE
                value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token{code}

          This leads me to think, that the AWS plugins, is somehow attempting to use the Instance Profile, not the Service Accounts role.
          Deploying Jenkins via Helm chart, and configuring with Configuration as code.

           

          Jenkins Master has a service Account created, which is annotated to point to a Role.

          Jenkins Agent has a Service Account Created, which is also annotated to a role.

           

          I can see that the Jenkins Master role, is not getting called, so this is almost certainly where the problem is.

          Sample Job Im testing with

           
          {code:java}
          pipeline {
            agent {
              kubernetes {
                defaultContainer 'jnlp'
                yaml """
          apiVersion: v1
          kind: Pod
          metadata:
          labels:
            component: ci
          spec:
            serviceAccountName: jenkins-qa-agent
            containers:
            - name: aws
              image: amazon/aws-cli
              command:
              - cat
              tty: true
          """
          }
             }
            stages {
              stage('Test') {
                steps {
                  container('aws') {
                    sh "aws s3 ls s3://s3-us-east-1-jenkins-mgmt-qa"
                    sh "echo test > test.txt"
                    sh "echo test > test2.txt"
                    sh "aws s3 cp test2.txt s3://s3-us-east-1-jenkins-mgmt-qa/artifacts/test2.txt"
                    archiveArtifacts artifacts: 'test.txt', followSymlinks: false
                  }
                }
              }
            }
          } The result{code}
           
          {code:java}
          + aws s3 cp test2.txt s3://s3-us-east-1-jenkins-mgmt-qa/artifacts/test2.txt
          Completed 5 Bytes/5 Bytes (33 Bytes/s) with 1 file(s) remaining
          upload: ./test2.txt to s3://s3-us-east-1-jenkins-mgmt-qa/artifacts/test2.txt

          Archiving artifacts
          ERROR: Failed to upload /home/jenkins/agent/workspace/test/test.txt to https://s3-us-east-1-jenkins-mgmt-qa.s3.amazonaws.com/artifacts/test/5/artifacts/test.txt?…, response: 403 Forbidden, body: <?xml version="1.0" encoding="UTF-8"?>
          <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>ES9A612YHZRSHJ30</RequestId><HostId>xChB27LSJU1YG66XDMiU7wuLKPM1j30nVZaj+ce3p9g2iYZjqAaShTDub/L8NTYfjPFHfjSxCgo=</HostId></Error>
          Finished: FAILURE
          {code}
          So you can see the agent has access via its role, the Archive step, fails, because I assume its getting ran from the Master.

           

          Deployment code

           
          {code:java}
          serviceAccount:
            create: true
            annotations:
              eks.amazonaws.com/role-arn: arn:aws:iam::29105xxxx:role/eks-epx-mgmt-jenkins-qa-masterserviceAccountAgent:
            create: true
            annotations:
              eks.amazonaws.com/role-arn: arn:aws:iam::29105xxxx:role/eks-epx-mgmt-jenkins-qa-worker

          {code}
           

          JCasC Code

           
          {code:java}
          controller:
            JCasC:
                globalconfig: |
                  unclassified:
                    artifactManager:
                      artifactManagerFactories:
                        - jclouds:
                            provider: s3
                  aws:
                    awsCredentials:
                      region: "us-east-1"
                    s3:
                      container: "s3-us-east-1-jenkins-mgmt-qa"
                      prefix: "artifacts/"{code}
          In k8s, I can see that the maser pod, has a service account defined, and it is getting the AWS role information injected into it.

           
          {code:java}
           serviceAccount: jenkins-qa
           serviceAccountName: jenkins-qa
            env:
              - name: AWS_DEFAULT_REGION
                value: us-east-1
              - name: AWS_REGION
                value: us-east-1
              - name: AWS_ROLE_ARN
                value: arn:aws:iam::291053455966:role/eks-epx-mgmt-jenkins-qa-worker
              - name: AWS_WEB_IDENTITY_TOKEN_FILE
                value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token{code}

          This leads me to think, that the AWS plugins, is somehow attempting to use the Instance Profile, not the Service Accounts role.
          jseiser Justin Seiser made changes -
          Component/s aws-global-configuration-plugin [ 23929 ]
          Hide
          jtancer Jon Tancer added a comment -

          Can you confirm which role is being used by the aws container? You can determine this by running `aws sts get-caller-identity`.

          Also can you confirm which version of the aws cli is running inside the aws container? Only newer versions of the aws sdk support assuming an identity with a web token.

          Please confirm that the version meets the minimum required version as defined on this AWS support page - https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html

          Show
          jtancer Jon Tancer added a comment - Can you confirm which role is being used by the aws container? You can determine this by running `aws sts get-caller-identity`. Also can you confirm which version of the aws cli is running inside the aws container? Only newer versions of the aws sdk support assuming an identity with a web token. Please confirm that the version meets the minimum required version as defined on this AWS support page -  https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            jseiser Justin Seiser
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated: