-
Bug
-
Resolution: Fixed
-
Major
-
None
-
Artifact Manager on S3: 894.v29efa_d1a_6383
Jenkins: 2.479.2
-
Powered by SuggestiMate -
140.vc08280b_30015
After installing version v139.v0b_c2603876b_c of the AWS Global Configuration plugin, our Jenkins jobs that call stash/unstash (handled by the Artifact Manager on S3 plugin) started failing with the below error. It appears the jobs are no longer assuming the role defined by IRSA (service account role annotation) and are instead using default instance profile rights which are limited by design. For reference, these jobs are running as pods in an EKS cluster.
The Artifact Manager on S3 is configured to use "IAM instance Profile/user AWS configuration". This works correctly after downgrading AWS Global Configuration to the prior plugin version 130.v35b_7b_96f53c3. I suspect some of the changes in sessionCredentialsFromInstanceProfile may be involved but need to research more.
Error seen with v139.v0b_c2603876b_c:
2024-12-17 17:26:29.555+0000 [id=86] WARNING hudson.model.Run#getArtifactsUpTo
{{hudson.AbortException: Authorization failed: User: arn:aws:sts::XXXX:assumed-role/my-role/i-iid is not authorized to perform: s3:ListBucket on resource: "arn:aws:s3:::my-bucket" because no identity-based policy allows the s3:ListBucket action request GET https://my-bucket.s3-us-east-1.amazonaws.com/?prefix=plugin/utilities/lighthouseci/248/artifacts/ HTTP/1.1 failed with code 403, error: AWSError{requestId='NR9PB1SP6J3QTVC0', requestToken='8JeF4F7LhuA5AY91+IY4rXXogXq3UrYxrO/5+AylR0p9nCiivrZQrBPTl6PXRYO63/kfVoiLfSeVFx7WqgNUjyiWln3lfnIajNKNplILv9Q=', code='AccessDenied', message='User: arn:aws:sts::XXXX:assumed-role/my-role/i-iid is not authorized to perform: s3:ListBucket on resource: "arn:aws:s3:::my-bucket" because no identity-based policy allows the s3:ListBucket action', context='
'}}}
at PluginClassLoader for artifact-manager-s3//io.jenkins.plugins.artifact_manager_jclouds.JCloudsVirtualFile.run(JCloudsVirtualFile.java:336)
[JENKINS-75014] Plugin v139.v0b_c2603876b_c breaks IRSA access for Artifact Manager on S3
zdvickery Does checking the "Disable Session Token" checkbox chase away the problem?
Also, can you elaborate on how you expect Jenkins to find the AWS access key ID, secret access key, and session token? Since you said you checked "IAM instance Profile/user AWS configuration", are you using a ~/.aws/credentials file with aws_access_key_id, aws_secret_access_key, and aws_session_token defined? If not, how are the AWS access key ID, secret access key, and session token being exposed to Jenkins?
I think I see the problem. For IRSA the WebIdentityCredentialsUtils class is used, which doesn't have access to the STS modules on its classpath. I will post a fix shortly.
zdvickery Incremental build 140.v335a_9a_a_75e89 is available for testing. Download link:
Plugin Installation Manager input format: (documentation)
aws-global-configuration:incrementals;io.jenkins.plugins;140.v335a_9a_a_75e89
Plugin installation instructions
Can you please test with this incremental build and confirm the fix is working?
The new version works great! Thank you so much for your quick fix on this!!
Fixed in jenkinsci/aws-global-configuration-plugin#95. Released in 140.vc08280b_30015.
This morning I encountered the below stack on some jobs attempting a stash using the new version. I'm not sure if this is related or a new bug entirely?
09:01:58 Also: org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 53e7dbc4-e1e7-4ff8-b69d-808e1d9b44d7
09:01:58 java.lang.IllegalStateException: Connection pool shut down
09:01:58 at PluginClassLoader for apache-httpcomponents-client-4-api//org.apache.http.util.Asserts.check(Asserts.java:34)
09:01:58 at PluginClassLoader for apache-httpcomponents-client-4-api//org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:75)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$InstrumentedHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:57)
09:01:58 at PluginClassLoader for apache-httpcomponents-client-4-api//org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176)
09:01:58 at PluginClassLoader for apache-httpcomponents-client-4-api//org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
09:01:58 at PluginClassLoader for apache-httpcomponents-client-4-api//org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
09:01:58 at PluginClassLoader for apache-httpcomponents-client-4-api//org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
09:01:58 at PluginClassLoader for apache-httpcomponents-client-4-api//org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:254)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:104)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:231)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:228)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:102)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:79)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:57)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:40)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:74)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:43)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:79)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:41)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.executeRequest(RetryableStage2.java:93)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:56)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:36)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
09:01:58 at PluginClassLoader for aws-java-sdk2-sts//software.amazon.awssdk.services.sts.DefaultStsClient.assumeRoleWithWebIdentity(DefaultStsClient.java:755)
09:01:58 at PluginClassLoader for aws-java-sdk2-sts//software.amazon.awssdk.services.sts.auth.StsAssumeRoleWithWebIdentityCredentialsProvider.getUpdatedCredentials(StsAssumeRoleWithWebIdentityCredentialsProvider.java:76)
09:01:58 at PluginClassLoader for aws-java-sdk2-sts//software.amazon.awssdk.services.sts.auth.StsCredentialsProvider.updateSessionCredentials(StsCredentialsProvider.java:93)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.utils.cache.CachedSupplier.lambda$jitteredPrefetchValueSupplier$8(CachedSupplier.java:300)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.utils.cache.CachedSupplier$PrefetchStrategy.fetch(CachedSupplier.java:448)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:208)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:135)
09:01:58 at PluginClassLoader for aws-java-sdk2-sts//software.amazon.awssdk.services.sts.auth.StsCredentialsProvider.resolveCredentials(StsCredentialsProvider.java:106)
09:01:58 at PluginClassLoader for aws-java-sdk2-sts//software.amazon.awssdk.services.sts.internal.StsWebIdentityCredentialsProviderFactory$StsWebIdentityCredentialsProvider.resolveCredentials(StsWebIdentityCredentialsProviderFactory.java:109)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider.resolveCredentials(WebIdentityTokenFileCredentialsProvider.java:141)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.auth.credentials.AwsCredentialsProvider.resolveIdentity(AwsCredentialsProvider.java:54)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.identity.spi.IdentityProvider.resolveIdentity(IdentityProvider.java:60)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:103)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45)
09:01:58 at PluginClassLoader for aws-java-sdk2-core//software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:129)
09:01:58 at PluginClassLoader for aws-global-configuration//io.jenkins.plugins.aws.global_configuration.CredentialsAwsGlobalConfiguration.sessionCredentialsFromInstanceProfile(CredentialsAwsGlobalConfiguration.java:188)
09:01:58 at PluginClassLoader for aws-global-configuration//io.jenkins.plugins.aws.global_configuration.CredentialsAwsGlobalConfiguration.sessionCredentials(CredentialsAwsGlobalConfiguration.java:220)
09:01:58 at PluginClassLoader for aws-global-configuration//io.jenkins.plugins.aws.global_configuration.CredentialsAwsGlobalConfiguration.sessionCredentials(CredentialsAwsGlobalConfiguration.java:203)
09:01:58 at PluginClassLoader for artifact-manager-s3//io.jenkins.plugins.artifact_manager_jclouds.s3.S3BlobStore.getCredentialsSupplier(S3BlobStore.java:168)
09:01:58 at PluginClassLoader for artifact-manager-s3//io.jenkins.plugins.artifact_manager_jclouds.s3.S3BlobStore.getContext(S3BlobStore.java:129)
09:01:58 at PluginClassLoader for artifact-manager-s3//io.jenkins.plugins.artifact_manager_jclouds.JCloudsArtifactManager.getContext(JCloudsArtifactManager.java:384)
09:01:58 at PluginClassLoader for artifact-manager-s3//io.jenkins.plugins.artifact_manager_jclouds.JCloudsArtifactManager.stash(JCloudsArtifactManager.java:223)
09:01:58 at PluginClassLoader for workflow-api//org.jenkinsci.plugins.workflow.flow.StashManager.stash(StashManager.java:118)
09:01:58 at PluginClassLoader for workflow-basic-steps//org.jenkinsci.plugins.workflow.support.steps.stash.StashStep$Execution.run(StashStep.java:119)
09:01:58 at PluginClassLoader for workflow-basic-steps//org.jenkinsci.plugins.workflow.support.steps.stash.StashStep$Execution.run(StashStep.java:107)
09:01:58 at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
09:01:58 at PluginClassLoader for opentelemetry-api//io.opentelemetry.context.Context.lambda$wrap$1(Context.java:241)
09:01:58 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
09:01:58 at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
09:01:58 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
09:01:58 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
09:01:58 at java.base/java.lang.Thread.run(Unknown Source)
I'm not sure if this is related or a new bug entirely
Please file a separate issue with more information, including your scenario and steps to reproduce the problem from scratch on a new Jenkins installation.
I'm unable to duplicate the problem based on the issue description. Please provide a set of steps to reproduce the issue from scratch on a clean Jenkins installation. Alternatively, you can debug the issue and file a pull request if you find a solution.