-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
AWS EKS 1.19
configuration-as-code-secret-ssm-plugin:1.0.1
aws-java-sdk:1.11.995
The latest version of the aws-java-sdk plugin (aws-java-sdk:1.11.995)breaks the IRSA functionality of the configuration-as-code-secret-ssm-plugin. When deploying a fresh Jenkins instance, instead of using the mounted web identity token from IRSA to retrieve the SSM parameter value, the configuration-as-code-secret-ssm-plugin uses the node role instead. Because the node role doesn't have access to the credential in SSM, this causes an error on bootup. (full stack trace listed below)
This issue can be bypassed by pinning the aws-java-sdk plugin to the current-1 version (aws-java-sdk:1.11.976). When using the older version of the aws-java-sdk plugin, the configuration-as-code-secret-ssm-plugin correctly uses IRSA to retrieve the SSM parameter instead of the EKS node role.
2021-04-22 14:04:11.367+0000 [id=34] SEVERE c.b.j.p.c.s.s.AwsSsmSecretSource#reveal: Error getting ssm secret: /jenkins/google/client_secret com.amazonaws.services.simplesystemsmanagement.model.AWSSimpleSystemsManagementException: User: arn:aws:sts::xxxxxxxxxxx:assumed-role/cluster-node-role-xxxxxxxxxxxx/x-xxxxxxxxxx at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1695) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1350) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1101) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:758) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:732) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:714) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:674) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:656) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:520) at com.amazonaws.services.simplesystemsmanagement.AWSSimpleSystemsManagementClient.doInvoke(AWSSimpleSystemsManagementClient.java:8219) at com.amazonaws.services.simplesystemsmanagement.AWSSimpleSystemsManagementClient.invoke(AWSSimpleSystemsManagementClient.java:8186) at com.amazonaws.services.simplesystemsmanagement.AWSSimpleSystemsManagementClient.invoke(AWSSimpleSystemsManagementClient.java:8175) at com.amazonaws.services.simplesystemsmanagement.AWSSimpleSystemsManagementClient.executeGetParameter(AWSSimpleSystemsManagementClient.java:4952) at com.amazonaws.services.simplesystemsmanagement.AWSSimpleSystemsManagementClient.getParameter(AWSSimpleSystemsManagementClient.java:4924) at com.bambora.jenkins.plugin.casc.secrets.ssm.AwsSsmSecretSource.reveal(AwsSsmSecretSource.java:35) at io.jenkins.plugins.casc.SecretSourceResolver$ConfigurationContextStringLookup.lambda$lookup$ad236547$1(SecretSourceResolver.java:136) at io.vavr.CheckedFunction0.lambda$unchecked$52349c75$1(CheckedFunction0.java:247) at io.jenkins.plugins.casc.SecretSourceResolver$ConfigurationContextStringLookup.lambda$lookup$0(SecretSourceResolver.java:136) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1361) at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:531) at io.jenkins.plugins.casc.SecretSourceResolver$ConfigurationContextStringLookup.lookup(SecretSourceResolver.java:138) at org.apache.commons.text.lookup.InterpolatorStringLookup.lookup(InterpolatorStringLookup.java:144) at org.apache.commons.text.StringSubstitutor.resolveVariable(StringSubstitutor.java:1067) at org.apache.commons.text.StringSubstitutor.substitute(StringSubstitutor.java:1433) at org.apache.commons.text.StringSubstitutor.substitute(StringSubstitutor.java:1308) at org.apache.commons.text.StringSubstitutor.replaceIn(StringSubstitutor.java:1019) at io.jenkins.plugins.casc.SecretSourceResolver.resolve(SecretSourceResolver.java:104) at io.jenkins.plugins.casc.impl.configurators.PrimitiveConfigurator.configure(PrimitiveConfigurator.java:44) at io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.tryConstructor(DataBoundConfigurator.java:160) at io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.instance(DataBoundConfigurator.java:77) at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:267) at io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.configure(DataBoundConfigurator.java:83) at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.lambda$doConfigure$16668e2$1(HeteroDescribableConfigurator.java:277) at io.vavr.CheckedFunction0.lambda$unchecked$52349c75$1(CheckedFunction0.java:247) at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.doConfigure(HeteroDescribableConfigurator.java:277) at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.lambda$configure$2(HeteroDescribableConfigurator.java:86) at io.vavr.control.Option.map(Option.java:392) at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.lambda$configure$3(HeteroDescribableConfigurator.java:86) at io.vavr.Tuple2.apply(Tuple2.java:238) at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.configure(HeteroDescribableConfigurator.java:83) at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.configure(HeteroDescribableConfigurator.java:55) at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:352) at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:270) at io.jenkins.plugins.casc.ConfigurationAsCode.lambda$configureWith$6(ConfigurationAsCode.java:745) at io.jenkins.plugins.casc.ConfigurationAsCode.invokeWith(ConfigurationAsCode.java:689) at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:745) at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:614) at io.jenkins.plugins.casc.ConfigurationAsCode.configure(ConfigurationAsCode.java:298) at io.jenkins.plugins.casc.ConfigurationAsCode.init(ConfigurationAsCode.java:290) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104) at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1131) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
For what it's worth, the version number of the `aws-java-sdk` plugin doesn't seem to affect fresh Jenkins deployments which are deployed using the Jenkins helm chart version `3.3.0` and prior. Anything `3.3.1` or after suffers this issue. Seems that the only change to the chart involves an additional secrets array, which I'm not using. See changeset here - https://github.com/jenkinsci/helm-charts/commit/f6316c95d264dbf064d0c3cc51836b364650273e