Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67452

Fail/traceback on AWS Credentials 1.33 and EC2 Fleet

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • aws-credentials-plugin
    • None
    • Jenkins 2.319.1, Cloudbees AWS credentials plugin 1.33, EC2 Fleet 2.4.1
    • 189.v3551d5642995

      After updating to 1.33, repeated failures in EC2 Fleet trying to get Autoscaling status from AWS. Traceback below, reverting to 1.32 returns to working order.

      2021-12-24 19:28:53.554+0000 [id=38] INFO c.a.jenkins.ec2fleet.CloudNanny#doRun: Error during fleet 'FleetCloud' stats update
 com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: 1 validation error detected: Value '' at 'externalId' failed to satisfy constraint: Member must have length greater than or equal to 2 (Service: AWSSecurityTokenService; Status Code: 400; Error Code: ValidationError; Request ID: 8fc4dd97-bb40-4ffb-99f3-c0d464cf9a27; Proxy: null)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1862)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1415)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1154)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:811)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
 at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1682)
 at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1649)
 at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1638)
 at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:498)
 at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:467)
 at com.cloudbees.jenkins.plugins.awscredentials.AWSCredentialsImpl.getCredentials(AWSCredentialsImpl.java:161)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1266)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:842)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:792)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
 at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.doInvoke(AmazonAutoScalingClient.java:4931)
 at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.invoke(AmazonAutoScalingClient.java:4898)
 at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.invoke(AmazonAutoScalingClient.java:4887)
 at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.executeDescribeAutoScalingGroups(AmazonAutoScalingClient.java:1847)
 at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.describeAutoScalingGroups(AmazonAutoScalingClient.java:1815)
 at com.amazon.jenkins.ec2fleet.fleet.AutoScalingGroupFleet.getState(AutoScalingGroupFleet.java:78)
 at com.amazon.jenkins.ec2fleet.EC2FleetCloud.update(EC2FleetCloud.java:464)
 at com.amazon.jenkins.ec2fleet.CloudNanny.doRun(CloudNanny.java:55)
 at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:90)
 at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

          [JENKINS-67452] Fail/traceback on AWS Credentials 1.33 and EC2 Fleet

          Alan Sparks added a comment -

          This also seems very similar to issue in GitHub:
          https://github.com/jenkinsci/aws-credentials-plugin/issues/116

          Alan Sparks added a comment - This also seems very similar to issue in GitHub: https://github.com/jenkinsci/aws-credentials-plugin/issues/116

          Alan Sparks added a comment -

          How can we get some attention on this issue? Updating to 1.33 version is a breaking change for us.

          Alan Sparks added a comment - How can we get some attention on this issue? Updating to 1.33 version is a breaking change for us.

          Alan Sparks added a comment -

          Bumping level on this, since I cannot upgrade this plugin until this issue is resolved.

          Alan Sparks added a comment - Bumping level on this, since I cannot upgrade this plugin until this issue is resolved.

          Alan Sparks added a comment -

          Can this issue get attention? It's a blocker for me.
          -Alan

          Alan Sparks added a comment - Can this issue get attention? It's a blocker for me. -Alan

          James Nord added a comment -

          Please try https://ci.jenkins.io/job/Plugins/job/aws-credentials-plugin/job/PR-125/1/artifact/org/jenkins-ci/plugins/aws-credentials/1.34-rc181.b8986bf42f7e/aws-credentials-1.34-rc181.b8986bf42f7e.hpi and see if it fixes your issue

          James Nord added a comment - Please try https://ci.jenkins.io/job/Plugins/job/aws-credentials-plugin/job/PR-125/1/artifact/org/jenkins-ci/plugins/aws-credentials/1.34-rc181.b8986bf42f7e/aws-credentials-1.34-rc181.b8986bf42f7e.hpi and see if it fixes your issue

          Alan Sparks added a comment -

          This unfortunately didn't seem to work. I uploaded this plugin and restarted, and got a large traceback, starting like:

          2022-02-04 19:44:21.312+0000 [id=30] SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed ConfigurationAsCode.init
          java.lang.NullPointerException
          at com.cloudbees.jenkins.plugins.awscredentials.AWSCredentialsImpl.createAssumeRoleRequest(AWSCredentialsImpl.java:232)
          at com.cloudbees.jenkins.plugins.awscredentials.AWSCredentialsImpl.getCredentials(AWSCredentialsImpl.java:158)
          at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1266)
          at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:842)
          at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:792)
          at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
          at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
          at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
          at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
          at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
          at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
          at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.doInvoke(AmazonAutoScalingClient.java:4931)
          at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.invoke(AmazonAutoScalingClient.java:4898)
          at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.invoke(AmazonAutoScalingClient.java:4887)
          at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.executeDescribeAutoScalingGroups(AmazonAutoScalingClient.java:1847)
          at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.describeAutoScalingGroups(AmazonAutoScalingClient.java:1815)
          at com.amazon.jenkins.ec2fleet.fleet.AutoScalingGroupFleet.getState(AutoScalingGroupFleet.java:78)
          at com.amazon.jenkins.ec2fleet.EC2FleetCloud.<init>(EC2FleetCloud.java:221)
          Caused: java.lang.reflect.InvocationTargetException
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
          at io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.tryConstructor(DataBoundConfigurator.java:173)
          Caused: io.jenkins.plugins.casc.ConfiguratorException: eC2Fleet: Failed to construct instance of class com.amazon.jenkins.ec2fleet.EC2FleetCloud.
          Constructor: public com.amazon.jenkins.ec2fleet.EC2FleetCloud(java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,hudson.slaves.ComputerConnector,boolean,boolean,java.lang.Integer,int,int,int,int,boolean,boolean,java.lang.String,boolean,java.lang.Integer,java.lang.Integer,boolean,java.lang.Integer,boolean).
          Arguments: [java.lang.String, null, java.lang.String, null, java.lang.String, null, java.lang.String, java.lang.String, null, hudson.plugins.sshslaves.SSHConnector, java.lang.Boolean, java.lang.Boolean, java.lang.Integer, java.lang.Integer, java.lang.Integer, java.lang.Integer, java.lang.Integer, java.lang.Boolean, java.lang.Boolean, null, java.lang.Boolean, java.lang.Integer, java.lang.Integer, java.lang.Boolean, java.lang.Integer, java.lang.Boolean].
          Expected Parameters: name java.lang.String, oldId java.lang.String, awsCredentialsId java.lang.String, credentialsId java.lang.String, region java.lang.String, endpoint java.lang.String, fleet java.lang.String, labelString java.lang.String, fsRoot java.lang.String, computerConnector hudson.slaves.ComputerConnector, privateIpUsed boolean, alwaysReconnect boolean, idleMinutes java.lang.Integer, minSize int, maxSize int, minSpareSize int, numExecutors int, addNodeOnlyIfRunning boolean, restrictUsage boolean, maxTotalUses java.lang.String, disableTaskResubmit boolean, initOnlineTimeoutSec java.lang.Integer, initOnlineCheckIntervalSec java.lang.Integer, scaleExecutorsByWeight boolean, cloudStatusIntervalSec java.lang.Integer, noDelayProvision boolean

          Alan Sparks added a comment - This unfortunately didn't seem to work. I uploaded this plugin and restarted, and got a large traceback, starting like: 2022-02-04 19:44:21.312+0000 [id=30] SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed ConfigurationAsCode.init java.lang.NullPointerException at com.cloudbees.jenkins.plugins.awscredentials.AWSCredentialsImpl.createAssumeRoleRequest(AWSCredentialsImpl.java:232) at com.cloudbees.jenkins.plugins.awscredentials.AWSCredentialsImpl.getCredentials(AWSCredentialsImpl.java:158) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1266) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:842) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:792) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539) at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.doInvoke(AmazonAutoScalingClient.java:4931) at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.invoke(AmazonAutoScalingClient.java:4898) at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.invoke(AmazonAutoScalingClient.java:4887) at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.executeDescribeAutoScalingGroups(AmazonAutoScalingClient.java:1847) at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.describeAutoScalingGroups(AmazonAutoScalingClient.java:1815) at com.amazon.jenkins.ec2fleet.fleet.AutoScalingGroupFleet.getState(AutoScalingGroupFleet.java:78) at com.amazon.jenkins.ec2fleet.EC2FleetCloud.<init>(EC2FleetCloud.java:221) Caused: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.tryConstructor(DataBoundConfigurator.java:173) Caused: io.jenkins.plugins.casc.ConfiguratorException: eC2Fleet: Failed to construct instance of class com.amazon.jenkins.ec2fleet.EC2FleetCloud. Constructor: public com.amazon.jenkins.ec2fleet.EC2FleetCloud(java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,hudson.slaves.ComputerConnector,boolean,boolean,java.lang.Integer,int,int,int,int,boolean,boolean,java.lang.String,boolean,java.lang.Integer,java.lang.Integer,boolean,java.lang.Integer,boolean). Arguments: [java.lang.String, null, java.lang.String, null, java.lang.String, null, java.lang.String, java.lang.String, null, hudson.plugins.sshslaves.SSHConnector, java.lang.Boolean, java.lang.Boolean, java.lang.Integer, java.lang.Integer, java.lang.Integer, java.lang.Integer, java.lang.Integer, java.lang.Boolean, java.lang.Boolean, null, java.lang.Boolean, java.lang.Integer, java.lang.Integer, java.lang.Boolean, java.lang.Integer, java.lang.Boolean] . Expected Parameters: name java.lang.String, oldId java.lang.String, awsCredentialsId java.lang.String, credentialsId java.lang.String, region java.lang.String, endpoint java.lang.String, fleet java.lang.String, labelString java.lang.String, fsRoot java.lang.String, computerConnector hudson.slaves.ComputerConnector, privateIpUsed boolean, alwaysReconnect boolean, idleMinutes java.lang.Integer, minSize int, maxSize int, minSpareSize int, numExecutors int, addNodeOnlyIfRunning boolean, restrictUsage boolean, maxTotalUses java.lang.String, disableTaskResubmit boolean, initOnlineTimeoutSec java.lang.Integer, initOnlineCheckIntervalSec java.lang.Integer, scaleExecutorsByWeight boolean, cloudStatusIntervalSec java.lang.Integer, noDelayProvision boolean

          Basil Crow added a comment -

          Fixed in jenkinsci/aws-credentials-plugin#125. Released in 189.v3551d5642995.

          Basil Crow added a comment - Fixed in jenkinsci/aws-credentials-plugin#125 . Released in 189.v3551d5642995 .

            teilo James Nord
            webminster Alan Sparks
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: