EC2ConnectionUpdater is not resestablishing connection

This issue is archived. You can view it, but you can't modify it. Learn more

XMLWordPrintable

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major
    • Component/s: ec2-plugin
    • Environment:

      Hi there, we have been noticing this issue where we get the following exception sporadically.

       

      hudson.plugins.ec2.EC2Cloud#provision: SlaveTemplate{description='cbci-amznl2-agent', labels='aws-aml2 aws-linux'}. Exception during provisioning
com.amazonaws.services.ec2.model.AmazonEC2Exception: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 3de0d618-7b0c-4ef1-82f4-8eedf5bd8c88; Proxy: null)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
    at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:34698)
    at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:34665)
    at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:34654)
    at com.amazonaws.services.ec2.AmazonEC2Client.executeDescribeImages(AmazonEC2Client.java:15117)
    at com.amazonaws.services.ec2.AmazonEC2Client.describeImages(AmazonEC2Client.java:15085)
    at hudson.plugins.ec2.SlaveTemplate.getImage(SlaveTemplate.java:1347)
    at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:901)
    at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:717)
    at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:743)
    at hudson.slaves.Cloud.provision(Cloud.java:210)
    at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:726)
    at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:325)
    at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823)
    at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:94)
    at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:69)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

       

      In order to unblock ourselves, we have been saving the cloud configuration with no changes, this seems to reestablish the connection to AWS.

       

      We have noticed that in some cases, there have been logs from the EC2ConnectionUpdater between instances of the above stack trace.

       

      I noticed that the EC2ConnectionUpdater works by catching an AmazonClientException and then triggers the reconnectToEc2 method on EC2Cloud.

       

      The only real difference I can see between the calls made to EC2 from this plugin is that the EC2ConnectionUpdater calls describeInstances whereas the SlaveTemplate calls describeImages.

       

      If you need any more information from me please feel free to reach out.

       

      Thanks,

      Alan.

            Assignee:
            Raihaan Shouhell
            Reporter:
            Alan
            Archiver:
            Jenkins Service Account

              Created:
              Updated:
              Archived: