Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-26854

EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • ec2-plugin
    • None
    • * EC2 plugin version 1.26.
      * Jenkins 1.580.2 running inside the official Jenkins Docker LTS image.
      * Host O/S: Ubuntu 14.04 LTS 64-bit on an EC2 master.
      * EC2 rights are conferred via an EC2 InstanceProfile.

      After Jenkins first starts it is able to launch EC2 slaves, both manually and when jobs indicate they need to use the slave label.

      A few hours later (not sure how long, maybe 24 hours?) slaves no longer start, manually or automatically. In "Manage Jenkins -> System Log -> All Jenkins Logs" the following error occurs repeatedly. Restarting Jenkins solves the problem.

      Started EC2 alive slaves monitor
Feb 09, 2015 5:14:47 AM INFO hudson.model.AsyncPeriodicWork$1 run
Finished EC2 alive slaves monitor. 0 ms
Feb 09, 2015 5:15:51 AM INFO hudson.plugins.ec2.EC2Cloud provision
Excess workload after pending Spot instances: 1
Feb 09, 2015 5:15:53 AM WARNING hudson.plugins.ec2.EC2Cloud provision
Failed to count the # of live instances on EC2
com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 59f7935f-15f0-455c-a6f1-f6057f5ffc77)
	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:886)
	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:484)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:256)
	at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:8798)
	at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4137)
	at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:8087)
	at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:228)
	at hudson.plugins.ec2.EC2Cloud.addProvisionedSlave(EC2Cloud.java:299)
	at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:389)
	at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:281)
	at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51)
	at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:368)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

      I also don't understand the log statement Excess workload after pending Spot instances: 1 as I have not ticked the "Use Spot instance" tick box.

      In my cloud settings I have ticked the "Use EC2 instance profile to obtain credentials" and have set both the access key and secret key values to "THIS VALUE IS NOT USED - THE INSTANCE PROFILE IS USED INSTEAD".

          [JENKINS-26854] EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"

          Ximon Eighteen created issue -
          Ximon Eighteen made changes -
          Description Original: After Jenkins first starts it is able to launch EC2 slaves, both manually and when jobs indicate they need to use the slave label.

          A few hours later (not sure how long, maybe 24 hours?) slaves no longer start, manually or automatically. In "Manage Jenkins -> System Log -> All Jenkins Logs" the following error occurs repeatedly. Restarting Jenkins solves the problem.

          {code}Started EC2 alive slaves monitor
          Feb 09, 2015 5:14:47 AM INFO hudson.model.AsyncPeriodicWork$1 run
          Finished EC2 alive slaves monitor. 0 ms
          Feb 09, 2015 5:15:51 AM INFO hudson.plugins.ec2.EC2Cloud provision
          Excess workload after pending Spot instances: 1
          Feb 09, 2015 5:15:53 AM WARNING hudson.plugins.ec2.EC2Cloud provision
          Failed to count the # of live instances on EC2
          com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 59f7935f-15f0-455c-a6f1-f6057f5ffc77)
          at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:886)
          at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:484)
          at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:256)
          at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:8798)
          at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4137)
          at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:8087)
          at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:228)
          at hudson.plugins.ec2.EC2Cloud.addProvisionedSlave(EC2Cloud.java:299)
          at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:389)
          at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:281)
          at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51)
          at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:368)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745){code}

          I also don't understand the log statement {{Excess workload after pending Spot instances: 1}} as I have not ticked the "Use Spot instance" tick box.           		New: After Jenkins first starts it is able to launch EC2 slaves, both manually and when jobs indicate they need to use the slave label.

          A few hours later (not sure how long, maybe 24 hours?) slaves no longer start, manually or automatically. In "Manage Jenkins -> System Log -> All Jenkins Logs" the following error occurs repeatedly. Restarting Jenkins solves the problem.

          {code}Started EC2 alive slaves monitor
          Feb 09, 2015 5:14:47 AM INFO hudson.model.AsyncPeriodicWork$1 run
          Finished EC2 alive slaves monitor. 0 ms
          Feb 09, 2015 5:15:51 AM INFO hudson.plugins.ec2.EC2Cloud provision
          Excess workload after pending Spot instances: 1
          Feb 09, 2015 5:15:53 AM WARNING hudson.plugins.ec2.EC2Cloud provision
          Failed to count the # of live instances on EC2
          com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 59f7935f-15f0-455c-a6f1-f6057f5ffc77)
          at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:886)
          at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:484)
          at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:256)
          at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:8798)
          at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4137)
          at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:8087)
          at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:228)
          at hudson.plugins.ec2.EC2Cloud.addProvisionedSlave(EC2Cloud.java:299)
          at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:389)
          at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:281)
          at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51)
          at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:368)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745){code}

          I also don't understand the log statement {{Excess workload after pending Spot instances: 1}} as I have not ticked the "Use Spot instance" tick box.

          In my cloud settings I have ticked the "Use EC2 instance profile to obtain credentials" and have set both the access key and secret key values to "THIS VALUE IS NOT USED - THE INSTANCE PROFILE IS USED INSTEAD".
          Environment Original: Jenkins 1.580.2 running inside the official Jenkins Docker LTS image, running inside Ubuntu 14.04 LTS 64-bit on an EC2 master. New: Jenkins 1.580.2 running inside the official Jenkins Docker LTS image, running inside Ubuntu 14.04 LTS 64-bit on an EC2 master. EC2 rights are conferred via an EC2 InstanceProfile.
          EC2 plugin version 1.26.
          Ximon Eighteen made changes -
          Environment Original: Jenkins 1.580.2 running inside the official Jenkins Docker LTS image, running inside Ubuntu 14.04 LTS 64-bit on an EC2 master. EC2 rights are conferred via an EC2 InstanceProfile.
          EC2 plugin version 1.26.           		New: * EC2 plugin version 1.26.
          * Jenkins 1.580.2 running inside the official Jenkins Docker LTS image.
          * Host O/S: Ubuntu 14.04 LTS 64-bit on an EC2 master.
          * EC2 rights are conferred via an EC2 InstanceProfile.
          Francis Upton made changes -
          Resolution New: Fixed [ 1 ]
          Status Original: Open [ 1 ] New: Resolved [ 5 ]
          Francis Upton made changes -
          Status Original: Resolved [ 5 ] New: Closed [ 6 ]
          R. Tyler Croy made changes -
          Workflow Original: JNJira [ 161028 ] New: JNJira + In-Review [ 208427 ]

            francisu Francis Upton
            ximon18 Ximon Eighteen
            Votes:
            5 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: