Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-26854

EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"



    • Bug
    • Resolution: Fixed
    • Blocker
    • ec2-plugin
    • None
    • * EC2 plugin version 1.26.
      * Jenkins 1.580.2 running inside the official Jenkins Docker LTS image.
      * Host O/S: Ubuntu 14.04 LTS 64-bit on an EC2 master.
      * EC2 rights are conferred via an EC2 InstanceProfile.


      After Jenkins first starts it is able to launch EC2 slaves, both manually and when jobs indicate they need to use the slave label.

      A few hours later (not sure how long, maybe 24 hours?) slaves no longer start, manually or automatically. In "Manage Jenkins -> System Log -> All Jenkins Logs" the following error occurs repeatedly. Restarting Jenkins solves the problem.

      Started EC2 alive slaves monitor
      Feb 09, 2015 5:14:47 AM INFO hudson.model.AsyncPeriodicWork$1 run
      Finished EC2 alive slaves monitor. 0 ms
      Feb 09, 2015 5:15:51 AM INFO hudson.plugins.ec2.EC2Cloud provision
      Excess workload after pending Spot instances: 1
      Feb 09, 2015 5:15:53 AM WARNING hudson.plugins.ec2.EC2Cloud provision
      Failed to count the # of live instances on EC2
      com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 59f7935f-15f0-455c-a6f1-f6057f5ffc77)
      	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:886)
      	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:484)
      	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:256)
      	at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:8798)
      	at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4137)
      	at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:8087)
      	at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:228)
      	at hudson.plugins.ec2.EC2Cloud.addProvisionedSlave(EC2Cloud.java:299)
      	at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:389)
      	at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:281)
      	at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51)
      	at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:368)
      	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)

      I also don't understand the log statement Excess workload after pending Spot instances: 1 as I have not ticked the "Use Spot instance" tick box.

      In my cloud settings I have ticked the "Use EC2 instance profile to obtain credentials" and have set both the access key and secret key values to "THIS VALUE IS NOT USED - THE INSTANCE PROFILE IS USED INSTEAD".




            francisu Francis Upton
            ximon18 Ximon Eighteen
            5 Vote for this issue
            10 Start watching this issue