Details
-
Bug
-
Status: Closed (View Workflow)
-
Blocker
-
Resolution: Fixed
-
None
-
* EC2 plugin version 1.26.
* Jenkins 1.580.2 running inside the official Jenkins Docker LTS image.
* Host O/S: Ubuntu 14.04 LTS 64-bit on an EC2 master.
* EC2 rights are conferred via an EC2 InstanceProfile.
Description
After Jenkins first starts it is able to launch EC2 slaves, both manually and when jobs indicate they need to use the slave label.
A few hours later (not sure how long, maybe 24 hours?) slaves no longer start, manually or automatically. In "Manage Jenkins -> System Log -> All Jenkins Logs" the following error occurs repeatedly. Restarting Jenkins solves the problem.
Started EC2 alive slaves monitor Feb 09, 2015 5:14:47 AM INFO hudson.model.AsyncPeriodicWork$1 run Finished EC2 alive slaves monitor. 0 ms Feb 09, 2015 5:15:51 AM INFO hudson.plugins.ec2.EC2Cloud provision Excess workload after pending Spot instances: 1 Feb 09, 2015 5:15:53 AM WARNING hudson.plugins.ec2.EC2Cloud provision Failed to count the # of live instances on EC2 com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 59f7935f-15f0-455c-a6f1-f6057f5ffc77) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:886) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:484) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:256) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:8798) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4137) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:8087) at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:228) at hudson.plugins.ec2.EC2Cloud.addProvisionedSlave(EC2Cloud.java:299) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:389) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:281) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:368) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
I also don't understand the log statement Excess workload after pending Spot instances: 1 as I have not ticked the "Use Spot instance" tick box.
In my cloud settings I have ticked the "Use EC2 instance profile to obtain credentials" and have set both the access key and secret key values to "THIS VALUE IS NOT USED - THE INSTANCE PROFILE IS USED INSTEAD".
Attachments
Activity
James Judd
added a comment - francisu Just curious when 1.28 will be released. 
Francis Upton
added a comment - I think the release actually worked this time, so by tomorrow it should be present on the wiki page and available.
Francis Upton
added a comment - I think the release actually worked this time, so by tomorrow it should be present on the wiki page and available. 
Vincent Rivellino
added a comment - I updated to 1.28 yesterday and have disabled my auto-restart job.
After 19 hours of uptime, no expired credentials. I think things are copacetic.
Vincent Rivellino
added a comment - I updated to 1.28 yesterday and have disabled my auto-restart job.
After 19 hours of uptime, no expired credentials. I think things are copacetic. Francis Upton
added a comment - Fixed in 1.28. 
@Ximon, I had intended to do the release and I think something went wrong. I will try to fix this today.