Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
Jenkins 1.572, EC2 plugin 1.21, Node Iterator API Plugin 1.5
Description
The Jenkins EC2 plugin no longer launches stopped nodes. Unfortunately I'm not sure exactly when it stopped working - I wasn't sure that was the issue until later, due to unrelated issues caused by too many nodes spawning and having to be killed.
If I use Manage Jenkins -> Manage Nodes to start a stopped EC2 node that a build is waiting on manually, the build proceeds.
Builds succeed when the EC2 plugin spawns a new node for the first time. It's only a problem if the node is stopped for idleness - the plugin doesn't seem to restart it.
Builds get stuck with output like:
Triggering bdr_linux ? x64,debian7 Triggering bdr_linux ? x86,amazonlinux201209 Triggering bdr_linux ? x86,debian6 Triggering bdr_linux ? x64,amazonlinux201209 Configuration bdr_linux ? x86,amazonlinux201209 is still in the queue: Amazon Linux 2012.09 EBS 32-bit (i-b848fbfa) is offline Configuration bdr_linux ? x86,amazonlinux201209 is still in the queue: All nodes of label ?amazonlinux201209&&x86? are offline
where there's at least one node with that label stopped, ready to start and use.
There's no sign that any attempt is made to start the node.
Attachments
Issue Links
- is related to
-
-
- Resolved
-
-
-
- Closed
-
Activity
Francis Upton
added a comment - @Ted, I don't see how that line broke it. Note that I changed EC2Cloud to look up any running (or pending) instances and check that they are known. Only if they are known will they count against the existing instances. I will test it again though, to try multiple instances with the same AMI. The reason the filter was removed is that we want to see if there is a running instance that's not known, and if so, use that one.
Francis Upton
added a comment - @Ted, I don't see how that line broke it. Note that I changed EC2Cloud to look up any running (or pending) instances and check that they are known. Only if they are known will they count against the existing instances. I will test it again though, to try multiple instances with the same AMI. The reason the filter was removed is that we want to see if there is a running instance that's not known, and if so, use that one. Francis Upton
added a comment - I believe I have now fixed this. Francis Upton
added a comment - Should be released in 1.30 Francis Upton
added a comment - Fixed a problem with this of it not respecting the instance caps for on-demand nodes.
Francis Upton
added a comment - Fixed a problem with this of it not respecting the instance caps for on-demand nodes. 
I think the commit https://github.com/jenkinsci/ec2-plugin/commit/c9d69a5da8c9be094701d4c191ba7b1d06c200c9 breaks plugin, user can not launch multiple instances for same AMI since DescribeInstancesRequest returns all instances instead of stopped instances, and no instances can be launched if there is any running
below line was removed
diFilters.add(new Filter("instance-state-name").withValues(InstanceStateName.Stopped.toString(), InstanceStateName.Stopping.toString()));