Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-35708

EC2 Plugin: Ability to round robin EC2 availability zones

    XMLWordPrintable

Details

    • Improvement
    • Status: Fixed but Unreleased (View Workflow)
    • Minor
    • Resolution: Fixed
    • ec2-plugin
    • None
    • 1.41

    Description

      Is it possible to have instances to spin up across several availability zones (a, b, c, d) to help maximize the availability of creating spot instance in zones that may have greater availability. We have tried to setup different clouds and different instances with different availability zones, but with the same label only to find it takes the first occurrence in the configuration.

      Attachments

        Activity

          brockoli Brock Ackley created issue -
          rtyler R. Tyler Croy made changes -
          Field Original Value New Value
          Workflow JNJira [ 172137 ] JNJira + In-Review [ 184578 ]
          nikk699 Faheem Nadeem added a comment -

          This could be useful for us as well. It is a problem to hardcode to a single availability zone / vpc subnet for spot instances as there could be price spikes and capacity bottlenecks.
          Tried comma separated subnet IDs for VPC. It did not work

          nikk699 Faheem Nadeem added a comment - This could be useful for us as well. It is a problem to hardcode to a single availability zone / vpc subnet for spot instances as there could be price spikes and capacity bottlenecks. Tried comma separated subnet IDs for VPC. It did not work

          But when adding multiple AMIs in different subnets (-> AZs) this should work, right?

          stephenking Steffen Gebert added a comment - But when adding multiple AMIs in different subnets (-> AZs) this should work, right?
          mixologic Ryan Aslett added a comment -

          Ideally the availability zone isnt entered at all when using spot instances, and instead you would select which region you would like your nodes in. Then the ec2 plugin could poll each region's zones for the current spot prices and choose the least expensive option.  Is there a use case for restricting your nodes to a single az, or is that just an artifact of the api call to start an instance?

          mixologic Ryan Aslett added a comment - Ideally the availability zone isnt entered at all when using spot instances, and instead you would select which region you would like your nodes in. Then the ec2 plugin could poll each region's zones for the current spot prices and choose the least expensive option.  Is there a use case for restricting your nodes to a single az, or is that just an artifact of the api call to start an instance?
          ryeates Robert Yeates added a comment -

          EC2Cloud receives a template parameter which pulls from the configured AMI's. It's possible to change

          hudson.plugins.ec2.EC2Cloud#getTemplate(java.lang.String)

          to check which template has the lowest spot price and return that. That approach feels pretty hacky.

          What determines which template description is supplied to  

          public HttpResponse doProvision(@QueryParameter String template)
          

          ?

          ryeates Robert Yeates added a comment - EC2Cloud receives a template parameter which pulls from the configured AMI's. It's possible to change hudson.plugins.ec2.EC2Cloud#getTemplate(java.lang. String ) to check which template has the lowest spot price and return that. That approach feels pretty hacky. What determines which template description is supplied to   public HttpResponse doProvision(@QueryParameter String template) ?
          scardon Daniel Leong added a comment -

          Subnet ID for VPC only takes a string and 1 subnet id?

           

          I tried leaving availability zone empty but use the subnet id method instead and was only allowed to have 1.

          scardon Daniel Leong added a comment - Subnet ID for VPC only takes a string and 1 subnet id?   I tried leaving availability zone empty but use the subnet id method instead and was only allowed to have 1.
          levipl Levi Lovelock added a comment -

          Any progress or workarounds here? This hurts when using instances that have lower availability (i.e. graphics instances)

          levipl Levi Lovelock added a comment - Any progress or workarounds here? This hurts when using instances that have lower availability (i.e. graphics instances)
          davidfluck David Fluck added a comment - - edited

          I'm working on a patch. The goal is to allow the subnet ID text box to take a list of subnet IDs. When a new instance is launched, a round-robin AZ will be chosen from that list. Hopefully this will spread the load out a bit. It would be kind of cool to watch out for ICE errors and then do some kind of backoff for that region, but that seems unnecessary for now.

          davidfluck David Fluck added a comment - - edited I'm working on a patch. The goal is to allow the subnet ID text box to take a list of subnet IDs. When a new instance is launched, a round-robin AZ will be chosen from that list. Hopefully this will spread the load out a bit. It would be kind of cool to watch out for ICE errors and then do some kind of backoff for that region, but that seems unnecessary for now.
          mgaunin Mikael Gaunin added a comment -

          Hi francisu, any chance to review davidfluck's PR (#295) soon?

          mgaunin Mikael Gaunin added a comment - Hi francisu , any chance to review davidfluck 's PR ( #295 ) soon?
          mgaunin Mikael Gaunin added a comment -

          Hi davidfluck, we have a good chance to see your PR #295 reviewed soon since new maintainers work actively on this plugin! \o/

          Did you see jvz 's comment on your PR?

          mgaunin Mikael Gaunin added a comment - Hi davidfluck , we have a good chance to see your PR  #295  reviewed soon since new maintainers work actively on this plugin! \o/ Did you see jvz  's comment on your PR?
          davidfluck David Fluck added a comment -

          mgaunin Sorry for the wait! I did, yeah. I should be able to rebase that shortly. Thank you!

          davidfluck David Fluck added a comment - mgaunin Sorry for the wait! I did, yeah. I should be able to rebase that shortly. Thank you!
          jvz Matt Sicker made changes -
          Released As 1.41
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Fixed but Unreleased [ 10203 ]
          mgaunin Mikael Gaunin added a comment -

          Hi francisu, I guess the status does not reflect the reality. If I'm not wrong, the fix has been delivered into the 1.41 release.

          mgaunin Mikael Gaunin added a comment - Hi francisu , I guess the status does not reflect the reality. If I'm not wrong, the fix has been delivered into the 1.41 release.
          nforbes Nicola Forbes added a comment - - edited

          Hi,

          Is this feature now live?  Are there any instructions on how to use it?  I've tried providing a comma-separated list of subnets in different AZs but receive error
          "com.amazonaws.AmazonClientException: Security groups must all be VPC security groups to work in a VPC context". I can only launch slaves if I specify one subnet and the related AZ.

          nforbes Nicola Forbes added a comment - - edited Hi, Is this feature now live?  Are there any instructions on how to use it?  I've tried providing a comma-separated list of subnets in different AZs but receive error "com.amazonaws.AmazonClientException: Security groups must all be VPC security groups to work in a VPC context". I can only launch slaves if I specify one subnet and the related AZ.
          davidfluck David Fluck added a comment -

          It looks like this change has indeed been released. I've also encountered that error before. How are you listing the subnets? Are you using subnet IDs? Are all of the subnets in the same VPC?

          This is the relevant code:

          private List<String> getEc2SecurityGroups(AmazonEC2 ec2) throws AmazonClientException {
              List<String> groupIds = new ArrayList<String>();
          
              DescribeSecurityGroupsResult groupResult = getSecurityGroupsBy("group-name", securityGroupSet, ec2);
              if (groupResult.getSecurityGroups().size() == 0) {
                  groupResult = getSecurityGroupsBy("group-id", securityGroupSet, ec2);
              }
          
              for (SecurityGroup group : groupResult.getSecurityGroups()) {
                  if (group.getVpcId() != null && !group.getVpcId().isEmpty()) {
                      List<Filter> filters = new ArrayList<Filter>();
                      filters.add(new Filter("vpc-id").withValues(group.getVpcId()));
                      filters.add(new Filter("state").withValues("available"));
                      filters.add(new Filter("subnet-id").withValues(getCurrentSubnetId()));
          
                      DescribeSubnetsRequest subnetReq = new DescribeSubnetsRequest();
                      subnetReq.withFilters(filters);
                      DescribeSubnetsResult subnetResult = ec2.describeSubnets(subnetReq);
          
                      List<Subnet> subnets = subnetResult.getSubnets();
                      if (subnets != null && !subnets.isEmpty()) {
                          groupIds.add(group.getGroupId());
                      }
                  }
              }
          
              if (securityGroupSet.size() != groupIds.size()) {
                  throw new AmazonClientException("Security groups must all be VPC security groups to work in a VPC context");
              }
          
              return groupIds;
          }
          
          davidfluck David Fluck added a comment - It looks like this change has indeed been released. I've also encountered that error before. How are you listing the subnets? Are you using subnet IDs? Are all of the subnets in the same VPC? This is the relevant code: private List< String > getEc2SecurityGroups(AmazonEC2 ec2) throws AmazonClientException { List< String > groupIds = new ArrayList< String >(); DescribeSecurityGroupsResult groupResult = getSecurityGroupsBy( "group-name" , securityGroupSet, ec2); if (groupResult.getSecurityGroups().size() == 0) { groupResult = getSecurityGroupsBy( "group-id" , securityGroupSet, ec2); } for (SecurityGroup group : groupResult.getSecurityGroups()) { if (group.getVpcId() != null && !group.getVpcId().isEmpty()) { List<Filter> filters = new ArrayList<Filter>(); filters.add( new Filter( "vpc-id" ).withValues(group.getVpcId())); filters.add( new Filter( "state" ).withValues( "available" )); filters.add( new Filter( "subnet-id" ).withValues(getCurrentSubnetId())); DescribeSubnetsRequest subnetReq = new DescribeSubnetsRequest(); subnetReq.withFilters(filters); DescribeSubnetsResult subnetResult = ec2.describeSubnets(subnetReq); List<Subnet> subnets = subnetResult.getSubnets(); if (subnets != null && !subnets.isEmpty()) { groupIds.add(group.getGroupId()); } } } if (securityGroupSet.size() != groupIds.size()) { throw new AmazonClientException( "Security groups must all be VPC security groups to work in a VPC context" ); } return groupIds; }
          nforbes Nicola Forbes added a comment -

          Hi David,

          Thanks for replying so quickly. I'm supplying the subnet ids as a comma-separated list (subnet-<string1>, subnet-<string2>, subnet-<string3>). All of the subnets are part of the same VPC but belong to different AZs.

          subnet-<string1> is part of AZ us-east-1a
          subnet-<string2> is part of AZ us-east-1b
          subnet-<string3> is part of AZ us-east-1c

          I am also listing the three AZs as a comma-separated list in the "availability zone" text box ie. us-east-1a, us-east-1b, us-east-1c

          It is possible that I have misunderstood the new feature. Do the subnets have to be part of the same availability zone?

          nforbes Nicola Forbes added a comment - Hi David, Thanks for replying so quickly. I'm supplying the subnet ids as a comma-separated list (subnet-<string1>, subnet-<string2>, subnet-<string3>). All of the subnets are part of the same VPC but belong to different AZs. subnet-<string1> is part of AZ us-east-1a subnet-<string2> is part of AZ us-east-1b subnet-<string3> is part of AZ us-east-1c I am also listing the three AZs as a comma-separated list in the "availability zone" text box ie. us-east-1a, us-east-1b, us-east-1c It is possible that I have misunderstood the new feature. Do the subnets have to be part of the same availability zone?
          equick Edward Quick added a comment -

          Hi Nicola, I could be wrong but a colleague of mine mentioned that you need to provide the AZ list as space delimited (not comma delimited).

          Ed

           

          equick Edward Quick added a comment - Hi Nicola, I could be wrong but a colleague of mine mentioned that you need to provide the AZ list as space delimited (not comma delimited). Ed  
          nforbes Nicola Forbes added a comment -

          Hi Ed,

          Thanks for the suggestion. I've just tried it out but unfortunately still get the same error (various combos tried). I haven't had a chance to read through the source code, but it looks like az can only take one availability zone, not a list, and so all subnets must be part of the same az.

          Nicky

          nforbes Nicola Forbes added a comment - Hi Ed, Thanks for the suggestion. I've just tried it out but unfortunately still get the same error (various combos tried). I haven't had a chance to read through the source code, but it looks like az can only take one availability zone, not a list, and so all subnets must be part of the same az. Nicky

          People

            francisu Francis Upton
            brockoli Brock Ackley
            Votes:
            3 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: