Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-25832

Launch multiple slaves in parallel for jobs with same node label

    XMLWordPrintable

Details

    • New Feature
    • Status: Resolved (View Workflow)
    • Critical
    • Resolution: Fixed
    • ec2-plugin
    • None

    Description

      I use build flow plugin to kick off 200+ jobs. Each of these jobs has the same label. During this scenario, 200+ jobs are queued up. The ec2 plugin seems to process one job at time and this is a painfully slow process.

      It seems that the AWS JAVA SDK allows for launching multiple ec2 instances. Can the ec2 plugin detect the number of jobs with the same label in the queue, and then launch that many ec2 instances in parallel?

      http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/ec2/model/RunInstancesRequest.html#RunInstancesRequest%28java.lang.String,%20java.lang.Integer,%20java.lang.Integer%29

       private EC2AbstractSlave provisionOndemand(TaskListener listener) throws AmazonClientException, IOException {
PrintStream logger = listener.getLogger();
AmazonEC2 ec2 = getParent().connect();
try {
String msg = "Launching " + ami + " for template " + description;
logger.println(msg);
LOGGER.info(msg);
KeyPair keyPair = getKeyPair(ec2);
RunInstancesRequest riRequest = new RunInstancesRequest(ami, 1, 1);

      Attachments

        Activity

          Ascending order - Click to sort in descending order
          View 12 older comments
          pashastryhelski Pavel Stryhelski added a comment -

          Hi, we are facing the same issue. 6 jobs in parallel give us the latest job to be started in ~50 mins after the main build is run.

          pashastryhelski Pavel Stryhelski added a comment - Hi, we are facing the same issue. 6 jobs in parallel give us the latest job to be started in ~50 mins after the main build is run.
          luispiedra Luis Piedra-Márquez added a comment -

          Hi,

          Plugin is prepared to launch instances until the excess workload is satisfied, but it seems this behavior was broken at some point.

          I have filled a pull request with a fix:

          https://github.com/jenkinsci/ec2-plugin/pull/241

          It's working fine for me.

          luispiedra Luis Piedra-Márquez added a comment - Hi, Plugin is prepared to launch instances until the excess workload is satisfied, but it seems this behavior was broken at some point. I have filled a pull request with a fix: https://github.com/jenkinsci/ec2-plugin/pull/241 It's working fine for me.
          gtunon Guiomar Tuñón added a comment -

          Hi, there is any news on it? 

          We have a critical testing process that consumes quite a lot of time and parallel slaves are mandatory for the process.

          gtunon Guiomar Tuñón added a comment - Hi, there is any news on it?  We have a critical testing process that consumes quite a lot of time and parallel slaves are mandatory for the process.
          jimilian Alexander A added a comment -

          gtunon, see https://github.com/jenkinsci/ec2-plugin/pull/252

          jimilian Alexander A added a comment - gtunon , see https://github.com/jenkinsci/ec2-plugin/pull/252
          raihaan Raihaan Shouhell added a comment -

          This has already been fixed in 1.40

          raihaan Raihaan Shouhell added a comment - This has already been fixed in 1.40

          People

            francisu Francis Upton
            arazauci arazauci
            Votes:
            16 Vote for this issue
            Watchers:
            27 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: