Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65052

Excessive Time to Provision EC2 Machines

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: ec2-plugin
    • Labels:
      None
    • Environment:
      Jenkins v2.263.4
      EC2 plugin 1.56
    • Similar Issues:

      Description

      The EC2 plugin occasionally takes an excessive amount of time to provision new machines. The example I have shows that it took 15 minutes from start to finish to provision and connect to 88 new machines, comprised of 11 minutes to create the PlannedNode instances and another 4 minutes to call start on each of the new Node instances being added into Jenkins.

      The 4-minute period appears now to have been caused by a second cloud plugin and that issue has been resolved, the 11-minute period is still outstanding. Moreover, this period happens entirely in the NodeProvisioner.update thread, so it occurs with the provisioning lock being held and does not allow our second cloud plugin to provision machines during this time. We do not see this all the time, but I suspect it comes as more of an issue during heavy periods and we may be running into request throttling from the AWS side.

      I believe this is due to a line in the EC2AbstractSlave constructor, which fetches instance data from EC2 on creation of the Slave instance. Reading the code, this data is designed to be fetched when required, so there's no reason to fetch it in the EC2AbstractSlave constructor and block the provisioning thread. I've opened PR #574 to make this change.

      Please find filtered logs (based on thread ID) at https://gist.github.com/mtughan/cbad70db249b8eedb485f9253acb135b. Sensitive company information has been stripped from the log as well. If there are any other potential causes of the slowness that we can work to resolve, I am open for discussion.

        Attachments

          Activity

          mtughan Michael Tughan created issue -
          mtughan Michael Tughan made changes -
          Field Original Value New Value
          Released As https://github.com/jenkinsci/ec2-plugin/releases/tag/ec2-1.57
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]

            People

            Assignee:
            thoulen FABRIZIO MANFREDI
            Reporter:
            mtughan Michael Tughan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: