Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65052

Excessive Time to Provision EC2 Machines

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: ec2-plugin
    • Labels:
      None
    • Environment:
      Jenkins v2.263.4
      EC2 plugin 1.56
    • Similar Issues:

      Description

      The EC2 plugin occasionally takes an excessive amount of time to provision new machines. The example I have shows that it took 15 minutes from start to finish to provision and connect to 88 new machines, comprised of 11 minutes to create the PlannedNode instances and another 4 minutes to call start on each of the new Node instances being added into Jenkins.

      The 4-minute period appears now to have been caused by a second cloud plugin and that issue has been resolved, the 11-minute period is still outstanding. Moreover, this period happens entirely in the NodeProvisioner.update thread, so it occurs with the provisioning lock being held and does not allow our second cloud plugin to provision machines during this time. We do not see this all the time, but I suspect it comes as more of an issue during heavy periods and we may be running into request throttling from the AWS side.

      I believe this is due to a line in the EC2AbstractSlave constructor, which fetches instance data from EC2 on creation of the Slave instance. Reading the code, this data is designed to be fetched when required, so there's no reason to fetch it in the EC2AbstractSlave constructor and block the provisioning thread. I've opened PR #574 to make this change.

      Please find filtered logs (based on thread ID) at https://gist.github.com/mtughan/cbad70db249b8eedb485f9253acb135b. Sensitive company information has been stripped from the log as well. If there are any other potential causes of the slowness that we can work to resolve, I am open for discussion.

        Attachments

          Activity

          Hide
          mtughan Michael Tughan added a comment -

          I had opened https://github.com/jenkinsci/ec2-plugin/pull/574 to try to resolve the issue, which was merged into master over the weekend. I will test this again in the next EC2 plugin release and we'll see if this reoccurs, as it's not constant.

          Show
          mtughan Michael Tughan added a comment - I had opened https://github.com/jenkinsci/ec2-plugin/pull/574 to try to resolve the issue, which was merged into master over the weekend. I will test this again in the next EC2 plugin release and we'll see if this reoccurs, as it's not constant.
          Hide
          mtughan Michael Tughan added a comment -

          This should be fixed in EC2 release 1.57 now.

          Show
          mtughan Michael Tughan added a comment - This should be fixed in EC2 release 1.57 now.

            People

            Assignee:
            thoulen FABRIZIO MANFREDI
            Reporter:
            mtughan Michael Tughan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: