Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62529

EC2 Plugin: Nodes failing to provision and then never being deleted

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: ec2-plugin
    • Labels:
      None
    • Environment:
      Jenkins: v2.238
      EC2 Plugin: 1.50.3
    • Similar Issues:

      Description

      For the last few months we've seen many cases of nodes failing to provision, and the EC2 plugin continues to try forever without ever failing.  An example set of logs:

       

      May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud
      INFO: Launching instance: i-02dd1ad432030144c
      May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud
      INFO: bootstrap()
      May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud
      INFO: Getting keypair...
      May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud
      INFO: Using private key jenkins-master-syncs (SHA-1 fingerprint 56:38:ff:50:89:30:ac:23:60:22:63:d1:66:77:04:71)
      May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud
      INFO: Authenticating as ubuntu
      May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud
      INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
      May 29, 2020 6:36:38 PM hudson.plugins.ec2.EC2Cloud
      INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
      May 29, 2020 6:36:38 PM hudson.plugins.ec2.EC2Cloud
      INFO: Waiting for SSH to come up. Sleeping 5.
      May 29, 2020 6:36:43 PM hudson.plugins.ec2.EC2Cloud
      INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
      May 29, 2020 6:36:53 PM hudson.plugins.ec2.EC2Cloud
      INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
      May 29, 2020 6:36:53 PM hudson.plugins.ec2.EC2Cloud
      INFO: Waiting for SSH to come up. Sleeping 5.
      May 29, 2020 6:36:58 PM hudson.plugins.ec2.EC2Cloud
      INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
      May 29, 2020 6:37:08 PM hudson.plugins.ec2.EC2Cloud
      INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
      May 29, 2020 6:37:08 PM hudson.plugins.ec2.EC2Cloud
      INFO: Waiting for SSH to come up. Sleeping 5.
      May 29, 2020 6:37:13 PM hudson.plugins.ec2.EC2Cloud
      INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
      May 29, 2020 6:37:23 PM hudson.plugins.ec2.EC2Cloud
      INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
      May 29, 2020 6:37:23 PM hudson.plugins.ec2.EC2Cloud
      INFO: Waiting for SSH to come up. Sleeping 5.
      May 29, 2020 6:37:28 PM hudson.plugins.ec2.EC2Cloud
      INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
      May 29, 2020 6:37:38 PM hudson.plugins.ec2.EC2Cloud
      INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
      May 29, 2020 6:37:38 PM hudson.plugins.ec2.EC2Cloud
      INFO: Waiting for SSH to come up. Sleeping 5.
      May 29, 2020 6:37:43 PM hudson.plugins.ec2.EC2Cloud
      INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
      May 29, 2020 6:37:53 PM hudson.plugins.ec2.EC2Cloud
      INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
      May 29, 2020 6:37:53 PM hudson.plugins.ec2.EC2Cloud
      INFO: Waiting for SSH to come up. Sleeping 5.
      May 29, 2020 6:37:58 PM hudson.plugins.ec2.EC2Cloud
      INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
      May 29, 2020 6:38:08 PM hudson.plugins.ec2.EC2Cloud
      INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
      May 29, 2020 6:38:08 PM hudson.plugins.ec2.EC2Cloud
      INFO: Waiting for SSH to come up. Sleeping 5.
      May 29, 2020 6:38:13 PM hudson.plugins.ec2.EC2Cloud
      INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
      May 29, 2020 6:38:23 PM hudson.plugins.ec2.EC2Cloud
      INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
      May 29, 2020 6:38:23 PM hudson.plugins.ec2.EC2Cloud
      INFO: Waiting for SSH to come up. Sleeping 5.
      May 29, 2020 6:38:28 PM hudson.plugins.ec2.EC2Cloud
      INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
      May 29, 2020 6:38:38 PM hudson.plugins.ec2.EC2Cloud
      INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
      May 29, 2020 6:38:38 PM hudson.plugins.ec2.EC2Cloud
      INFO: Waiting for SSH to come up. Sleeping 5.
      May 29, 2020 6:38:43 PM hudson.plugins.ec2.EC2Cloud
      INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
      May 29, 2020 6:38:53 PM hudson.plugins.ec2.EC2Cloud
      INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
      May 29, 2020 6:38:53 PM hudson.plugins.ec2.EC2Cloud
      INFO: Waiting for SSH to come up. Sleeping 5.
      May 29, 2020 6:38:59 PM hudson.plugins.ec2.EC2Cloud
      INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
      May 29, 2020 6:39:09 PM hudson.plugins.ec2.EC2Cloud
      INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
      May 29, 2020 6:39:09 PM hudson.plugins.ec2.EC2Cloud
      INFO: Waiting for SSH to come up. Sleeping 5.
      May 29, 2020 6:39:14 PM hudson.plugins.ec2.EC2Cloud
      INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
      May 29, 2020 6:39:24 PM hudson.plugins.ec2.EC2Cloud
      INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
      May 29, 2020 6:39:24 PM hudson.plugins.ec2.EC2Cloud
      INFO: Waiting for SSH to come up. Sleeping 5.
      May 29, 2020 6:39:29 PM hudson.plugins.ec2.EC2Cloud
      INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
      May 29, 2020 6:39:39 PM hudson.plugins.ec2.EC2Cloud
      INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
      May 29, 2020 6:39:39 PM hudson.plugins.ec2.EC2Cloud
      INFO: Waiting for SSH to come up. Sleeping 5.
      May 29, 2020 6:39:44 PM hudson.plugins.ec2.EC2Cloud
      INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
      May 29, 2020 6:39:54 PM hudson.plugins.ec2.EC2Cloud
      INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
      [snip]
      NFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
      Jun 01, 2020 4:14:18 PM hudson.plugins.ec2.EC2Cloud
      INFO: Waiting for SSH to come up. Sleeping 5.Jun 01, 2020 4:14:24 PM hudson.plugins.ec2.EC2Cloud
      

      The node itself failed (with 1/2 status checks passed), but Jenkins will never, ever, stop trying to connect no matter how long this takes.  I have to sweep through and clean up the cluster every few days (we launch thousands of nodes a day).

        Attachments

          Activity

          There are no comments yet on this issue.

            People

            Assignee:
            thoulen FABRIZIO MANFREDI
            Reporter:
            apenney Ashley Penney
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated: