EC2 Plugin: Nodes failing to provision and then never being deleted

This issue is archived. You can view it, but you can't modify it. Learn more

XMLWordPrintable

      For the last few months we've seen many cases of nodes failing to provision, and the EC2 plugin continues to try forever without ever failing.  An example set of logs:

       

      May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud
INFO: Launching instance: i-02dd1ad432030144c
May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud
INFO: bootstrap()
May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud
INFO: Getting keypair...
May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud
INFO: Using private key jenkins-master-syncs (SHA-1 fingerprint 56:38:ff:50:89:30:ac:23:60:22:63:d1:66:77:04:71)
May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud
INFO: Authenticating as ubuntu
May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud
INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
May 29, 2020 6:36:38 PM hudson.plugins.ec2.EC2Cloud
INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
May 29, 2020 6:36:38 PM hudson.plugins.ec2.EC2Cloud
INFO: Waiting for SSH to come up. Sleeping 5.
May 29, 2020 6:36:43 PM hudson.plugins.ec2.EC2Cloud
INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
May 29, 2020 6:36:53 PM hudson.plugins.ec2.EC2Cloud
INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
May 29, 2020 6:36:53 PM hudson.plugins.ec2.EC2Cloud
INFO: Waiting for SSH to come up. Sleeping 5.
May 29, 2020 6:36:58 PM hudson.plugins.ec2.EC2Cloud
INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
May 29, 2020 6:37:08 PM hudson.plugins.ec2.EC2Cloud
INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
May 29, 2020 6:37:08 PM hudson.plugins.ec2.EC2Cloud
INFO: Waiting for SSH to come up. Sleeping 5.
May 29, 2020 6:37:13 PM hudson.plugins.ec2.EC2Cloud
INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
May 29, 2020 6:37:23 PM hudson.plugins.ec2.EC2Cloud
INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
May 29, 2020 6:37:23 PM hudson.plugins.ec2.EC2Cloud
INFO: Waiting for SSH to come up. Sleeping 5.
May 29, 2020 6:37:28 PM hudson.plugins.ec2.EC2Cloud
INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
May 29, 2020 6:37:38 PM hudson.plugins.ec2.EC2Cloud
INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
May 29, 2020 6:37:38 PM hudson.plugins.ec2.EC2Cloud
INFO: Waiting for SSH to come up. Sleeping 5.
May 29, 2020 6:37:43 PM hudson.plugins.ec2.EC2Cloud
INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
May 29, 2020 6:37:53 PM hudson.plugins.ec2.EC2Cloud
INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
May 29, 2020 6:37:53 PM hudson.plugins.ec2.EC2Cloud
INFO: Waiting for SSH to come up. Sleeping 5.
May 29, 2020 6:37:58 PM hudson.plugins.ec2.EC2Cloud
INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
May 29, 2020 6:38:08 PM hudson.plugins.ec2.EC2Cloud
INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
May 29, 2020 6:38:08 PM hudson.plugins.ec2.EC2Cloud
INFO: Waiting for SSH to come up. Sleeping 5.
May 29, 2020 6:38:13 PM hudson.plugins.ec2.EC2Cloud
INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
May 29, 2020 6:38:23 PM hudson.plugins.ec2.EC2Cloud
INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
May 29, 2020 6:38:23 PM hudson.plugins.ec2.EC2Cloud
INFO: Waiting for SSH to come up. Sleeping 5.
May 29, 2020 6:38:28 PM hudson.plugins.ec2.EC2Cloud
INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
May 29, 2020 6:38:38 PM hudson.plugins.ec2.EC2Cloud
INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
May 29, 2020 6:38:38 PM hudson.plugins.ec2.EC2Cloud
INFO: Waiting for SSH to come up. Sleeping 5.
May 29, 2020 6:38:43 PM hudson.plugins.ec2.EC2Cloud
INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
May 29, 2020 6:38:53 PM hudson.plugins.ec2.EC2Cloud
INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
May 29, 2020 6:38:53 PM hudson.plugins.ec2.EC2Cloud
INFO: Waiting for SSH to come up. Sleeping 5.
May 29, 2020 6:38:59 PM hudson.plugins.ec2.EC2Cloud
INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
May 29, 2020 6:39:09 PM hudson.plugins.ec2.EC2Cloud
INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
May 29, 2020 6:39:09 PM hudson.plugins.ec2.EC2Cloud
INFO: Waiting for SSH to come up. Sleeping 5.
May 29, 2020 6:39:14 PM hudson.plugins.ec2.EC2Cloud
INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
May 29, 2020 6:39:24 PM hudson.plugins.ec2.EC2Cloud
INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
May 29, 2020 6:39:24 PM hudson.plugins.ec2.EC2Cloud
INFO: Waiting for SSH to come up. Sleeping 5.
May 29, 2020 6:39:29 PM hudson.plugins.ec2.EC2Cloud
INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
May 29, 2020 6:39:39 PM hudson.plugins.ec2.EC2Cloud
INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
May 29, 2020 6:39:39 PM hudson.plugins.ec2.EC2Cloud
INFO: Waiting for SSH to come up. Sleeping 5.
May 29, 2020 6:39:44 PM hudson.plugins.ec2.EC2Cloud
INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000.
May 29, 2020 6:39:54 PM hudson.plugins.ec2.EC2Cloud
INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
[snip]
NFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
Jun 01, 2020 4:14:18 PM hudson.plugins.ec2.EC2Cloud
INFO: Waiting for SSH to come up. Sleeping 5.Jun 01, 2020 4:14:24 PM hudson.plugins.ec2.EC2Cloud

      The node itself failed (with 1/2 status checks passed), but Jenkins will never, ever, stop trying to connect no matter how long this takes.  I have to sweep through and clean up the cluster every few days (we launch thousands of nodes a day).

            Assignee:
            FABRIZIO MANFREDI
            Reporter:
            Ashley Penney
            Archiver:
            Jenkins Service Account

              Created:
              Updated:
              Archived: