Details
-
Type:
Bug
-
Status: Open (View Workflow)
-
Priority:
Major
-
Resolution: Unresolved
-
Component/s: ec2-plugin
-
Labels:None
-
Environment:Jenkins: v2.238
EC2 Plugin: 1.50.3
-
Similar Issues:
Description
For the last few months we've seen many cases of nodes failing to provision, and the EC2 plugin continues to try forever without ever failing. An example set of logs:
May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud INFO: Launching instance: i-02dd1ad432030144c May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud INFO: bootstrap() May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud INFO: Getting keypair... May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud INFO: Using private key jenkins-master-syncs (SHA-1 fingerprint 56:38:ff:50:89:30:ac:23:60:22:63:d1:66:77:04:71) May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud INFO: Authenticating as ubuntu May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:36:38 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:36:38 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:36:43 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:36:53 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:36:53 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:36:58 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:37:08 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:37:08 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:37:13 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:37:23 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:37:23 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:37:28 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:37:38 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:37:38 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:37:43 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:37:53 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:37:53 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:37:58 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:38:08 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:38:08 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:38:13 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:38:23 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:38:23 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:38:28 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:38:38 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:38:38 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:38:43 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:38:53 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:38:53 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:38:59 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:39:09 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:39:09 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:39:14 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:39:24 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:39:24 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:39:29 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:39:39 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:39:39 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:39:44 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:39:54 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. [snip] NFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. Jun 01, 2020 4:14:18 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5.Jun 01, 2020 4:14:24 PM hudson.plugins.ec2.EC2Cloud
The node itself failed (with 1/2 status checks passed), but Jenkins will never, ever, stop trying to connect no matter how long this takes. I have to sweep through and clean up the cluster every few days (we launch thousands of nodes a day).
