-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins: v2.238
EC2 Plugin: 1.50.3
For the last few months we've seen many cases of nodes failing to provision, and the EC2 plugin continues to try forever without ever failing. An example set of logs:
May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud INFO: Launching instance: i-02dd1ad432030144c May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud INFO: bootstrap() May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud INFO: Getting keypair... May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud INFO: Using private key jenkins-master-syncs (SHA-1 fingerprint 56:38:ff:50:89:30:ac:23:60:22:63:d1:66:77:04:71) May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud INFO: Authenticating as ubuntu May 29, 2020 6:36:28 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:36:38 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:36:38 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:36:43 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:36:53 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:36:53 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:36:58 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:37:08 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:37:08 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:37:13 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:37:23 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:37:23 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:37:28 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:37:38 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:37:38 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:37:43 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:37:53 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:37:53 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:37:58 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:38:08 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:38:08 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:38:13 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:38:23 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:38:23 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:38:28 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:38:38 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:38:38 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:38:43 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:38:53 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:38:53 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:38:59 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:39:09 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:39:09 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:39:14 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:39:24 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:39:24 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:39:29 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:39:39 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. May 29, 2020 6:39:39 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5. May 29, 2020 6:39:44 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.0.25.187 on port 22, with timeout 10000. May 29, 2020 6:39:54 PM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. [snip] NFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. Jun 01, 2020 4:14:18 PM hudson.plugins.ec2.EC2Cloud INFO: Waiting for SSH to come up. Sleeping 5.Jun 01, 2020 4:14:24 PM hudson.plugins.ec2.EC2Cloud
The node itself failed (with 1/2 status checks passed), but Jenkins will never, ever, stop trying to connect no matter how long this takes. I have to sweep through and clean up the cluster every few days (we launch thousands of nodes a day).
