-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
Operating System: AWS EC2, Amazon Linux and Amazon Linux 2
JRE/JDK: 8
Jenkins version: 2.235.1 (LTS)
Amazon EC2 plugin version: 1.50.3
Amazon Web Services SDK version: 1.11.799
We have been using the Amazon EC2 plugin for quite some time. In the past it would launch EC2 slaves very quickly. Now, it is taking a very long time. It appears to take over 7 minutes for each slave.
We have lots of logs that look like:
INFO: The instance EC2 (us-east-1) - Java 11 Slave (i-xyz) has a blank console. Maybe the console is yet not available. If enough time has passed, consider changing the key verification strategy or the AMI used by one printing out the host key in the instance console
This seems to take long for both check-new-hard and check-new-soft slaves.
I'm not sure how the EC2 plugin checks this. But, I did open the EC2 web console and performed "Actions->Instance Settings->Get System Log"
I see lines like this a full 5 minutes before the EC2 plugin can verify the keys and connect:
{{<14>Jul 2 21:09:05 ec2: }}
<14>Jul 2 21:09:05 ec2: #############################################################
<14>Jul 2 21:09:05 ec2: ----BEGIN SSH HOST KEY FINGERPRINTS----
<14>Jul 2 21:09:05 ec2: 256 SHA256:XYZ no comment (ECDSA)
<14>Jul 2 21:09:05 ec2: 256 SHA256:XYZ no comment (ED25519)
<14>Jul 2 21:09:05 ec2: 2048 SHA256:XYZ no comment (RSA)
<14>Jul 2 21:09:05 ec2: ----END SSH HOST KEY FINGERPRINTS----
<14>Jul 2 21:09:05 ec2: #############################################################
----BEGIN SSH HOST KEY KEYS----
ecdsa-sha2-nistp256 XYZ
ssh-ed25519 XYZ
ssh-rsa XYZ
To be quite clear, the slaves do eventually connect. That is quite consistent. The issue here is the length of time has increased greatly.
For me it looks like the time between the initial "Looking for existing instances with describe-instance" and the actual launch takes around 2 minutes.
The Looking for existing instances with describe-instance
The next line: