-
Bug
-
Resolution: Not A Defect
-
Major
-
Jenkins: 2.241
EC2 Plugin 1.50.3
I recently upgraded from 1.49.1. Now Jenkins cannot use any EC2 workers. I am using the latest standard AWS linux2 AMI.
The node log says, over and over :
INFO: Waiting for SSH to come up. Sleeping 5.
Jun 19, 2020 10:54:07 AM hudson.plugins.ec2.EC2Cloud
INFO: Connecting to 10.66.2.89 on port 22, with timeout 10000.
Jun 19, 2020 10:54:07 AM hudson.plugins.ec2.EC2Cloud
INFO: The instance EC2 (Jenkins) - Default Slave (i-04821a9ba3e3e5cf3) has a blank console. Maybe the console is yet not available. If enough time has passed, consider changing the key verification strategy or the AMI used by one printing out the host key in the instance console
Jun 19, 2020 10:54:07 AM hudson.plugins.ec2.EC2Cloud
INFO: The instance console is blank. Cannot check the key. The connection to EC2 (Jenkins) - Default Slave (i-04821a9ba3e3e5cf3) is not allowed
Jun 19, 2020 10:54:07 AM hudson.plugins.ec2.EC2Cloud
INFO: Failed to connect via ssh: There was a problem while connecting to 10.66.2.89:22
Jun 19, 2020 10:54:07 AM hudson.plugins.ec2.EC2Cloud
And yet, I can query the instance console log from the Jenkins master without any problems and see the ssh key sections (obfuscated here) :
aws ec2 get-console-output --instance-id i-04821a9ba3e3e5cf3aws ec2 get-console-output --instance-id i-04821a9ba3e3e5cf3{ "InstanceId": "i-04821a9ba3e3e5cf3", "Output": " 102/117 \r\n Installing : libcom_err-devel-1.43.5-2.43.amzn1.x86_64 103/117 \r\n Installing : libverto-devel-0.2.5-4.9.amzn1.x86_64 104/117 \r\n Installing : libsepol-devel-2.1.7-3.12.amzn1.x86_64 105/117 \r\n Installing : libselinux-devel-2.1.10-3.22.amzn1.x86_64 106/117 \r\n Installing : krb5-devel-1.15.1-46.48.amzn1.x86_64 107/117 \r\n Installing : 1:openssl-devel-1.0.2k-16.151.amzn1.x86_64 108/117 \r\n Installing : nodejs-devel-0.10.48-3.el6.x86_64 109/117 \r\n Installing : node-gyp-0.10.6-2.el6.noarch 110/117 \r\n Installing : npm-1.3.6-5.el6.noarch 111/117 \r\n Installing : gcc-c++-4.8.5-1.22.amzn1.noarch 112/117 \r\n Installing : jq-1.5-1.2.amzn1.x86_64 113/117 \r\n Cleanup : 1:openssl-1.0.2k-13.111.amzn1.x86_64 ........
{{ Jenkins2 slave \r\nec2: \r\nec2: #############################################################\r\nec2: ----BEGIN SSH HOST KEY FINGERPRINTS---\r\nec2: 1024 SHA256:+I3roaPC4ojmofR9/4a3oLGQF2N6gdbQ+HWP2J3ANKc root@ip-10-66-2-89 (DSA)\r\nec2: 256 SHA256:BCenkdcTlXQww4Rao/f+6+VxxxAp5OpB1g root@ip-10-66-2-89 (ECDSA)\r\nec2: 256 SHA256:YikvW/GI6ST7IH9YOxxxxu21nkVbvrb6SNSE no comment (ED25519)\r\nec2: 2048 SHA256:IhlV9fmdi3fWP/YOdlVmHtxxxxwNrQ6xuu9eM root@ip-10-66-2-89 (RSA)\r\nec2: -----END SSH HOST KEY FINGERPRINTS---\r\nec2: #############################################################\r\n---BEGIN SSH HOST KEY KEYS---\r\necdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHxxA6dZ0uFCxer5B+LL4BbucJXd7Us2Zet/jHxHdrTrSR9i1n475IsnNDk0+HKMIdqnSMpM8Q5W+yMnY= root@ip-10-66-2-89\r\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBnyi6P5k5EJSMROnadSBRclaqcA6cvuPJVJGLEDJ+xF \r\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEsJ3k9xzI1WkaxxxAms7UXStJjfDQus2xxcDe4DWl6ruYVjYxFXMJT3yLdhWyyGj02+WQjb61eFCoECcEMMEM+38ovYts3zkF8lXsc3eMVazmuAjJQgFQhmqWdWkn2iR/0Vbodb3ZMjWJZMYw9dTe0QFxxxXx9Lzi0RS0Yy2bxJITdjskj+aLDLIPwLHjNidHtBZKnB9H58t06aqdRusKw4lJKl5u7TgMQy4Ywrof2Dx46G8/RsvatVVesGtZ+JaB6AmXfLj/OW9OQ3aK/Ls/WAZpFTvslaxxXtXoYL4qh root@ip-10-66-2-89\r\n---END SSH HOST KEY KEYS----\r\nCloud-init v. 0.7.6 finished at Fri, 19 Jun 2020 10:54:34 +0000. Datasource DataSourceEc2. Up 146.18 seconds\r\n\r\r\nAmazon Linux AMI release 2018.03\r\nKernel 4.14.77-70.59.amzn1.x86_64 on an x86_64\r\n\r\n", "Timestamp": "2020-06-19T10:58:20.000Z"}}}
I have changed "Host Key Verification Strategy" from "Check-new-soft" to "off" and it works again. But if it thinks the console log is empty I cannot use any of the safer options for validating the key. (And have to change all my node definitions...)
Is the plugin correctly using the instance role to get credentials to query the log? (guess?!)
[JENKINS-62724] EC2 plugin fails to get console output
Sadly it's true. This fact is documented: https://github.com/jenkinsci/ec2-plugin/#check-new-hard
The launch timeout should be long enough to allow the plugin to check the instance console. With this strategy, the plugin waits for the console to be available, which can take a few minutes. The Launch Timeout in seconds field should have a number to allow that, for example 600 (10 minutes). By default there is no timeout, so it's safe.
That said, this waiting period only happens once, to retrieve the host key. Once the plugin gets the key, stores it to accept future connections to the instance. You can use Accept New which accepts blindly the first one Jenkins connects to the instance. It's not so safe, but at least prevents future MitM attacks.
My bad. Or AWS's.
The console log takes ages to populate. Over 5 minutes. Nearer 10. So I either have to change the launch time out or the security mode. (or maybe send a big chunk of output to the syslog to encourage it to flush the buffer more promptly)