Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62724

EC2 plugin fails to get console output

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not A Defect
    • Icon: Major Major
    • ec2-plugin
    • Jenkins: 2.241
      EC2 Plugin 1.50.3

      I recently upgraded from 1.49.1. Now Jenkins cannot use any EC2 workers. I am using the latest standard AWS linux2 AMI.

      The node log says, over and over :

       

      INFO: Waiting for SSH to come up. Sleeping 5.
      Jun 19, 2020 10:54:07 AM hudson.plugins.ec2.EC2Cloud
      INFO: Connecting to 10.66.2.89 on port 22, with timeout 10000.
      Jun 19, 2020 10:54:07 AM hudson.plugins.ec2.EC2Cloud
      INFO: The instance EC2 (Jenkins) - Default Slave (i-04821a9ba3e3e5cf3) has a blank console. Maybe the console is yet not available. If enough time has passed, consider changing the key verification strategy or the AMI used by one printing out the host key in the instance console
      Jun 19, 2020 10:54:07 AM hudson.plugins.ec2.EC2Cloud
      INFO: The instance console is blank. Cannot check the key. The connection to EC2 (Jenkins) - Default Slave (i-04821a9ba3e3e5cf3) is not allowed
      Jun 19, 2020 10:54:07 AM hudson.plugins.ec2.EC2Cloud
      INFO: Failed to connect via ssh: There was a problem while connecting to 10.66.2.89:22
      Jun 19, 2020 10:54:07 AM hudson.plugins.ec2.EC2Cloud

       

      And yet, I can query the instance console log from the Jenkins master without any problems and see the ssh key sections (obfuscated here) :

      aws ec2 get-console-output --instance-id i-04821a9ba3e3e5cf3aws ec2 get-console-output --instance-id i-04821a9ba3e3e5cf3{    "InstanceId": "i-04821a9ba3e3e5cf3",     "Output": "                 102/117 \r\n  Installing : libcom_err-devel-1.43.5-2.43.amzn1.x86_64                103/117 \r\n  Installing : libverto-devel-0.2.5-4.9.amzn1.x86_64                    104/117 \r\n  Installing : libsepol-devel-2.1.7-3.12.amzn1.x86_64                   105/117 \r\n  Installing : libselinux-devel-2.1.10-3.22.amzn1.x86_64                106/117 \r\n  Installing : krb5-devel-1.15.1-46.48.amzn1.x86_64                     107/117 \r\n  Installing : 1:openssl-devel-1.0.2k-16.151.amzn1.x86_64               108/117 \r\n  Installing : nodejs-devel-0.10.48-3.el6.x86_64                        109/117 \r\n  Installing : node-gyp-0.10.6-2.el6.noarch                             110/117 \r\n  Installing : npm-1.3.6-5.el6.noarch                                   111/117 \r\n  Installing : gcc-c++-4.8.5-1.22.amzn1.noarch                          112/117 \r\n  Installing : jq-1.5-1.2.amzn1.x86_64                                  113/117 \r\n  Cleanup    : 1:openssl-1.0.2k-13.111.amzn1.x86_64                   ........
      {{ Jenkins2 slave \r\nec2: \r\nec2: #############################################################\r\nec2: ----BEGIN SSH HOST KEY FINGERPRINTS---\r\nec2: 1024 SHA256:+I3roaPC4ojmofR9/4a3oLGQF2N6gdbQ+HWP2J3ANKc root@ip-10-66-2-89 (DSA)\r\nec2: 256 SHA256:BCenkdcTlXQww4Rao/f+6+VxxxAp5OpB1g root@ip-10-66-2-89 (ECDSA)\r\nec2: 256 SHA256:YikvW/GI6ST7IH9YOxxxxu21nkVbvrb6SNSE no comment (ED25519)\r\nec2: 2048 SHA256:IhlV9fmdi3fWP/YOdlVmHtxxxxwNrQ6xuu9eM root@ip-10-66-2-89 (RSA)\r\nec2: -----END SSH HOST KEY FINGERPRINTS---\r\nec2: #############################################################\r\n---BEGIN SSH HOST KEY KEYS---\r\necdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHxxA6dZ0uFCxer5B+LL4BbucJXd7Us2Zet/jHxHdrTrSR9i1n475IsnNDk0+HKMIdqnSMpM8Q5W+yMnY= root@ip-10-66-2-89\r\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBnyi6P5k5EJSMROnadSBRclaqcA6cvuPJVJGLEDJ+xF \r\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEsJ3k9xzI1WkaxxxAms7UXStJjfDQus2xxcDe4DWl6ruYVjYxFXMJT3yLdhWyyGj02+WQjb61eFCoECcEMMEM+38ovYts3zkF8lXsc3eMVazmuAjJQgFQhmqWdWkn2iR/0Vbodb3ZMjWJZMYw9dTe0QFxxxXx9Lzi0RS0Yy2bxJITdjskj+aLDLIPwLHjNidHtBZKnB9H58t06aqdRusKw4lJKl5u7TgMQy4Ywrof2Dx46G8/RsvatVVesGtZ+JaB6AmXfLj/OW9OQ3aK/Ls/WAZpFTvslaxxXtXoYL4qh root@ip-10-66-2-89\r\n---END SSH HOST KEY KEYS----\r\nCloud-init v. 0.7.6 finished at Fri, 19 Jun 2020 10:54:34 +0000. Datasource DataSourceEc2.  Up 146.18 seconds\r\n\r\r\nAmazon Linux AMI release 2018.03\r\nKernel 4.14.77-70.59.amzn1.x86_64 on an x86_64\r\n\r\n",     "Timestamp": "2020-06-19T10:58:20.000Z"}}}

       

       

      I have changed  "Host Key Verification Strategy" from "Check-new-soft" to "off" and it works again. But if it thinks the console log is empty I cannot use any of the safer options for validating the key. (And have to change all my node definitions...)

      Is the plugin correctly using the instance role to get credentials to query the log? (guess?!)

            thoulen FABRIZIO MANFREDI
            max_allan max allan
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: