Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62641

Ec2 plugin: Cannot create windows node with ec2 1.50.3

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • ec2-plugin
    • None
    • Jenkins 2.222.4
      ec2 plugin: 1.50.3

      I've tried to use the latest (1.50.3) version of the ec2 plugin and encountered an issue when creating a windows node.

      When trying to start the ec2 instance, it boots and then it got stuck. I didn't manage to find why. Here the only log I have:

       

      EC2 (elastic-cloud) - windows-node (i-1234) booted at 1591718074000
      Connecting to (xx.xx.xx.xx) with WinRM as Administrator

      Enabling the logging gave me no information. I was hoping having some logs enabling logging for hudson.plugins.ec2.win but without success

      The code seems stuck I don't have log saying it is doing something.

      I downgrade to 1.50.2.1 and it is working correctly. With the logging enabled I managed to get this line that I don't have with the 1.50.3

       

      Jun 09, 2020 4:05:31 PM FINE hudson.plugins.ec2.win.WinConnection checking SMB connection to xx.xx.xx.xx

       

       

       

          [JENKINS-62641] Ec2 plugin: Cannot create windows node with ec2 1.50.3

          Kyle L added a comment -

          I'm experiencing this same issue with 1.0.53. In watching tcpdump, it's not attempting to make any connections on 445 or 5985. Also, nothing useful in the logs.

          Kyle L added a comment - I'm experiencing this same issue with 1.0.53. In watching tcpdump, it's not attempting to make any connections on 445 or 5985. Also, nothing useful in the logs.

          Dave Barker added a comment - - edited

          I've seen similar behavior. On 1.50.2  I've been able to work around the issue by deleting the agent on the hung EC2 instance via the Jenkins console.  I will have a VM, that was created by the plug-in overnight that is hung on startup   Once that agent is deleted the plugin spins up another instance and the job runs normally.  

          On 1.50.3 The work around does not work.  Any Windows VM the ec2 plugin starts never starts up successfully.

          I reverted to 1.50.2.

          Dave Barker added a comment - - edited I've seen similar behavior. On 1.50.2  I've been able to work around the issue by deleting the agent on the hung EC2 instance via the Jenkins console.  I will have a VM, that was created by the plug-in overnight that is hung on startup   Once that agent is deleted the plugin spins up another instance and the job runs normally.   On 1.50.3 The work around does not work.  Any Windows VM the ec2 plugin starts never starts up successfully. I reverted to 1.50.2.

          We're experiencing same issue with EC2 plugin.
          Jenkins Version: 2.235.1

          EC2 plugin: 1.50.3
          In order to work I have to downgrade EC2 plugin to 1.45. Any version between 1.46 and 1.50.3 is failing to launch Windows machine (EC2)

          Lakshmi Ravipati added a comment - We're experiencing same issue with EC2 plugin. Jenkins Version: 2.235.1 EC2 plugin: 1.50.3 In order to work I have to downgrade EC2 plugin to 1.45. Any version between 1.46 and 1.50.3 is failing to launch Windows machine (EC2)

          tapvir virk added a comment - - edited

          I am also facing the same issue.
          For now, as suggested, I will point to 1.50.2

          Looking forward for a fix

          tapvir virk added a comment - - edited I am also facing the same issue. For now, as suggested, I will point to 1.50.2 Looking forward for a fix

          Alexey Yambarshev added a comment - - edited

          Jenkins 2.222.4 with EC2 plugin 1.50.3 launches EC2 nodes (Windows as well).
          But it works with VPC in our case.
          We set following parameters for AMIs:

          • "Security group names"
          • "Subnet IDs for VPC"

          Alexey Yambarshev added a comment - - edited Jenkins 2.222.4 with EC2 plugin 1.50.3 launches EC2 nodes (Windows as well). But it works with VPC in our case. We set following parameters for AMIs: "Security group names" "Subnet IDs for VPC"

          rsandell added a comment -

          Since the versions you mention there has been a couple of security fixes. One IIRC has to do with avoiding MiM attacks and the plugin needs to parse the instance log or something to get the identity to validate which can take a lot of time.

          Try setting "Host Key Verification Strategy" to a less strict value (like off) and see if that solves the issue.

          rsandell added a comment - Since the versions you mention there has been a couple of security fixes. One IIRC has to do with avoiding MiM attacks and the plugin needs to parse the instance log or something to get the identity to validate which can take a lot of time. Try setting "Host Key Verification Strategy" to a less strict value (like off) and see if that solves the issue.

          Ramon Leon added a comment -

          The Host key verification doesn't affect the connection to Windows instances. The field in use is the Allow self signed certificates. To allow the connection to an ec2 windows instance you have to configure it creating the certificate with the IP in use, open the ports, ... Take a look at the documentation of GitHub and the one in the configuration page of the plugin in your Jenkins instance.

          tapvir, rlnchow dbarker randomvoids Could you try to do what ayambarshev suggested?. Specifying this two fields properly. This settings have changed from version to version. They were required on older versions, on 1.46-1.49 IIRC weren't required, but it's again required as of 1.50. This requirement has nothing to do with the security fixes, more likely with some refactoring in the code.

          If it works for you, we can add some information to the documentation to improve the configuration experience, which is not all good it should be.

          Thanks

          Ramon Leon added a comment - The Host key verification doesn't affect the connection to Windows instances. The field in use is the Allow self signed certificates . To allow the connection to an ec2 windows instance you have to configure it creating the certificate with the IP in use, open the ports, ... Take a look at the documentation of GitHub and the one in the configuration page of the plugin in your Jenkins instance. tapvir , rlnchow dbarker randomvoids Could you try to do what ayambarshev suggested? . Specifying this two fields properly. This settings have changed from version to version. They were required on older versions, on 1.46-1.49 IIRC weren't required, but it's again required as of 1.50. This requirement has nothing to do with the security fixes, more likely with some refactoring in the code. If it works for you, we can add some information to the documentation to improve the configuration experience, which is not all good it should be. Thanks

          tapvir virk added a comment -

          Hi mramonleon - I already have security groups and subnet id's configured. THough my Jenkins version is 2.235.1

          I tried Host Key Verification to set it to off. Still doesn't work.

          Thanks

          tapvir virk added a comment - Hi mramonleon - I already have security groups and subnet id's configured. THough my Jenkins version is 2.235.1 I tried Host Key Verification to set it to off. Still doesn't work. Thanks

          Dave Barker added a comment -

          I'll have to tweak with versions in this configuration.  I do have the two fields set ayambarshev mentions.  I always have.

          Dave Barker added a comment - I'll have to tweak with versions in this configuration.  I do have the two fields set ayambarshev mentions.  I always have.

          tapvir virk added a comment -

          dbarker - What do you mean by tweak with versions in this configuration?

          tapvir virk added a comment - dbarker - What do you mean by tweak with versions in this configuration?

          Dave Barker added a comment -

          I'm on Jenkins 2.235.1

          Dave Barker added a comment - I'm on Jenkins 2.235.1

          tapvir virk added a comment -

          dbarker - I'm on the same version. Are you able to attach the windows slave?

          tapvir virk added a comment - dbarker - I'm on the same version. Are you able to attach the windows slave?

          Adrien Zieba added a comment -

          On my side I have a configuration similar to ayambarshev but I still have an issue with provisioning windows nodes.

          Adrien Zieba added a comment - On my side I have a configuration similar to ayambarshev but I still have an issue with provisioning windows nodes.

          tapvir virk added a comment -

          Same with me

          tapvir virk added a comment - Same with me

          Dave Barker added a comment -

          I am able to RDP into the node Jenkins won't connect to.  

          Dave Barker added a comment - I am able to RDP into the node Jenkins won't connect to.  

          tapvir virk added a comment -

          I believe which is fine and is not related to the issue. As jenkins using winrm in order to connect to windows instance.

          Is the windows slave gets connected succesfully to your jenkins master?

          tapvir virk added a comment - I believe which is fine and is not related to the issue. As jenkins using winrm in order to connect to windows instance. Is the windows slave gets connected succesfully to your jenkins master?

          Alexey Yambarshev added a comment - - edited

          One more note.
          We uses Jenkins outside AWS.
          So, the following significant fields are actual for us in Jenkins (in Advanced subfields for each AMI):
           - Connection Strategy - Public IP
           - Host Key Verification Strategy - accept-new  . It will be updated in the future for security when we update all necessary keys in AMI and Jenkins.

          Alexey Yambarshev added a comment - - edited One more note. We uses Jenkins outside AWS . So, the following significant fields are actual for us in Jenkins (in Advanced subfields for each AMI):  - Connection Strategy - Public IP  - Host Key Verification Strategy - accept-new   . It will be updated in the future for security when we update all necessary keys in AMI and Jenkins.

            thoulen FABRIZIO MANFREDI
            adrienzieba Adrien Zieba
            Votes:
            5 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated: