• Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • ec2-plugin
    • None
    • Jenkins 2.1, Windows Server 2012 R2, EC2 version 1.31.

      Hello All,

      I've been trying to run this plugin with Windows 2012 and i'm stuck in some WinRM loop where it cannot connect. I tried setting the launch delay up to 600 seconds to ensure my instance is up before trying any WinRM commands. No luck. Here is what i'm seeing:

      May 04, 2016 4:42:34 PM FINE hudson.plugins.ec2.win.winrm.WinRMClient
      opening winrm shell to: http://172.31.202.208:5985/wsman
      May 04, 2016 4:42:34 PM FINEST hudson.plugins.ec2.win.winrm.WinRMClient
      Request:
      POST http://172.31.202.208:5985/wsman
      <?xml version="1.0" encoding="UTF-8"?>
      <env:Envelope xmlns:env="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:rsp="http://schemas.microsoft.com/wbem/wsman/1/windows/shell" xmlns:w="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd" xmlns:p="http://schemas.microsoft.com/wbem/wsman/1/wsman.xsd"><env:Header><a:To>http://172.31.202.208:5985/wsman</a:To><a:ReplyTo><a:Address mustUnderstand="true">http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous</a:Address></a:ReplyTo><w:MaxEnvelopeSize mustUnderstand="true">153600</w:MaxEnvelopeSize><a:MessageID>uuid:D6FA9229-05D6-4199-B7B6-ACFA6D195CBE</a:MessageID><w:Locale mustUnderstand="false" xml:lang="en-US"/><p:DataLocale mustUnderstand="false" xml:lang="en-US"/><w:OperationTimeout>PT60S</w:OperationTimeout><a:Action mustUnderstand="true">http://schemas.xmlsoap.org/ws/2004/09/transfer/Create</a:Action><w:ResourceURI mustUnderstand="true">http://schemas.microsoft.com/wbem/wsman/1/windows/shell/cmd</w:ResourceURI><w:OptionSet><w:Option Name="WINRS_NOPROFILE">FALSE</w:Option><w:Option Name="WINRS_CODEPAGE">437</w:Option></w:OptionSet></env:Header><env:Body><rsp:Shell><rsp:InputStreams>stdin</rsp:InputStreams><rsp:OutputStreams>stdout stderr</rsp:OutputStreams></rsp:Shell></env:Body></env:Envelope>
      May 04, 2016 4:42:35 PM FINEST hudson.plugins.ec2.win.winrm.WinRMClient
      Response:
      <?xml version="1.0" encoding="UTF-8"?>
      <s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:x="http://schemas.xmlsoap.org/ws/2004/09/transfer" xmlns:w="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd" xmlns:rsp="http://schemas.microsoft.com/wbem/wsman/1/windows/shell" xmlns:p="http://schemas.microsoft.com/wbem/wsman/1/wsman.xsd" xml:lang="en-US"><s:Header><a:Action>http://schemas.xmlsoap.org/ws/2004/09/transfer/CreateResponse</a:Action><a:MessageID>uuid:792A9FDE-030D-4646-89A5-8432A425C947</a:MessageID><a:To>http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous</a:To><a:RelatesTo>uuid:D6FA9229-05D6-4199-B7B6-ACFA6D195CBE</a:RelatesTo></s:Header><s:Body><x:ResourceCreated><a:Address>http://172.31.202.208:5985/wsman</a:Address><a:ReferenceParameters><w:ResourceURI>http://schemas.microsoft.com/wbem/wsman/1/windows/shell/cmd</w:ResourceURI><w:SelectorSet><w:Selector Name="ShellId">96758FBF-0CF8-4E59-9E41-FF96E16903AC</w:Selector></w:SelectorSet></a:ReferenceParameters></x:ResourceCreated><rsp:Shell><rsp:ShellId>96758FBF-0CF8-4E59-9E41-FF96E16903AC</rsp:ShellId><rsp:ResourceUri>http://schemas.microsoft.com/wbem/wsman/1/windows/shell/cmd</rsp:ResourceUri><rsp:Owner>JENKINSSLAVE\Administrator</rsp:Owner><rsp:ClientIP>172.31.202.241</rsp:ClientIP><rsp:IdleTimeOut>PT7200.000S</rsp:IdleTimeOut><rsp:InputStreams>stdin</rsp:InputStreams><rsp:OutputStreams>stdout stderr</rsp:OutputStreams><rsp:ShellRunTime>P0DT0H0M0S</rsp:ShellRunTime><rsp:ShellInactivity>P0DT0H0M0S</rsp:ShellInactivity></rsp:Shell></s:Body></s:Envelope>
      May 04, 2016 4:42:35 PM FINER hudson.plugins.ec2.win.winrm.WinRMClient
      shellid: 96758FBF-0CF8-4E59-9E41-FF96E16903AC
      May 04, 2016 4:42:35 PM FINE hudson.plugins.ec2.win.winrm.WinRMClient
      closing winrm shell 96758FBF-0CF8-4E59-9E41-FF96E16903AC
      
      May 4, 2016 4:20:46 PM WARNING hudson.plugins.ec2.win.winrm.WinRMClient sendRequest
      
      winrm returned 401 - shouldn't happen though - retrying in 2 minutes
      
      Waiting for WinRM to come up. Sleeping 10s.
      Connecting to ec2-windows-host.compute-1.amazonaws.com(<<host ip>>) with WinRM as svc....
      Waiting for WinRM to come up. Sleeping 10s.
      etc
      etc
      

      I know WinRM is setup and working correctly, i can connect over Telnet. And I've run the commands suggested from the plugin:

      winrm quickconfig
      winrm set winrm/config/service/Auth @{Basic="true"}
      winrm set winrm/config/service @{AllowUnencrypted="true"}
      winrm set winrm/config/winrs @{MaxMemoryPerShellMB="1024"}
      

      I do have a question in addition to the bug and that is: Do we just need to specify and administrative user, not precisely "Administrator" ?

      I've spent the entire day working on this and I'm simply out of options.

        1. EC2-gpedit1.png
          EC2-gpedit1.png
          340 kB
        2. EC2-gpedit2.png
          EC2-gpedit2.png
          247 kB
        3. JenkinsEC2Setup1.png
          JenkinsEC2Setup1.png
          32 kB
        4. JenkinsEC2Setup2.png
          JenkinsEC2Setup2.png
          26 kB
        5. Screenshot 2019-05-09 11.58.27.png
          Screenshot 2019-05-09 11.58.27.png
          241 kB

          [JENKINS-34610] Windows AMI in endless loop

          Peter Backx added a comment -

          Thank you Brendan for this information.

          I also found another cause of connection issues: the Windows 10 server AMIs on Amazon will block incoming WinRM traffic from the public network by default (I think this is the default setting in Windows). This needs to be changed in the firewall rules.

          (BTW I would suggest to only allow traffic from the Jenkins master IP, either through the Windows firewall or the Amazon security group or both)

          Peter Backx added a comment - Thank you Brendan for this information. I also found another cause of connection issues: the Windows 10 server AMIs on Amazon will block incoming WinRM traffic from the public network by default (I think this is the default setting in Windows). This needs to be changed in the firewall rules. (BTW I would suggest to only allow traffic from the Jenkins master IP, either through the Windows firewall or the Amazon security group or both)

          Ultimately, opposite of my previous comment, we ended up giving up on this plugin for various reasons.

          But, we did enable all WinRm ports, including secure as well. That wasn't the cause of this issue. Thank you for getting back though.

          We hope to try and use it again in the future. But for our configuration it just wasn't up to the task. Perhaps it is today though, through bug fixing.

          Brendan Stewart added a comment - Ultimately, opposite of my previous comment, we ended up giving up on this plugin for various reasons. But, we did enable all WinRm ports, including secure as well. That wasn't the cause of this issue. Thank you for getting back though. We hope to try and use it again in the future. But for our configuration it just wasn't up to the task. Perhaps it is today though, through bug fixing.

          Aiden Brusby added a comment -

          francisu mumbles76 What is the status of this? did it get resolved?. I'm encountering this issue when trying to launch windows server 2016 AMIs via the plugin. Would be a shame to not have this functionality working.

           

          Aiden Brusby added a comment - francisu mumbles76 What is the status of this? did it get resolved?. I'm encountering this issue when trying to launch windows server 2016 AMIs via the plugin. Would be a shame to not have this functionality working.  

          abrusby  i'm facing the same issue, did you resolve in some way?

          Alessandro De Angelis added a comment - abrusby   i'm facing the same issue, did you resolve in some way?

          I'm facing the same issue, though a bit proficient in Java and will try to submit a PR together with the changes coming from snallami in JENKINS-4995, perhaps I'll have something to test later this week, but was wondering if I can use an external library to handle the WinRM communication (like https://github.com/cloudsoft/winrm4j), but not really sure who on the project I need to contact to check if it's a valid approach or not.

          Gonzalo Garcia added a comment - I'm facing the same issue, though a bit proficient in Java and will try to submit a PR together with the changes coming from snallami in  JENKINS-4995 , perhaps I'll have something to test later this week, but was wondering if I can use an external library to handle the WinRM communication (like https://github.com/cloudsoft/winrm4j),  but not really sure who on the project I need to contact to check if it's a valid approach or not.

          James Green added a comment -

          Some observations having spent the past few days swearing lots to get this reliably working:

          1. It may be necessary to create a WinRMRemoteWMIUsers__ group - this persists across instance launches
          2. Any winrm configuration commands given when creating your AMI may be missing on new instance launch. I've added into the "User Data" section the following text which seems to contribute to overcoming the reported problems in connecting:
            <script>
            winrm quickconfig -q
            winrm set winrm/config/service/Auth @{Basic="true"}
            winrm set winrm/config/service @{AllowUnencrypted="true"}
            </script>
            However I have noticed that even these instructions can be ignored on next instance boot and I am not convinced they are needed - they got thrown into the configuration along with these other points and left there
          3. winrm (which is used under the hood) is extremely slow. I mean so slow you think it has broken. I've observed `httpRequest` calls taking multiple minutes - when executing the same thing on the slave instance through Chrome has worked instantly. Subsequent builds on the same instance appear much faster to operate.
          4. The Jenkins master can in a muddle when instances are rebooted (for instance to create a new AMI image) which results in high load on the master and in one case two consecutive builds failed when trying to issue a command-line instruction during the build sequence. Rebooting the master resulted in no errors the next attempt.

          In short, this appears to be an unreliable means to operate a Windows server, and perhaps not the fault of this plugin author or even the winrm plugin's author. It may be wise to consider launching a Windows slave agent with a Jenkins service instead, not something we want to do but there may no other tenable way forward.

          James Green added a comment - Some observations having spent the past few days swearing lots to get this reliably working: It may be necessary to create a WinRMRemoteWMIUsers__ group - this persists across instance launches Any winrm configuration commands given when creating your AMI may be missing on new instance launch. I've added into the "User Data" section the following text which seems to contribute to overcoming the reported problems in connecting: <script> winrm quickconfig -q winrm set winrm/config/service/Auth @{Basic="true"} winrm set winrm/config/service @{AllowUnencrypted="true"} </script> However I have noticed that even these instructions can be ignored on next instance boot and I am not convinced they are needed - they got thrown into the configuration along with these other points and left there winrm (which is used under the hood) is extremely slow. I mean so slow you think it has broken. I've observed `httpRequest` calls taking multiple minutes - when executing the same thing on the slave instance through Chrome has worked instantly. Subsequent builds on the same instance appear much faster to operate. The Jenkins master can in a muddle when instances are rebooted (for instance to create a new AMI image) which results in high load on the master and in one case two consecutive builds failed when trying to issue a command-line instruction during the build sequence. Rebooting the master resulted in no errors the next attempt. In short, this appears to be an unreliable means to operate a Windows server, and perhaps not the fault of this plugin author or even the winrm plugin's author. It may be wise to consider launching a Windows slave agent with a Jenkins service instead, not something we want to do but there may no other tenable way forward.

          Alex Taylor added a comment - - edited

          We were able to get the windows agent to run by setting the "Allow remote server management through WinRM" IP settings to "*". Screenshot attached.

          We also had to run the below commands.

          winrm quickconfig 
          winrm set winrm/config/service/Auth @{Basic="true"} 
          winrm set winrm/config/service @{AllowUnencrypted="true"} 
          winrm set winrm/config/winrs @{MaxMemoryPerShellMB="10240"}
          

          Should this exist on the wiki page francisu? I was thinking there is not much "official troubleshooting" for windows EC2 connections and it seems lots of people are having troubles.

          Alex Taylor added a comment - - edited We were able to get the windows agent to run by setting the "Allow remote server management through WinRM" IP settings to "*". Screenshot attached. We also had to run the below commands. winrm quickconfig winrm set winrm/config/service/Auth @{Basic= " true " } winrm set winrm/config/service @{AllowUnencrypted= " true " } winrm set winrm/config/winrs @{MaxMemoryPerShellMB= "10240" } Should this exist on the wiki page francisu ? I was thinking there is not much "official troubleshooting" for windows EC2 connections and it seems lots of people are having troubles.

          ovi craciun added a comment - - edited

          I am facing the same issue I have tried what Alex Taylor and Brendan Stewart have suggested (all the settings for WinRM and everything else) but the master is still logging the same error:

           EC2 (AWS ID account_id) - windows slave (sir-31i1js7h) booted at 1574726631000
           Connecting to (17.30.6.34) with WinRM as Administrator
           Waiting for WinRM to come up. Sleeping 10s.
           Waiting for WinRM to come up. Sleeping 10s.
          

          However, I can telnet the slave on the 5985 port from master jenkins server and it connects. Does it mean it can't pass authentication or there's something else going won when the plugin is trying to connect via WinRM?

          The plugin doc is saying "By default, the plugin connects to the slave agent using an in-process Java SSH client.". What is the port is using the connect to this in-process ssh client?

          Any other ideas what should I try here? I've lost already two days to get this to work.

          Thank you

          ovi craciun added a comment - - edited I am facing the same issue I have tried what  Alex Taylor  and  Brendan Stewart have suggested (all the settings for WinRM and everything else) but the master is still logging the same error: EC2 (AWS ID account_id) - windows slave (sir-31i1js7h) booted at 1574726631000 Connecting to (17.30.6.34) with WinRM as Administrator Waiting for WinRM to come up. Sleeping 10s. Waiting for WinRM to come up. Sleeping 10s. However, I can telnet the slave on the 5985 port from master jenkins server and it connects. Does it mean it can't pass authentication or there's something else going won when the plugin is trying to connect via WinRM? The plugin doc is saying "By default, the plugin connects to the slave agent using an in-process Java SSH client.". What is the port is using the connect to this in-process ssh client? Any other ideas what should I try here? I've lost already two days to get this to work. Thank you

          ovi craciun added a comment - - edited

          I have managed to fix it, the culprit: port 445 needs to be opened on the slave machines. More details here: https://stackoverflow.com/questions/38011683/how-to-run-windows-instance-on-ec2-from-jenkins

          ovi craciun added a comment - - edited I have managed to fix it, the culprit: port 445 needs to be opened on the slave machines. More details here: https://stackoverflow.com/questions/38011683/how-to-run-windows-instance-on-ec2-from-jenkins

          KRUNAL PATIL added a comment -

          I was facing same problem, I solved it by mentioning Remote user as "administrator" and then specifying admin password. Also make sure that SMB port 445 and Winrm port 5985(for http) or else winrm port 5986( in case you are using https) must allow incoming connections to it.

          KRUNAL PATIL added a comment - I was facing same problem, I solved it by mentioning Remote user as "administrator" and then specifying admin password. Also make sure that SMB port 445 and Winrm port 5985(for http) or else winrm port 5986( in case you are using https) must allow incoming connections to it.

            francisu Francis Upton
            mumbles76 Brendan Stewart
            Votes:
            3 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated: