-
Improvement
-
Resolution: Fixed
-
Major
-
None
-
Powered by SuggestiMate -
ec2 plugin 1.24
Supporting windows would be really useful: we ship our products on windows as well as Linux, but all our devs run Linux
[JENKINS-4995] Support windows AMI's in EC2 plugin
It should be possible, since Jenkins has support for WMI, and actually supports starting slave agents on Windows machines remotely. But it requires some changes to the plugin itself.
Having the EC2 plugin support Windows would be great; it would keep me from writing custom scripts. However, has there been much progress of late? I do see that the issue has been "In Progress" for the last year and change.
I didn't made any progress on this issue, as I stopped using Jenkins together with EC2. I still believe that it would be fairly easy to integrate using the built-in WMI support in Jenkins.
Has anyone had success launching and connecting a Windows instance using this plugin?
I tried using a Windows instance that had the slave service pre-installed with no luck. (The launch log just kept waiting for ssh to connect.)
I see there is an option for an initialization script in the configuration. Maybe a Groovy guru could do something with that?
Perhaps the route might be to get Windows to connect with ssh?
As a temporary measure, you can make a script in your Windows instance to connect with JNLP, the spot folks have used that.
Code changed in jenkins
User: Francis Upton IV
Path:
pom.xml
src/main/java/hudson/plugins/ec2/AMITypeData.java
src/main/java/hudson/plugins/ec2/EC2AbstractSlave.java
src/main/java/hudson/plugins/ec2/EC2OndemandSlave.java
src/main/java/hudson/plugins/ec2/EC2SpotSlave.java
src/main/java/hudson/plugins/ec2/SlaveTemplate.java
src/main/java/hudson/plugins/ec2/UnixData.java
src/main/java/hudson/plugins/ec2/WindowsData.java
src/main/java/hudson/plugins/ec2/win/EC2WindowsLauncher.java
src/main/java/hudson/plugins/ec2/win/WinConnection.java
src/main/java/hudson/plugins/ec2/win/winrm/RuntimeIOException.java
src/main/java/hudson/plugins/ec2/win/winrm/WinRM.java
src/main/java/hudson/plugins/ec2/win/winrm/WinRMClient.java
src/main/java/hudson/plugins/ec2/win/winrm/WinRMConnectException.java
src/main/java/hudson/plugins/ec2/win/winrm/WindowsProcess.java
src/main/java/hudson/plugins/ec2/win/winrm/request/AbstractWinRMRequest.java
src/main/java/hudson/plugins/ec2/win/winrm/request/DeleteShellRequest.java
src/main/java/hudson/plugins/ec2/win/winrm/request/ExecuteCommandRequest.java
src/main/java/hudson/plugins/ec2/win/winrm/request/GetOutputRequest.java
src/main/java/hudson/plugins/ec2/win/winrm/request/OpenShellRequest.java
src/main/java/hudson/plugins/ec2/win/winrm/request/RequestFactory.java
src/main/java/hudson/plugins/ec2/win/winrm/request/SendInputRequest.java
src/main/java/hudson/plugins/ec2/win/winrm/request/SignalRequest.java
src/main/java/hudson/plugins/ec2/win/winrm/request/WinRMRequest.java
src/main/java/hudson/plugins/ec2/win/winrm/soap/Header.java
src/main/java/hudson/plugins/ec2/win/winrm/soap/HeaderBuilder.java
src/main/java/hudson/plugins/ec2/win/winrm/soap/MessageBuilder.java
src/main/java/hudson/plugins/ec2/win/winrm/soap/Namespaces.java
src/main/java/hudson/plugins/ec2/win/winrm/soap/Option.java
src/main/resources/hudson/plugins/ec2/EC2Computer/configure.jelly
src/main/resources/hudson/plugins/ec2/SlaveTemplate/config.jelly
src/main/resources/hudson/plugins/ec2/SlaveTemplate/help-amiType.html
src/main/resources/hudson/plugins/ec2/UnixData/config.jelly
src/main/resources/hudson/plugins/ec2/WindowsData/config.jelly
src/main/resources/hudson/plugins/ec2/WindowsData/help-bootDelay.html
src/test/java/hudson/plugins/ec2/EC2AbstractSlaveTest.java
src/test/java/hudson/plugins/ec2/EC2OndemandSlaveTest.java
src/test/java/hudson/plugins/ec2/EC2RetentionStrategyTest.java
src/test/java/hudson/plugins/ec2/SlaveTemplateTest.java
src/test/java/hudson/plugins/ec2/TemplateLabelsTest.java
src/test/java/hudson/plugins/ec2/WinRMMessageTest.java
http://jenkins-ci.org/commit/ec2-plugin/859dd4ea8aeedaad2484454df58d31b94fdcd472
Log:
JENKINS-4995 Support Windows AMIs
Code changed in jenkins
User: Francis Upton
Path:
pom.xml
src/main/java/hudson/plugins/ec2/AMITypeData.java
src/main/java/hudson/plugins/ec2/EC2AbstractSlave.java
src/main/java/hudson/plugins/ec2/EC2OndemandSlave.java
src/main/java/hudson/plugins/ec2/EC2SpotSlave.java
src/main/java/hudson/plugins/ec2/SlaveTemplate.java
src/main/java/hudson/plugins/ec2/UnixData.java
src/main/java/hudson/plugins/ec2/WindowsData.java
src/main/java/hudson/plugins/ec2/win/EC2WindowsLauncher.java
src/main/java/hudson/plugins/ec2/win/WinConnection.java
src/main/java/hudson/plugins/ec2/win/winrm/RuntimeIOException.java
src/main/java/hudson/plugins/ec2/win/winrm/WinRM.java
src/main/java/hudson/plugins/ec2/win/winrm/WinRMClient.java
src/main/java/hudson/plugins/ec2/win/winrm/WinRMConnectException.java
src/main/java/hudson/plugins/ec2/win/winrm/WindowsProcess.java
src/main/java/hudson/plugins/ec2/win/winrm/request/AbstractWinRMRequest.java
src/main/java/hudson/plugins/ec2/win/winrm/request/DeleteShellRequest.java
src/main/java/hudson/plugins/ec2/win/winrm/request/ExecuteCommandRequest.java
src/main/java/hudson/plugins/ec2/win/winrm/request/GetOutputRequest.java
src/main/java/hudson/plugins/ec2/win/winrm/request/OpenShellRequest.java
src/main/java/hudson/plugins/ec2/win/winrm/request/RequestFactory.java
src/main/java/hudson/plugins/ec2/win/winrm/request/SendInputRequest.java
src/main/java/hudson/plugins/ec2/win/winrm/request/SignalRequest.java
src/main/java/hudson/plugins/ec2/win/winrm/request/WinRMRequest.java
src/main/java/hudson/plugins/ec2/win/winrm/soap/Header.java
src/main/java/hudson/plugins/ec2/win/winrm/soap/HeaderBuilder.java
src/main/java/hudson/plugins/ec2/win/winrm/soap/MessageBuilder.java
src/main/java/hudson/plugins/ec2/win/winrm/soap/Namespaces.java
src/main/java/hudson/plugins/ec2/win/winrm/soap/Option.java
src/main/resources/hudson/plugins/ec2/EC2Computer/configure.jelly
src/main/resources/hudson/plugins/ec2/SlaveTemplate/config.jelly
src/main/resources/hudson/plugins/ec2/SlaveTemplate/help-amiType.html
src/main/resources/hudson/plugins/ec2/UnixData/config.jelly
src/main/resources/hudson/plugins/ec2/WindowsData/config.jelly
src/main/resources/hudson/plugins/ec2/WindowsData/help-bootDelay.html
src/test/java/hudson/plugins/ec2/EC2AbstractSlaveTest.java
src/test/java/hudson/plugins/ec2/EC2OndemandSlaveTest.java
src/test/java/hudson/plugins/ec2/EC2RetentionStrategyTest.java
src/test/java/hudson/plugins/ec2/SlaveTemplateTest.java
src/test/java/hudson/plugins/ec2/TemplateLabelsTest.java
src/test/java/hudson/plugins/ec2/WinRMMessageTest.java
http://jenkins-ci.org/commit/ec2-plugin/a3bd08def1ed2d2f30c5fa73be707fb702c2a9e5
Log:
Merge pull request #105 from jenkinsci/feature/windows-slave-support
JENKINS-4995 Support Windows AMIs
Compare: https://github.com/jenkinsci/ec2-plugin/compare/04cd780ebe71...a3bd08def1ed
Even with the checkins to date, this is not quite complete. More testing it required. It has been integrated with the master source in the plugin so that hopefully others can test it.
I'm trying to utilize this plugin for both Linux and Windows slave provisioning. Linux of course functions just fine. For my Windows machines, I am running into an infinite loop during the WinRM ping() process (I think). The firewall on my build agent is wide open and I've verified WinRM availability in my configuration with a separate instance in the same EC2 availability zone, so I think an exception is occurring somewhere in the ping() loop but I don't have access to detailed log info or a stack trace. I am willing to help develop and test Windows utilization, but could use some pointers on how to move forward or any troubleshooting you can suggest
@Bryan, it would be great to get your help on this. Can't give you any hints right this minute. Let me look into things and get back to you. Ping me again early next week if you don't hear from me.
@Bryan, I've run into an infinte loop when starting windows instances too. Maybe it is the same bug. There was a NullPointerException showing up in the log every 10s. I wrote a small fix for it, not sure if I'm fixing the root cause, but for me it works. I've been able to start windows Instances!
Code is here:
https://github.com/smurn/ec2-plugin
Pull request is here:
https://github.com/jenkinsci/ec2-plugin/pull/113
@Francis
First, thanks for your work! This is super useful for me.
Since you asked others to test it, here are my experiences with windows instances so far:
- The current release fails because it attempts to connect to WinRM using the instance's local IP. This seems to be fixed in the master branch.
- The master branch version pings forever because of an NullPointerException. See https://github.com/jenkinsci/ec2-plugin/pull/113
- With that fix in place, ping still looped forever, no exceptions or error messages in the logs.
- I've semi-randomly added some log statements to various places in the code and found the cause to be an WinRM authenification failure. Apparently some exceptions get ignored without logging which makes it very hard to track down problems.
- The authenification failure was an unsupported authentification method issue thrown deep inside apache's http library. The authentification method was reported as "negotiate".
- I had "basic" auth enabled for WinRM as suggested in the help-text in jenkins configuration section.
- I finally got it to work by disabling "negotiate" with winrm set winrm/config/service/auth @{Negotiate="false"}. I've also disabled "Kerberos" while I was on it, but I'm not sure if it had any effect on the issue.
- This has the unfourtnate draw-back that `winrm` commands in the local console will no longer work (apparently they use this negotiate method), unless additional options are passed to it to use basic auth.
- There are still quite a number of exceptions showing up, before, during, and after the ping-phase, but the node comes up and runs jobs at the end of it.
- On a side note, I'm a bit worried about port 445. SMB isn't a protocol I'd usually run directy over the internet. Sys-admins will not like this one bit. Maybe there are ways to get the jar over using winRM only (http://stackoverflow.com/questions/10635238/send-files-over-pssession).
For anyone else using Windows Server 2008 R2 SP1 as a slave, I kept running into problems running slave.jar due to JVM complaining about memory.
It turns out there's a bug in Windows Server 2008 R2 SP1 that doesn't respect the WinRM setting for MaxMemoryPerShellMB and always uses 150MB no matter what you set.
You can read about this problem and download the hotfix (big button at the top of the page) here: http://support.microsoft.com/kb/2842230
Code changed in jenkins
User: Jason Mittertreiner
Path:
src/main/java/hudson/plugins/ec2/win/EC2WindowsLauncher.java
src/main/java/hudson/plugins/ec2/win/winrm/WinRMClient.java
http://jenkins-ci.org/commit/ec2-plugin/0e840f7129b91af5101cb8f08f938743dc188ff9
Log:
JENKINS-27260 SPNEGO for Windows in EC2 Plugin
Fixed the Windows temp directory getting set to ""
Enabled SPNEGO authentication
JENKINS-25385 and JENKINS-4995 both have comments complaining about
infinite loops when creating Windows slaves. Because SPNEGO is
unregistered for, the httpclient throws and exception that is silently
caught and causes the infinite loop.
I added NTLM with SPNEGO by pull request https://github.com/jenkinsci/ec2-plugin/pull/165, please check it out.
To test, you should enable ssl on WinRM and have a working Administrator account, and `Use HTTPS` is also enabled in the plugin
For those who want to use other local accounts except Administrator, please adjust windows registry
reg add HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System /v LocalAccountTokenFilterPolicy /t REG_DWORD /d 1 /f
Is there any documentation on this? I tried everything but could not get winrm work
Want to add that i could get windows slaves working using JNLP approach for my use case by modifying plugin.
Below is the approach i took.
1) Prepared custom windows AMI with java runtime.
2) From Jenkins, during windows slave creation passing boot script in "EC2 User Data".
Script does following
a) Downloads slave.jar from Jenkins master
b) Executes slave.jar using below command
java -jar slave.jar -secret <slaveSecret> -jnlpUrl <jenkinsServerURL>/jenkins/computer/<VMName>/slave-agent.jnlp
slave secret is generated using jenkins API "JnlpSlaveAgentProtocol.SLAVE_SECRET.mac(vmName);"
I could able to get it working with jenkins master on both Linux and Windows in AWS cloud.
I will submit a PR for this however would like to know how do you want to control between winrm and JNLP approach for windows slaves
Also looking forward for some guidance on how to get winrm working
I'm facing the same problem (winrm infinite loop)
is there a way (an hackish way maybe?), a documentation or a script to let the plugin work with a windows slave?
Can someone post it?
Regards
L.
Ideally the plugin should add support of standard ComputerLauncher extension implementations instead of defining them implicitly. DCOM Launcher in Windows Slaves plugin is a bloody mess, but it is likely possible to make it working on custom AMIs
The proposal from snallami is also reasonable, but I do not see the PR he was going to create
Folks, any update on this? I see others have posted the same question but its been almost a year. We also would like to manage/control both Linux and Windows build servers. So far we have had great luck with Linux box and this has helped us save a bunch of $$$. Thank you for a great plugin. Now here is hoping we can get it tweaked to support Windows AMI.
I think it works if you configure the Windows AMIs with userdata script to create a local user and enable WinRM.
jg_lgc - Hmmm can you share an example of a userdata script as we could not find it here.
- We do have a local user setup, we do have WinRM enabled.
- We did disable WInRM "negotiation" with
set winrm/config/service/auth @{Negotiate="false"}
- We also followed Ted's comment since we are using our own local user account so added this to allow other local accounts to work:
reg add HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System /v LocalAccountTokenFilterPolicy /t REG_DWORD /d 1 /f
But we get the same winrm infinite loop issue. With these values in the logs:
July 11, 2018 9:10:27 AM org.apache.http.client.protocol.RequestAuthenticationBase process
WARNING: NEGOTIATE authentication error: Invalid name provided (Mechanism level: KrbException: Cannot locate default realm)
Should we have disable Kerberos as well?
Thanks all for your help!
The maxmemorypershell is pretty important for the agent to be able to actually do anything once launched.
<powershell>
winrm set winrm/config @{MaxTimeoutms="300000"}
winrm set winrm/config/service/auth @{Basic="true"}
winrm set winrm/config/service/auth @{Kerberos="true"}
winrm set winrm/config/service @{AllowUnencrypted="true"}
winrm set winrm/config/client @{TrustedHosts="<jenkins master>"}
winrm set winrm/config/service @{EnableCompatibilityHttpListener="true"}
winrm set winrm/config/winrs @{MaxMemoryPerShellMB="12288"}
restart-service winrm
Hi,
I'm trying to setup the windows slave and getting below error. Can someone help?
Failed to verify connectivity to Windows slave
java.lang.IllegalStateException: Unsupported authentication scheme: CredSSP
at org.apache.http.auth.AuthSchemeRegistry.getAuthScheme(AuthSchemeRegistry.java:116)
at org.apache.http.auth.AuthSchemeRegistry$1.create(AuthSchemeRegistry.java:152)
at org.apache.http.impl.client.AuthenticationStrategyImpl.select(AuthenticationStrategyImpl.java:189)
at org.apache.http.impl.client.TargetAuthenticationStrategy.select(TargetAuthenticationStrategy.java:44)
at org.apache.http.impl.auth.HttpAuthenticator.handleAuthChallenge(HttpAuthenticator.java:154)
at org.apache.http.impl.client.HttpAuthenticator.authenticate(HttpAuthenticator.java:58)
at org.apache.http.impl.client.DefaultRequestDirector.handleResponse(DefaultRequestDirector.java:1056)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:514)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at hudson.plugins.ec2.win.winrm.WinRMClient.sendRequest(WinRMClient.java:217)
at hudson.plugins.ec2.win.winrm.WinRMClient.sendRequest(WinRMClient.java:188)
at hudson.plugins.ec2.win.winrm.WinRMClient.openShell(WinRMClient.java:82)
at hudson.plugins.ec2.win.winrm.WinRM.ping(WinRM.java:29)
at hudson.plugins.ec2.win.WinConnection.ping(WinConnection.java:132)
at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:136)
at hudson.plugins.ec2.win.EC2WindowsLauncher.launchScript(EC2WindowsLauncher.java:33)
at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48)
at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Getting the "java.lang.IllegalStateException: Unsupported authentication scheme: CredSSP" error message as well.
Haven't used the EC2 agent for windows agents for last month or so, but it was working perfectly before that.
Can RDP and WinRM in to the Windows agent that it spun up from my develop machine and our linux jenkins master. The agent just doesn't come on line, and we get the infinitely repeated "Waiting for WinRM to come up. Sleeping 10s." message. Can't find any corresponding logs on the Windows machine (don't you love :ms.
I think the last time we used the ec2 agent plugin we were using 1.39, so I'm guessing 1.40 or 1.41 have broken it?
Managed to get my windows jenkins agent online
Just disabled the Server role of WSManCredSSP.
PS C:\Users\Administrator> Disable-WSManCredSSP -Role Server
Didn't work in the init data, cos that only gets executed after agent has come online. So used schtasks to execute it on startup in my packer build of agent's ami.
I believe this issue could (or should) actually be closed as Resolved, and new issues be filed about the issues that are still present.
AFAIU, Francis worked on adding support (see commits linked as a comment in 2014 https://issues.jenkins-ci.org/browse/JENKINS-4995?focusedCommentId=206750&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-206750).
People have rightly reported various issues above (infinite loop, etc.?).
But then I also see people providing workarounds.
In other words, I believe closing and having people report more specific issues would provide clarity and make this more actionable.
I think I'll close this JIRA in the next days if people don't disagree. *Again*, this is not to say things are great and work perfectly already. The goal is actually to target more a win-win: better clarity, so the current EC2 plugin maintainers (I'm not one) can more easily understand what needs to be fixed and help users here with issues.
Thanks!
I also think it could be closed. Some documentation may be helpful, but I do not see anything outstanding left here.
Please correct me if I am wrong
We could wait another few months to close it after it's 10th anniversary.
I'm going to assume Francis was being sarcastic (which I like ), so closing to avoid keeping this misleading for another 10 months or so
.
Please file specific issues if you find something not working. Please read my last comment above: this is NOT being closed because there's no issue at all, but because we want specific issues with reproduction steps to be actually able to understand and fix remaining issues.
Thanks!
I'm told https://ec2-sshd.dev.java.net/ may be useful here