Details
-
Improvement
-
Status: Closed (View Workflow)
-
Blocker
-
Resolution: Fixed
-
None
-
jenkins: 1.597
ec2-plugin: 1.24
Description
The EC2 plugin is currently throwing the following error while trying to launch a build slave:
Jan 19, 2015 5:55:48 PM hudson.plugins.ec2.EC2Cloud provision WARNING: Failed to count the # of live instances on EC2 com.amazonaws.AmazonClientException: Unable to execute HTTP request: eu-central-1.ec2.amazonaws.com: Name or service not known
Performing a resolve directly against the SOA nameservers results in no answer:
dig eu-central-1.ec2.amazonaws.com @r1.amazonaws.com ; <<>> DiG 9.9.2-P2 <<>> eu-central-1.ec2.amazonaws.com @r1.amazonaws.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33907 ;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 6, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;eu-central-1.ec2.amazonaws.com. IN A ;; AUTHORITY SECTION: ec2.amazonaws.com. 907 IN NS ns-911.amazon.com. ec2.amazonaws.com. 907 IN NS ns-912.amazon.com. ec2.amazonaws.com. 907 IN NS ns-921.amazon.com. ec2.amazonaws.com. 907 IN NS ns-923.amazon.com. ec2.amazonaws.com. 907 IN NS ns-941.amazon.com. ec2.amazonaws.com. 907 IN NS ns-942.amazon.com. ;; Query time: 25 msec ;; SERVER: 205.251.192.27#53(205.251.192.27) ;; WHEN: Mon Jan 19 17:57:13 2015 ;; MSG SIZE rcvd: 192
The AWS API docs specify EC2 hostnames like the following:
... ec2.eu-central-1.amazonaws.com ...
So, the "ec2" and "eu-central-1" portions are flipped. I recommend that the API endpoints are updated to use the hostnames that are currently outlined in the API docs.
Attachments
Activity
Stephan Schultchen
added a comment - Hey, this bug is biting me as well. 
Francis Upton
added a comment - https://issues.jenkins-ci.org/browse/JENKINS-26854 Starting instances in eu-central-1 still does not work for me even though I use the latest version (1.29) of the Amazon EC2 plugin.
See excerpt of the jenkins.log:
Aug 21, 2015 6:51:10 AM com.amazonaws.http.AmazonHttpClient executeHelper
INFO: Unable to execute HTTP request: eu-central-1.ec2.amazonaws.com
java.net.UnknownHostException: eu-central-1.ec2.amazonaws.com
at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
...
at java.lang.Thread.run(Thread.java:745)
arkaitzj
added a comment - Hi
This is a problem for me as well, I don't know how the rest of the people are working with the current plugin.
According to the code and this EC2 documentation the API endpoints that the plugin constructs will never work.
This is just a simple URL constructing function that needs to be changed, there is even a pull request in github to get it fixed.
Could we get that merged or the problem fixed please?
arkaitzj
added a comment - Hi
This is a problem for me as well, I don't know how the rest of the people are working with the current plugin.
According to the code and this EC2 documentation the API endpoints that the plugin constructs will never work.
This is just a simple URL constructing function that needs to be changed, there is even a pull request in github to get it fixed.
Could we get that merged or the problem fixed please? 
Any idea if this is going to be fixed soon?
Toni Van de Voorde
added a comment - Got the same issue Any idea if this is going to be fixed soon? Francis Upton
added a comment - Sorry about the delay, I will have a look at this in the next day or so.
Francis Upton
added a comment - Sorry about the delay, I will have a look at this in the next day or so. Francis Upton
added a comment - I don't understand why these have to be wired into the code? Can someone help me here?
Francis Upton
added a comment - I don't understand why these have to be wired into the code? Can someone help me here? 
Johannes W
added a comment - Sorry for changing the assignee by mistake. I too have this problem with 1.29. Here is my take on a pull request: https://github.com/jenkinsci/ec2-plugin/pull/167
Johannes W
added a comment - Sorry for changing the assignee by mistake. I too have this problem with 1.29. Here is my take on a pull request: https://github.com/jenkinsci/ec2-plugin/pull/167 Francis Upton
added a comment - No worries about the assignee. I have finally begun to look at this a bit and have commented on a previous pull request (https://github.com/jenkinsci/ec2-plugin/pull/161) that also claims to fix this (along with addressing the s3 hostnames [though it's unclear that that's required]). Hopefully we will get that other pull request in in the next day or so.
Francis Upton
added a comment - No worries about the assignee. I have finally begun to look at this a bit and have commented on a previous pull request ( https://github.com/jenkinsci/ec2-plugin/pull/161 ) that also claims to fix this (along with addressing the s3 hostnames [though it's unclear that that's required] ). Hopefully we will get that other pull request in in the next day or so. Johannes W
added a comment - Any updates? I tested my pull request #167 now and it works for me. Francis Upton
added a comment - I have made the fix. I can't make a release at the moment because there is another critical bug in the unreleased version that I have to fix. I will try and get to it soon.
Francis Upton
added a comment - I have made the fix. I can't make a release at the moment because there is another critical bug in the unreleased version that I have to fix. I will try and get to it soon. Francis Upton
added a comment - Can you verify this is working in the current (1.30) snapshot, and then I can release it?
Francis Upton
added a comment - Can you verify this is working in the current (1.30) snapshot, and then I can release it? Francis Upton
added a comment - To be released in 1.30 I tested a snapshot but ran into problems. First, the aws-java-sdk plugin is only available in version 1.10.26 from within Jenkins but the EC2 plugin depends on 1.10.16.
You should probably update to 1.10.26 prior to release.
I got the 1.10.16 version of aws-java-sdk from the archive and could manually install the 1.30-SNAPSHOT of the ec2 plugin.
When I tried to test the connection from withing "Manage Jenkins" -> "Cloud" -> "Amazon EC2" for the AZ eu-central-1 i got an error:
AWS was not able to validate the provided access credentials (Service: AmazonEC2; Status Code: 401; Error Code: AuthFailure; Request ID: ea2e20e5-1e5f-4339-be55-721b6b1a66bc)
When I try to test the connection to any other availability zone where the the key is missing I get
The EC2 key pair private key isn't registered to this EC2 region (fingerprint is fd:1c:<redacted>:7e)
It seems like there is still a problem with the APIs and eu-central-1
Full stack trace:
Oct 26, 2015 8:14:12 AM hudson.plugins.ec2.EC2Cloud$DescriptorImpl doTestConnection
WARNING: Failed to check EC2 credential
com.amazonaws.AmazonServiceException: AWS was not able to validate the provided access credentials (Service: AmazonEC2; Status Code: 401; Error Code: AuthFailure; Request ID: ea2e20e5-1e5f-4339-be55-721b6b1a66bc)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1181)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:766)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:485)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:306)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:11901)
at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5940)
at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:11256)
at hudson.plugins.ec2.EC2Cloud$DescriptorImpl.doTestConnection(EC2Cloud.java:616)
at hudson.plugins.ec2.AmazonEC2Cloud$DescriptorImpl.doTestConnection(AmazonEC2Cloud.java:175)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:123)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
See my latest comment on this issue.
It could be that I did something strange but I think the error I got is worth looking into.
Johannes W
added a comment - @allotria: I had the same problem with aws-java-sdk and with the 401 error. But I discovered that in my case it was not the fault of the ec2-plugin: While the IAM credentials worked with aws-cli from my laptop, I got the same 401 error when using the aws-cli from the EC2 instance where Jenkins was running.
Can you confirm if you experience the same behavior?
Johannes W
added a comment - @ allotria : I had the same problem with aws-java-sdk and with the 401 error. But I discovered that in my case it was not the fault of the ec2-plugin: While the IAM credentials worked with aws-cli from my laptop, I got the same 401 error when using the aws-cli from the EC2 instance where Jenkins was running.
Can you confirm if you experience the same behavior? 
Joerg Baldzer
added a comment - Hi,
I just included this pull request (https://github.com/jenkinsci/ec2-plugin/pull/172) and rebuilded the ec2 pluging localy.
Now the plugin works fine with eu-central-1.
Br,
Jörg
Joerg Baldzer
added a comment - Hi,
I just included this pull request ( https://github.com/jenkinsci/ec2-plugin/pull/172 ) and rebuilded the ec2 pluging localy.
Now the plugin works fine with eu-central-1.
Br,
Jörg francisu, any update on the release schedule for 1.30? And will this issue be fixed in 1.30?
Brian L
added a comment - It would be really great to see a new release of this plugin. My team has seen a few issues working with the ec2-plugin, and we're hoping things will get smoother soon. On GitHub, there have been 14 commits to the master branch since the last release (August 2, 2015). One notable change (IMHO), is that the aws-java-sdk dependency has been upgraded from version 1.10.2 (used in the last release) to 1.10.26 (currently in the master branch).
Can someone please comment as to when an update is expected, or if another plugin should be preferred?
Brian L
added a comment - It would be really great to see a new release of this plugin. My team has seen a few issues working with the ec2-plugin, and we're hoping things will get smoother soon. On GitHub, there have been 14 commits to the master branch since the last release ( August 2, 2015 ). One notable change (IMHO), is that the aws-java-sdk dependency has been upgraded from version 1.10.2 ( used in the last release ) to 1.10.26 ( currently in the master branch ).
Can someone please comment as to when an update is expected, or if another plugin should be preferred? Francis Upton
added a comment - Unfortunately there are some regressions in the current master unreleased version. I'm going to try and address those in the next few days and once they are fixed and it seems reliable, I will do a new release.
Francis Upton
added a comment - Unfortunately there are some regressions in the current master unreleased version. I'm going to try and address those in the next few days and once they are fixed and it seems reliable, I will do a new release. 
Nico Schlömer
added a comment - I'd like to second bbl5660 here and hope that the regressions are smoothed out soon. The new release would save us a great deal of testing time if we can boots the slaves in the same data center as our master.
Nico Schlömer
added a comment - I'd like to second bbl5660 here and hope that the regressions are smoothed out soon. The new release would save us a great deal of testing time if we can boots the slaves in the same data center as our master. Francis Upton
added a comment - I have finished my work on 1.30 snapshot, I would like some additional testing before I make the release, which I can do any day now.
Francis Upton
added a comment - I have finished my work on 1.30 snapshot, I would like some additional testing before I make the release, which I can do any day now. Francis Upton
added a comment - Released in 1.30 
Because we have to start our Jenkins slaves in eu-central-1 I made the changes here.
I also had to upgrade the AWS SDK because the requests to eu-central need a Signature Version 4 (see Problem accessing eu-central-1 with API)
There seem to be some problems with newer versions of the AWS SDK so I'll have to investigate this before doing a pull request.
https://github.com/jenkinsci/ec2-plugin/pull/131
https://issues.jenkins-ci.org/browse/JENKINS-26854?focusedCommentId=221457&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-221457