-
Improvement
-
Resolution: Fixed
-
Major
-
Jenkins 1.525, Amazon EC2 plugin v1.19-unreleased (last commit: #1e009adfa3)
From the help message in EC2Cloud configuration, I need to let spot instance notify the Jenkins master that this slave is available.
As my AMI is based on CentOS, I cannot take the script directly as it mentioned. So I have gone through the script and here is my understanding on the callback process.
Jenkins master does the spot-instance request also with a default parameter in instance userdata, like:
JENKINS_URL=http://1.1.1.1:8080/&SLAVE_NAME=XXXXXXX&USER_DATA=
Then the callback script will first download the "slave.jar" and use it to notify Jenkins Master that this slave is alive:
java -jar slave.jar -jnlpUrl http://1.1.1.1:8080/XXXXXXX/slave-agent.jnlp
However, due to my Jenkins master deployed internally with no public access, Jenkins master will keep considering this slave not ready.
I also try to run these commands on an internal machine and wish Master could let it go. But only get the "403 forbidden" response.
Could you please help me on this issue?
I feel this deployment is also quite common then.
Thanks
Henry
[JENKINS-19059] [ec2-plugin] spot instance feature cannot be used when the master deployed internally (no public access)
Adam Irr
added a comment - As for your internal machine and 403, I assume you require a login for Jenkins. This means that the slave will not have access to the slave.jar. To get passed the 403 you can either, enable anonymous read access, which will allow anyone to access the slave.jar (should only be considered if running on a trusted closed network) or add authentication to your request using the -jnlpCredential parameter.
I have not tried it, but this post seems to show how to use the -jnlpCredential parameter. http://stackoverflow.com/questions/16880110/login-to-jenkins-via-url
Adam Irr
added a comment - As for your internal machine and 403, I assume you require a login for Jenkins. This means that the slave will not have access to the slave.jar. To get passed the 403 you can either, enable anonymous read access, which will allow anyone to access the slave.jar (should only be considered if running on a trusted closed network) or add authentication to your request using the -jnlpCredential parameter.
I have not tried it, but this post seems to show how to use the -jnlpCredential parameter. http://stackoverflow.com/questions/16880110/login-to-jenkins-via-url 
Kevin Fleming
added a comment - As Adam says, there is no elegant solution to this problem. Following the same method as On-demand instances is not practical either, as the master won't know the IP address of the slave until the spot instance request is fulfilled. At best, it would have to poll the spot instance request itself until it has been fulfilled and an instance ID has been created, then query for the IP address of that instance so it can start trying to connect using SSH.
Kevin Fleming
added a comment - As Adam says, there is no elegant solution to this problem. Following the same method as On-demand instances is not practical either, as the master won't know the IP address of the slave until the spot instance request is fulfilled. At best, it would have to poll the spot instance request itself until it has been fulfilled and an instance ID has been created, then query for the IP address of that instance so it can start trying to connect using SSH. 
crawfish101
added a comment - I actually like the idea of polling the spot instance and looking for an instance id associated with a fulfilled request. The requirement of forcing the Jenkins master to be publicly accessible is a serious limitation. This new polling mechanism could be a separate user configurable option, but if was actually working as expected then I don't know why a user would want to use the current inbound connection approach.
crawfish101
added a comment - I actually like the idea of polling the spot instance and looking for an instance id associated with a fulfilled request. The requirement of forcing the Jenkins master to be publicly accessible is a serious limitation. This new polling mechanism could be a separate user configurable option, but if was actually working as expected then I don't know why a user would want to use the current inbound connection approach. I have fixed this plugin to use with a master deployed internally and run it for over a year.
A branch which includes my fix is https://github.com/ikikko/ec2-plugin/tree/JENKINS-19059/ssh-launcher-for-spot-instance .
If there are some people who are interested in this fix and want to use it, I try to send it as a pull request.
Francis Upton
added a comment - @Tomonari - I made your branch into a pull request. I will refine this to make it an option.
Francis Upton
added a comment - @Tomonari - I made your branch into a pull request. I will refine this to make it an option. Code changed in jenkins
User: Michael Barrientos
Path:
src/main/java/hudson/plugins/ec2/EC2AbstractSlave.java
src/main/java/hudson/plugins/ec2/EC2ComputerLauncher.java
src/main/java/hudson/plugins/ec2/EC2SpotComputerLauncher.java
src/main/java/hudson/plugins/ec2/EC2SpotRetentionStrategy.java
src/main/java/hudson/plugins/ec2/EC2SpotSlave.java
src/main/java/hudson/plugins/ec2/SlaveTemplate.java
src/main/resources/hudson/plugins/ec2/SlaveTemplate/config.jelly
http://jenkins-ci.org/commit/ec2-plugin/ae7e53ab99028fd8ede1a3e41907812c2edb0eec
Log:
Poll for spot instances instead of JNLP launcher JENKINS-27529 JENKINS-19059
- No more need for special spot startup script.
- Changes spot slave name to use spot request ID (partially fixes
JENKINS-32399)
Code changed in jenkins
User: Francis Upton IV
Path:
src/main/java/hudson/plugins/ec2/EC2AbstractSlave.java
src/main/java/hudson/plugins/ec2/EC2Cloud.java
src/main/java/hudson/plugins/ec2/EC2ComputerLauncher.java
src/main/java/hudson/plugins/ec2/EC2SpotComputerLauncher.java
src/main/java/hudson/plugins/ec2/EC2SpotRetentionStrategy.java
src/main/java/hudson/plugins/ec2/EC2SpotSlave.java
src/main/java/hudson/plugins/ec2/SlaveTemplate.java
src/main/resources/hudson/plugins/ec2/SlaveTemplate/config.jelly
http://jenkins-ci.org/commit/ec2-plugin/e3bfb31528faba3dd23bef940d1a4a8fa2a227c3
Log:
Merge pull request #179 from syapse/spot-ssh-rebase
Poll for spot instances instead of JNLP launcher JENKINS-27529 JENKINS-19059
Compare: https://github.com/jenkinsci/ec2-plugin/compare/cc23601d9702...e3bfb31528fa
It sounds like your configuration is set up correctly. As you state, the issue is your non-public Jenkins master.
Since jnlp requires the slave to connect to the master, it will not be possible for the instance to find the master and connect to it. Since currently the only way to launch Spot instances through the plugin is with jnlp, I don't think this will be easily fixable.
The alternative is to give an option to launch the Spot instances in the same manner as On demand instance in which the Jenkins master connects to the slave. This would result in Jenkins polling EC2 frequently on the initial start up of a Spot instance to determine when there is a usable instance. Also if a persistent bid is used, and the Spot instance was terminated, the instance might eventually restart, but Jenkins would not know about it.