Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19059

[ec2-plugin] spot instance feature cannot be used when the master deployed internally (no public access)

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      From the help message in EC2Cloud configuration, I need to let spot instance notify the Jenkins master that this slave is available.
      As my AMI is based on CentOS, I cannot take the script directly as it mentioned. So I have gone through the script and here is my understanding on the callback process.

      Jenkins master does the spot-instance request also with a default parameter in instance userdata, like:

      JENKINS_URL=http://1.1.1.1:8080/&SLAVE_NAME=XXXXXXX&USER_DATA=

      Then the callback script will first download the "slave.jar" and use it to notify Jenkins Master that this slave is alive:

      java -jar slave.jar -jnlpUrl http://1.1.1.1:8080/XXXXXXX/slave-agent.jnlp

      However, due to my Jenkins master deployed internally with no public access, Jenkins master will keep considering this slave not ready.

      I also try to run these commands on an internal machine and wish Master could let it go. But only get the "403 forbidden" response.

      Could you please help me on this issue?
      I feel this deployment is also quite common then.

      Thanks
      Henry

        Attachments

          Activity

          Hide
          aji9861 Adam Irr added a comment -

          It sounds like your configuration is set up correctly. As you state, the issue is your non-public Jenkins master.

          Since jnlp requires the slave to connect to the master, it will not be possible for the instance to find the master and connect to it. Since currently the only way to launch Spot instances through the plugin is with jnlp, I don't think this will be easily fixable.

          The alternative is to give an option to launch the Spot instances in the same manner as On demand instance in which the Jenkins master connects to the slave. This would result in Jenkins polling EC2 frequently on the initial start up of a Spot instance to determine when there is a usable instance. Also if a persistent bid is used, and the Spot instance was terminated, the instance might eventually restart, but Jenkins would not know about it.

          Show
          aji9861 Adam Irr added a comment - It sounds like your configuration is set up correctly. As you state, the issue is your non-public Jenkins master. Since jnlp requires the slave to connect to the master, it will not be possible for the instance to find the master and connect to it. Since currently the only way to launch Spot instances through the plugin is with jnlp, I don't think this will be easily fixable. The alternative is to give an option to launch the Spot instances in the same manner as On demand instance in which the Jenkins master connects to the slave. This would result in Jenkins polling EC2 frequently on the initial start up of a Spot instance to determine when there is a usable instance. Also if a persistent bid is used, and the Spot instance was terminated, the instance might eventually restart, but Jenkins would not know about it.
          Hide
          aji9861 Adam Irr added a comment -

          As for your internal machine and 403, I assume you require a login for Jenkins. This means that the slave will not have access to the slave.jar. To get passed the 403 you can either, enable anonymous read access, which will allow anyone to access the slave.jar (should only be considered if running on a trusted closed network) or add authentication to your request using the -jnlpCredential parameter.

          I have not tried it, but this post seems to show how to use the -jnlpCredential parameter. http://stackoverflow.com/questions/16880110/login-to-jenkins-via-url

          Show
          aji9861 Adam Irr added a comment - As for your internal machine and 403, I assume you require a login for Jenkins. This means that the slave will not have access to the slave.jar. To get passed the 403 you can either, enable anonymous read access, which will allow anyone to access the slave.jar (should only be considered if running on a trusted closed network) or add authentication to your request using the -jnlpCredential parameter. I have not tried it, but this post seems to show how to use the -jnlpCredential parameter. http://stackoverflow.com/questions/16880110/login-to-jenkins-via-url
          Hide
          kpfleming Kevin Fleming added a comment -

          As Adam says, there is no elegant solution to this problem. Following the same method as On-demand instances is not practical either, as the master won't know the IP address of the slave until the spot instance request is fulfilled. At best, it would have to poll the spot instance request itself until it has been fulfilled and an instance ID has been created, then query for the IP address of that instance so it can start trying to connect using SSH.

          Show
          kpfleming Kevin Fleming added a comment - As Adam says, there is no elegant solution to this problem. Following the same method as On-demand instances is not practical either, as the master won't know the IP address of the slave until the spot instance request is fulfilled. At best, it would have to poll the spot instance request itself until it has been fulfilled and an instance ID has been created, then query for the IP address of that instance so it can start trying to connect using SSH.
          Hide
          crawfish101 crawfish101 added a comment -

          I actually like the idea of polling the spot instance and looking for an instance id associated with a fulfilled request. The requirement of forcing the Jenkins master to be publicly accessible is a serious limitation. This new polling mechanism could be a separate user configurable option, but if was actually working as expected then I don't know why a user would want to use the current inbound connection approach.

          Show
          crawfish101 crawfish101 added a comment - I actually like the idea of polling the spot instance and looking for an instance id associated with a fulfilled request. The requirement of forcing the Jenkins master to be publicly accessible is a serious limitation. This new polling mechanism could be a separate user configurable option, but if was actually working as expected then I don't know why a user would want to use the current inbound connection approach.
          Hide
          ikikko Tomonari Nakamura added a comment -

          I have fixed this plugin to use with a master deployed internally and run it for over a year.
          A branch which includes my fix is https://github.com/ikikko/ec2-plugin/tree/JENKINS-19059/ssh-launcher-for-spot-instance .

          If there are some people who are interested in this fix and want to use it, I try to send it as a pull request.

          Show
          ikikko Tomonari Nakamura added a comment - I have fixed this plugin to use with a master deployed internally and run it for over a year. A branch which includes my fix is https://github.com/ikikko/ec2-plugin/tree/JENKINS-19059/ssh-launcher-for-spot-instance . If there are some people who are interested in this fix and want to use it, I try to send it as a pull request.
          Hide
          francisu Francis Upton added a comment -

          @Tomonari - I made your branch into a pull request. I will refine this to make it an option.

          Show
          francisu Francis Upton added a comment - @Tomonari - I made your branch into a pull request. I will refine this to make it an option.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Michael Barrientos
          Path:
          src/main/java/hudson/plugins/ec2/EC2AbstractSlave.java
          src/main/java/hudson/plugins/ec2/EC2ComputerLauncher.java
          src/main/java/hudson/plugins/ec2/EC2SpotComputerLauncher.java
          src/main/java/hudson/plugins/ec2/EC2SpotRetentionStrategy.java
          src/main/java/hudson/plugins/ec2/EC2SpotSlave.java
          src/main/java/hudson/plugins/ec2/SlaveTemplate.java
          src/main/resources/hudson/plugins/ec2/SlaveTemplate/config.jelly
          http://jenkins-ci.org/commit/ec2-plugin/ae7e53ab99028fd8ede1a3e41907812c2edb0eec
          Log:
          Poll for spot instances instead of JNLP launcher JENKINS-27529 JENKINS-19059

          • No more need for special spot startup script.
          • Changes spot slave name to use spot request ID (partially fixes JENKINS-32399)
          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Michael Barrientos Path: src/main/java/hudson/plugins/ec2/EC2AbstractSlave.java src/main/java/hudson/plugins/ec2/EC2ComputerLauncher.java src/main/java/hudson/plugins/ec2/EC2SpotComputerLauncher.java src/main/java/hudson/plugins/ec2/EC2SpotRetentionStrategy.java src/main/java/hudson/plugins/ec2/EC2SpotSlave.java src/main/java/hudson/plugins/ec2/SlaveTemplate.java src/main/resources/hudson/plugins/ec2/SlaveTemplate/config.jelly http://jenkins-ci.org/commit/ec2-plugin/ae7e53ab99028fd8ede1a3e41907812c2edb0eec Log: Poll for spot instances instead of JNLP launcher JENKINS-27529 JENKINS-19059 No more need for special spot startup script. Changes spot slave name to use spot request ID (partially fixes JENKINS-32399 )
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Francis Upton IV
          Path:
          src/main/java/hudson/plugins/ec2/EC2AbstractSlave.java
          src/main/java/hudson/plugins/ec2/EC2Cloud.java
          src/main/java/hudson/plugins/ec2/EC2ComputerLauncher.java
          src/main/java/hudson/plugins/ec2/EC2SpotComputerLauncher.java
          src/main/java/hudson/plugins/ec2/EC2SpotRetentionStrategy.java
          src/main/java/hudson/plugins/ec2/EC2SpotSlave.java
          src/main/java/hudson/plugins/ec2/SlaveTemplate.java
          src/main/resources/hudson/plugins/ec2/SlaveTemplate/config.jelly
          http://jenkins-ci.org/commit/ec2-plugin/e3bfb31528faba3dd23bef940d1a4a8fa2a227c3
          Log:
          Merge pull request #179 from syapse/spot-ssh-rebase

          Poll for spot instances instead of JNLP launcher JENKINS-27529 JENKINS-19059

          Compare: https://github.com/jenkinsci/ec2-plugin/compare/cc23601d9702...e3bfb31528fa

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Francis Upton IV Path: src/main/java/hudson/plugins/ec2/EC2AbstractSlave.java src/main/java/hudson/plugins/ec2/EC2Cloud.java src/main/java/hudson/plugins/ec2/EC2ComputerLauncher.java src/main/java/hudson/plugins/ec2/EC2SpotComputerLauncher.java src/main/java/hudson/plugins/ec2/EC2SpotRetentionStrategy.java src/main/java/hudson/plugins/ec2/EC2SpotSlave.java src/main/java/hudson/plugins/ec2/SlaveTemplate.java src/main/resources/hudson/plugins/ec2/SlaveTemplate/config.jelly http://jenkins-ci.org/commit/ec2-plugin/e3bfb31528faba3dd23bef940d1a4a8fa2a227c3 Log: Merge pull request #179 from syapse/spot-ssh-rebase Poll for spot instances instead of JNLP launcher JENKINS-27529 JENKINS-19059 Compare: https://github.com/jenkinsci/ec2-plugin/compare/cc23601d9702...e3bfb31528fa

            People

            Assignee:
            francisu Francis Upton
            Reporter:
            henrysher Henry Huang
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: