Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-15081

Errors with multiple EC2 Clouds

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Blocker
    • Resolution: Fixed
    • ec2-plugin
    • None

    Description

      If multiple EC2 clouds are defined in the plugin, incorrect credentials seem to be used to issue commands for nodes in the second defined cloud. Instances for the second cloud can be created, but the Jenkins agent fails to connect, which results in a new node being created for each failed attempt. Once created, the nodes cannot be deleted until the first cloud is deleted.

      Attachments

        Activity

          The initial creation failure may be a result of not having the correct SSH key installed in the AMI. Investigating that now. The deletion problem is definitely real though.

          qhartman Quentin Hartman added a comment - The initial creation failure may be a result of not having the correct SSH key installed in the AMI. Investigating that now. The deletion problem is definitely real though.

          Nope, the initial creation failure was not an SSH key problem. It seems that if you have multiple EC2 clouds configured, the settings for the first cloud in the list are what get used when instantiating or deleting instances, regardless of which cloud they actually belong to. The initial step of telling AWS to start a new instance works correctly, but all the steps after that fail because of credential problems.

          qhartman Quentin Hartman added a comment - Nope, the initial creation failure was not an SSH key problem. It seems that if you have multiple EC2 clouds configured, the settings for the first cloud in the list are what get used when instantiating or deleting instances, regardless of which cloud they actually belong to. The initial step of telling AWS to start a new instance works correctly, but all the steps after that fail because of credential problems.
          rsennewald Ray Sennewald added a comment -

          I'm seeing this same issue. This creates a problem for me because I have instances on different zones (us-east and us-west2 for example) but because of this am limited to only one zone per Jenkins server. Any idea on when we may have a fix for this?

          rsennewald Ray Sennewald added a comment - I'm seeing this same issue. This creates a problem for me because I have instances on different zones (us-east and us-west2 for example) but because of this am limited to only one zone per Jenkins server. Any idea on when we may have a fix for this?
          rsennewald Ray Sennewald added a comment -

          I have a fix for this and I have a pull request open to get this pulled in to the main ec2-plugin repo. If your interested you can view the PR here: https://github.com/jenkinsci/ec2-plugin/pull/46 .

          Basically what I found is previously the plugin was typically retrieving the first EC2Cloud instead of the EC2Cloud that the specific slave belongs to. This is what I've addressed in my changes. I tested it and confirmed with two different EC2Clouds added and two templates (one from each cloud), that I could properly add and delete instances from each template, which I previously couldn't do with any templates from the second cloud.

          rsennewald Ray Sennewald added a comment - I have a fix for this and I have a pull request open to get this pulled in to the main ec2-plugin repo. If your interested you can view the PR here: https://github.com/jenkinsci/ec2-plugin/pull/46 . Basically what I found is previously the plugin was typically retrieving the first EC2Cloud instead of the EC2Cloud that the specific slave belongs to. This is what I've addressed in my changes. I tested it and confirmed with two different EC2Clouds added and two templates (one from each cloud), that I could properly add and delete instances from each template, which I previously couldn't do with any templates from the second cloud.

          Awesome. I'll check that out as soon as I'm able. I'm kinda buried at the moment, so Francis might beat me to it.

          qhartman Quentin Hartman added a comment - Awesome. I'll check that out as soon as I'm able. I'm kinda buried at the moment, so Francis might beat me to it.
          rsennewald Ray Sennewald added a comment -

          Francis got to it and asked me to pull in his recent updates for spot instances. So I've closed my original PR, merged in his code, revised my modifications and issued a new PR: https://github.com/jenkinsci/ec2-plugin/pull/48 .

          rsennewald Ray Sennewald added a comment - Francis got to it and asked me to pull in his recent updates for spot instances. So I've closed my original PR, merged in his code, revised my modifications and issued a new PR: https://github.com/jenkinsci/ec2-plugin/pull/48 .

          Code changed in jenkins
          User: ray.sennewald@gmail.com
          Path:
          src/main/java/hudson/plugins/ec2/EC2AbstractSlave.java
          src/main/java/hudson/plugins/ec2/EC2Computer.java
          src/main/java/hudson/plugins/ec2/EC2ComputerLauncher.java
          src/main/java/hudson/plugins/ec2/EC2OndemandSlave.java
          src/main/java/hudson/plugins/ec2/EC2SpotSlave.java
          src/main/java/hudson/plugins/ec2/SlaveTemplate.java
          src/main/java/hudson/plugins/ec2/ssh/EC2UnixLauncher.java
          http://jenkins-ci.org/commit/ec2-plugin/ec7e8e17979e60c4af631474611b1144a48b825f
          Log:
          Fix for JENKINS-15081. We store the EC2Cloud that a slave belongs to as an attribute in the EC2AbstractSlave class and have a function call to retrieve it in EC2Computer.

          All references to EC2Cloud.get changed to EC2AbstractSlave.cloud or EC2Computer.getCloud().
          This fix allows for us to always know the correct Cloud that an EC2AbstractSlave or EC2Computer belongs to so we can have multiple EC2Clouds configured on one Jenkins instance and have different templates for different clouds and have the instances for each spin up properly.

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: ray.sennewald@gmail.com Path: src/main/java/hudson/plugins/ec2/EC2AbstractSlave.java src/main/java/hudson/plugins/ec2/EC2Computer.java src/main/java/hudson/plugins/ec2/EC2ComputerLauncher.java src/main/java/hudson/plugins/ec2/EC2OndemandSlave.java src/main/java/hudson/plugins/ec2/EC2SpotSlave.java src/main/java/hudson/plugins/ec2/SlaveTemplate.java src/main/java/hudson/plugins/ec2/ssh/EC2UnixLauncher.java http://jenkins-ci.org/commit/ec2-plugin/ec7e8e17979e60c4af631474611b1144a48b825f Log: Fix for JENKINS-15081 . We store the EC2Cloud that a slave belongs to as an attribute in the EC2AbstractSlave class and have a function call to retrieve it in EC2Computer. All references to EC2Cloud.get changed to EC2AbstractSlave.cloud or EC2Computer.getCloud(). This fix allows for us to always know the correct Cloud that an EC2AbstractSlave or EC2Computer belongs to so we can have multiple EC2Clouds configured on one Jenkins instance and have different templates for different clouds and have the instances for each spin up properly.

          Code changed in jenkins
          User: Francis Upton
          Path:
          src/main/java/hudson/plugins/ec2/EC2AbstractSlave.java
          src/main/java/hudson/plugins/ec2/EC2Computer.java
          src/main/java/hudson/plugins/ec2/EC2ComputerLauncher.java
          src/main/java/hudson/plugins/ec2/EC2OndemandSlave.java
          src/main/java/hudson/plugins/ec2/EC2SpotSlave.java
          src/main/java/hudson/plugins/ec2/SlaveTemplate.java
          src/main/java/hudson/plugins/ec2/ssh/EC2UnixLauncher.java
          http://jenkins-ci.org/commit/ec2-plugin/5da8f6be66e839e33923cbf5e8b62b0f6c4b7bfc
          Log:
          Merge pull request #48 from rsennewald/jenkins-15081

          Fix to allow multiple EC2Clouds (fix for JENKINS-15081)

          Compare: https://github.com/jenkinsci/ec2-plugin/compare/881b2b80e488...5da8f6be66e8

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Francis Upton Path: src/main/java/hudson/plugins/ec2/EC2AbstractSlave.java src/main/java/hudson/plugins/ec2/EC2Computer.java src/main/java/hudson/plugins/ec2/EC2ComputerLauncher.java src/main/java/hudson/plugins/ec2/EC2OndemandSlave.java src/main/java/hudson/plugins/ec2/EC2SpotSlave.java src/main/java/hudson/plugins/ec2/SlaveTemplate.java src/main/java/hudson/plugins/ec2/ssh/EC2UnixLauncher.java http://jenkins-ci.org/commit/ec2-plugin/5da8f6be66e839e33923cbf5e8b62b0f6c4b7bfc Log: Merge pull request #48 from rsennewald/jenkins-15081 Fix to allow multiple EC2Clouds (fix for JENKINS-15081 ) Compare: https://github.com/jenkinsci/ec2-plugin/compare/881b2b80e488...5da8f6be66e8
          francisu Francis Upton added a comment -

          There remain problems with multiple cloud support that will be shortly addressed.

          francisu Francis Upton added a comment - There remain problems with multiple cloud support that will be shortly addressed.
          francisu Francis Upton added a comment -

          Just checked in fixes that should make this work properly. I tested it with multiple clouds and both spot and demand instances. Also clarified that the cloud has to be uniquely identified by the region (and made the UI such that it would not allow duplicate EC2 clouds to be created).

          francisu Francis Upton added a comment - Just checked in fixes that should make this work properly. I tested it with multiple clouds and both spot and demand instances. Also clarified that the cloud has to be uniquely identified by the region (and made the UI such that it would not allow duplicate EC2 clouds to be created).
          francisu Francis Upton added a comment -

          Here is the commit that had the remaining fixes. Note the bug report number in the commit message is incorrect:

          https://github.com/jenkinsci/ec2-plugin/commit/9095d318589809ea69cd7e1ca9e1ff94012794d6

          francisu Francis Upton added a comment - Here is the commit that had the remaining fixes. Note the bug report number in the commit message is incorrect: https://github.com/jenkinsci/ec2-plugin/commit/9095d318589809ea69cd7e1ca9e1ff94012794d6
          svmkphani Murali Phani added a comment -

          Hi,

          I am facing similar issue with EC2 1.29 as well.When two clouds are configured in EC2,the second one taking the first key to connect and failing..Please help which version has fix for this issue

          svmkphani Murali Phani added a comment - Hi, I am facing similar issue with EC2 1.29 as well.When two clouds are configured in EC2,the second one taking the first key to connect and failing..Please help which version has fix for this issue

          People

            francisu Francis Upton
            qhartman Quentin Hartman
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: