Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55556

EC2 unable to retrieve private IP (+ other buggy behaviour)

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • ec2-plugin
    • None
    • docker jenkins/jenkins:lts-slim
      ec2-plugin 1.42

      I must say any version > 1.39 seems ultra buggy for our use case (ec2 spot instances @ c4.xlarge):

       

      • when launching manually the plugin can not pick up the private IP and tries to connect to "null:22" endlessly
      • when launched "on-demand" the plugin will launch 10+ spot instances, non of them will work though

       

      Downgrading to 1.39 makes the plugin work on the exact same setup. Curious why AWS is not stepping in to give this plugin some love...

          [JENKINS-55556] EC2 unable to retrieve private IP (+ other buggy behaviour)

          Günter Grodotzki created issue -
          Günter Grodotzki made changes -
          Issue Type Original: New Feature [ 2 ] New: Bug [ 1 ]
          Günter Grodotzki made changes -
          Description Original: I must say any version > 1.39 seems ultra buggy for our use case (ec2 spot instances @ c4.xlarge):

           
           * when launching manually the plugin can not pick up the private IP and tries to connect to "null:22" endlessly
           * when launched "on-demand" the plugin will launch 10+ spot instances, non of them will work though

           

          Downgrading to 1.39 makes the plugin work on the exact same setup.
          New: I must say any version > 1.39 seems ultra buggy for our use case (ec2 spot instances @ c4.xlarge):

           
           * when launching manually the plugin can not pick up the private IP and tries to connect to "null:22" endlessly
           * when launched "on-demand" the plugin will launch 10+ spot instances, non of them will work though

           

          Downgrading to 1.39 makes the plugin work on the exact same setup. Curious why AWS is not stepping in to give this plugin some love...

          Hi,

          For the first problem, Can share more details on your configuration ?, how is your AWS network configuration , node configuration 

          In the 1.42 should not have changes on the connection ( in the next version there are some improvements)

          For the second problem, can you share any error message ? Did you update the IAM roles ?

          FABRIZIO MANFREDI added a comment - Hi, For the first problem, Can share more details on your configuration ?, how is your AWS network configuration , node configuration  In the 1.42 should not have changes on the connection ( in the next version there are some improvements) For the second problem, can you share any error message ? Did you update the IAM roles ?

          When downgrading from 1.42 to 1.39 everything works with the exact same setup (no changes, just downgrading by manually uploading the hpi).

          The second problem is a result of the first problem, but for some reason it would spin up 10+ instances almost instantly (could be that new setting "launch new instances right away"?) - the error was the same, that is unable to connect to "null:22"

          Setup:

          • VPC with private/public subnets - private subnets outgoing via natgw (https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/1.51.0)
          • jenkins nodes are launched of a custom ami (debian + openjdk8/encrypted boot) with their own custom instance-profile
          • jenkins nodes are launched in private subnet with no public ip - a security group that explicitly only gives the jenkins-master access via port 22

           

          Hope this info helps, else let me know  

          Günter Grodotzki added a comment - When downgrading from 1.42 to 1.39 everything works with the exact same setup (no changes, just downgrading by manually uploading the hpi). The second problem is a result of the first problem, but for some reason it would spin up 10+ instances almost instantly (could be that new setting "launch new instances right away"?) - the error was the same, that is unable to connect to "null:22" Setup: VPC with private/public subnets - private subnets outgoing via natgw ( https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/1.51.0) jenkins nodes are launched of a custom ami (debian + openjdk8/encrypted boot) with their own custom instance-profile jenkins nodes are launched in private subnet with no public ip - a security group that explicitly only gives the jenkins-master access via port 22   Hope this info helps, else let me know  

          A couple of questions :

          • did you update the IAM policy connected to the user/role used for raise new node ? (
            ...
            Effect": "Allow",
            "Action": [
            "iam:ListInstanceProfilesForRole",
            "iam:PassRole"
            ],)
          • Is Jenkins master with Public ip ? 
          • Jenkins master and slaves are in the same VPC  and subnet? 
          • What is the configuration of the ec-plugin (use public dns, ...), the option enabled?

           

          For the number of the nodes started, now the plugin is much more "reactive" to the status of the queue

           

          FABRIZIO MANFREDI added a comment - A couple of questions : did you update the IAM policy connected to the user/role used for raise new node ? ( ... Effect": "Allow", "Action": [ "iam:ListInstanceProfilesForRole", "iam:PassRole" ], ) Is Jenkins master with Public ip ?  Jenkins master and slaves are in the same VPC  and subnet?  What is the configuration of the ec-plugin (use public dns, ...), the option enabled?   For the number of the nodes started, now the plugin is much more "reactive" to the status of the queue  

          Joshua Spence added a comment - - edited

          Hitting this issue as well. I was able to work around it by not using spot instances.

          Joshua Spence added a comment - - edited Hitting this issue as well. I was able to work around it by not using spot instances.

          thoulen

          • yes, those IAM actions are in place
          • jenkins master is with private IP but reachable over a public IP behind an ALB
          • jenkins master and nodes are in the same VPC, private subnets, but might be in different subnet-ids depending on launched AZ
          • public-dns is not enabled

           

          joshuaspence might be correct, we are launching spot ec2 c4.xlarge - so might be an issue with spot instances

          Günter Grodotzki added a comment - thoulen yes, those IAM actions are in place jenkins master is with private IP but reachable over a public IP behind an ALB jenkins master and nodes are in the same VPC, private subnets, but might be in different subnet-ids depending on launched AZ public-dns is not enabled   joshuaspence might be correct, we are launching spot ec2 c4.xlarge - so might be an issue with spot instances

          Shaun Lawrie added a comment - - edited

          I have the same symptoms with spot instance private IP's not being identified by the ec2-plugin but ours also have public IP's assigned in the configuration.

          I downgraded to 1.39 so it behaves in the meantime.

          Shaun Lawrie added a comment - - edited I have the same symptoms with spot instance private IP's not being identified by the ec2-plugin but ours also have public IP's assigned in the configuration. I downgraded to 1.39 so it behaves in the meantime.

          Phuong Le added a comment -

          We are using spot instances too. Configuring master to connect to slaves using neither public IP or private IP works. Always

           

          Jan 21, 2019 6:41:58 AM hudson.plugins.ec2.EC2Cloud
          INFO: Failed to connect via ssh: There was a problem while connecting to null:22

           

          The plugin is indeed unusable for spot instance use case since 1.40, there was always different error in each version.

           

          Phuong Le added a comment - We are using spot instances too. Configuring master to connect to slaves using neither public IP or private IP works. Always   Jan 21, 2019 6:41:58 AM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: There was a problem while connecting to null:22   The plugin is indeed unusable for spot instance use case since 1.40, there was always different error in each version.  

            thoulen FABRIZIO MANFREDI
            lifeofguenter Günter Grodotzki
            Votes:
            4 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: