Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67190

EC2-plugin not spooling up stopped nodes, starting new nodes instead

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Blocker
    • Resolution: Unresolved
    • ec2-plugin
    • None
    • Jenkins 2.303.3, Ec2 plugin 1.66

    Description

      The Jenkins EC2 plugin isn't starting existing (stopped) nodes, instead, it always starts a new one. I can see the past instances as offline in the nodes tab, but instead of starting these nodes, the plugin always starts a new instance instead.

      Attachments

        Issue Links

          Activity

            Same issue on my instance, no error, no exception, no call is sent to AWS EC2 to initiate instance startup.

            The cap instance is not respected at global or AMI level.

            We have to force instances to be deleted to control the number of subnodes.

            It's a major issue as it lead to loss cost control.

             

            flocqueneux florian Locqueneux added a comment - Same issue on my instance, no error, no exception, no call is sent to AWS EC2 to initiate instance startup. The cap instance is not respected at global or AMI level. We have to force instances to be deleted to control the number of subnodes. It's a major issue as it lead to loss cost control.  
            cuong_you Cuong added a comment - - edited

            I also encountered this issue in my Jenkins setup.
            Can confirm that it is only on EC2 version 1.66. Downgrading to 1.65 and things start working.
            My bad. It was actually a rerun build that just use Jenkins scheduling to reuse the same node.
            Issue still persists in 1.65 when scheduling a new build.

            cuong_you Cuong added a comment - - edited I also encountered this issue in my Jenkins setup. Can confirm that it is only on EC2 version 1.66. Downgrading to 1.65 and things start working. My bad. It was actually a rerun build that just use Jenkins scheduling to reuse the same node. Issue still persists in 1.65 when scheduling a new build.

            greyjackal have you found a way to make it a reproducible situation? We experience almost the same but have not been able to consistently reproduce it. It seems to happen randomly? Also wondering for completeness what is your EC2 template configuration look like. Do you have additional arguments, boot delay, etc.

            matthias_glastra Matthias Glastra added a comment - greyjackal have you found a way to make it a reproducible situation? We experience almost the same but have not been able to consistently reproduce it. It seems to happen randomly? Also wondering for completeness what is your EC2 template configuration look like. Do you have additional arguments, boot delay, etc.
            sgdave David Drum added a comment - - edited

            I, too, am experiencing this problem, and may have something to contribute. In my first attempt, which resulted in the plugin launching a new instance, the logs show it looking for an instance that matched:

            hudson.plugins.ec2.SlaveTemplate#logProvisionInfo: SlaveTemplate{description='AWS Linux 2', labels='ec2'}. Looking for existing instances with describe-instance: {Filters: [{Name: image-id,Values: [ami-xxxxxxxxxxxxxxxxx]}, {Name: instance-type,Values: [c5.4xlarge]}, {Name: key-name,Values: [jenkins]}, {Name: availability-zone,Values: [us-east-2]}, {Name: tenancy,Values: [default]}, {Name: subnet-id,Values: [subnet-xxxxxxxx]}, {Name: tag:jenkins_slave_type,Values: [demand_AWS Linux 2]}, {Name: tag:jenkins_server_url,Values: [https://foo.com/jenkins/]}],InstanceIds: [],}
            

            I then logged in to the AWS EC2 console Instances dialog and began entering the filters above. The list of instances was fine until I added the availability-zone filter. AWS suggested us-east-2a, us-east-2b, and us-east-2c instead of plain us-east-2. The web dialog accepted us-east-2* and continued to list the appropriate instances, so I changed the plugin settings Availability Zone field to that value also. The second problematic filter was subnet-id, as in the plugin settings I have multiple subnets configured for round robin, but the existence of a specific subnet-id in the filter above would interfere with that. I removed the list of subnet IDs in the plugin. After making those two changes to the plugin settings, a subsequent build succeeded in finding and starting an existing instance:

            hudson.plugins.ec2.SlaveTemplate#logProvisionInfo: SlaveTemplate{description='AWS Linux 2', labels='ec2'}. Looking for existing instances with describe-instance: {Filters: [{Name: image-id,Values: [ami-xxxxxxxxxxxxxxxxx]}, {Name: instance-type,Values: [c5.4xlarge]}, {Name: key-name,Values: [jenkins]}, {Name: availability-zone,Values: [us-east-2*]}, {Name: tenancy,Values: [default]}, {Name: tag:jenkins_slave_type,Values: [demand_AWS Linux 2]}, {Name: tag:jenkins_server_url,Values: [https://mathkins.pfxdev.com/jenkins/]}],InstanceIds: [],}
            

            I hope this helps. Also, I'd much prefer subnet name to subnet ID, as that could be wildcarded.

             

            sgdave David Drum added a comment - - edited I, too, am experiencing this problem, and may have something to contribute. In my first attempt, which resulted in the plugin launching a new instance, the logs show it looking for an instance that matched: hudson.plugins.ec2.SlaveTemplate#logProvisionInfo: SlaveTemplate{description= 'AWS Linux 2' , labels= 'ec2' }. Looking for existing instances with describe-instance: {Filters: [{Name: image-id,Values: [ami-xxxxxxxxxxxxxxxxx]}, {Name: instance-type,Values: [c5.4xlarge]}, {Name: key-name,Values: [jenkins]}, {Name: availability-zone,Values: [us-east-2]}, {Name: tenancy,Values: [ default ]}, {Name: subnet-id,Values: [subnet-xxxxxxxx]}, {Name: tag:jenkins_slave_type,Values: [demand_AWS Linux 2]}, {Name: tag:jenkins_server_url,Values: [https: //foo.com/jenkins/]}],InstanceIds: [],} I then logged in to the AWS EC2 console Instances dialog and began entering the filters above. The list of instances was fine until I added the availability-zone filter. AWS suggested us-east-2a , us-east-2b , and us-east-2c instead of plain us-east-2 . The web dialog accepted us-east-2* and continued to list the appropriate instances, so I changed the plugin settings Availability Zone field to that value also. The second problematic filter was subnet-id , as in the plugin settings I have multiple subnets configured for round robin, but the existence of a specific subnet-id in the filter above would interfere with that. I removed the list of subnet IDs in the plugin. After making those two changes to the plugin settings, a subsequent build succeeded in finding and starting an existing instance: hudson.plugins.ec2.SlaveTemplate#logProvisionInfo: SlaveTemplate{description= 'AWS Linux 2' , labels= 'ec2' }. Looking for existing instances with describe-instance: {Filters: [{Name: image-id,Values: [ami-xxxxxxxxxxxxxxxxx]}, {Name: instance-type,Values: [c5.4xlarge]}, {Name: key-name,Values: [jenkins]}, {Name: availability-zone,Values: [us-east-2*]}, {Name: tenancy,Values: [ default ]}, {Name: tag:jenkins_slave_type,Values: [demand_AWS Linux 2]}, {Name: tag:jenkins_server_url,Values: [https: //mathkins.pfxdev.com/jenkins/]}],InstanceIds: [],} I hope this helps. Also, I'd much prefer subnet name to subnet ID, as that could be wildcarded.  

            Possibly related to JENKINS-64520 "EC2 node not start after stop/disconnect with parameter Idle termination time"?

            mwebber Matthew Webber added a comment - Possibly related to JENKINS-64520 "EC2 node not start after stop/disconnect with parameter Idle termination time"?

            People

              Unassigned Unassigned
              greyjackal Bruno Esteves
              Votes:
              4 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated: