Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33945

can't provision new nodes when a matching node is marked offline

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Duplicate
    • ec2-plugin
    • None
    • ec2 1.31

    Description

      You can not provision a new node if there is an existing node from the same SlaveTemplate which is marked offline. Offline nodes instance id will be returned and no new node will be provisioned.

      Attachments

        Issue Links

          Activity

            trbaker Trevor Baker added a comment - I provided PR 187 https://github.com/jenkinsci/ec2-plugin/pull/187

            Code changed in jenkins
            User: Trevor Baker
            Path:
            src/main/java/hudson/plugins/ec2/SlaveTemplate.java
            http://jenkins-ci.org/commit/ec2-plugin/8226bdb66579342b85766cc1eb12994e895f4685
            Log:
            JENKINS-33945 allow node provisioning when existing matching node is offline
            You can not provision a new node when an existing node from the template
            is marked offline. This change consults isOffline() to allow new node
            provisioning. Nodes can either be offline due to channel errors or user
            interaction. Either way, don't block new nodes.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Trevor Baker Path: src/main/java/hudson/plugins/ec2/SlaveTemplate.java http://jenkins-ci.org/commit/ec2-plugin/8226bdb66579342b85766cc1eb12994e895f4685 Log: JENKINS-33945 allow node provisioning when existing matching node is offline You can not provision a new node when an existing node from the template is marked offline. This change consults isOffline() to allow new node provisioning. Nodes can either be offline due to channel errors or user interaction. Either way, don't block new nodes.

            Code changed in jenkins
            User: Francis Upton IV
            Path:
            src/main/java/hudson/plugins/ec2/SlaveTemplate.java
            http://jenkins-ci.org/commit/ec2-plugin/1227f790c1049a24094ae503aab2ce5ea89cdf36
            Log:
            Merge pull request #187 from tjbaker/offline

            JENKINS-33945 allow node provisioning when matching node is offline

            Compare: https://github.com/jenkinsci/ec2-plugin/compare/e3bfb31528fa...1227f790c104

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Francis Upton IV Path: src/main/java/hudson/plugins/ec2/SlaveTemplate.java http://jenkins-ci.org/commit/ec2-plugin/1227f790c1049a24094ae503aab2ce5ea89cdf36 Log: Merge pull request #187 from tjbaker/offline JENKINS-33945 allow node provisioning when matching node is offline Compare: https://github.com/jenkinsci/ec2-plugin/compare/e3bfb31528fa...1227f790c104

            Code changed in jenkins
            User: Francis Upton IV
            Path:
            src/main/java/hudson/plugins/ec2/SlaveTemplate.java
            http://jenkins-ci.org/commit/ec2-plugin/7a3973e1129054029d5f260e6e4df2ddd17dcbaa
            Log:
            JENKINS-33945 Allow offline nodes to be provisions (back out change)

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Francis Upton IV Path: src/main/java/hudson/plugins/ec2/SlaveTemplate.java http://jenkins-ci.org/commit/ec2-plugin/7a3973e1129054029d5f260e6e4df2ddd17dcbaa Log: JENKINS-33945 Allow offline nodes to be provisions (back out change)
            francisu Francis Upton added a comment -

            trbaker Does this issue apply only to manually provisioning nodes? I think this is related to JENKINS-32690. I would like to fix this by ignoring offline nodes only in the manual provisioning case (as opposed to automatic provisioning due to workload). Would this be acceptable?

            francisu Francis Upton added a comment - trbaker Does this issue apply only to manually provisioning nodes? I think this is related to JENKINS-32690 . I would like to fix this by ignoring offline nodes only in the manual provisioning case (as opposed to automatic provisioning due to workload). Would this be acceptable?
            trbaker Trevor Baker added a comment -

            It is also a problem for nodes provisioned automatically via StandardStrategyImpl

            Create a 3x3 axis Multi-configuration job. Launch a single slave with the label that can run the job, then mark if offline. It won't start any new nodes and the job will be stuck.

            Mar 31, 2016 6:46:45 PM FINER hudson.slaves.NodeProvisioner
            Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6437ee5a provisioning strategy with state StrategyState{label=deployer, snapshot=LoadStatisticsSnapshot{definedExecutors=1, onlineExecutors=0, connectingExecutors=0, busyExecutors=0, idleExecutors=0, availableExecutors=0, queueLength=9}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0}
            Mar 31, 2016 6:46:45 PM FINE hudson.slaves.NodeProvisioner
            Excess workload 7.906 detected. (planned capacity=0,connecting capacity=0,Qlen=7.906,available=0.174&0,online=0,m=0.5)
            Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud
            Attempting provision, excess workload: 8
            Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud
            Counting current slaves:  All AMIS
            Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud
            Existing instance found: i-3a616ebe AMI: ami-ce2d26a4 Template: null
            Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud
            Counting current slaves:  AMI: ami-ce2d26a4
            Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud
            Existing instance found: i-3a616ebe AMI: ami-ce2d26a4 Template: Jenkins - Deployer Slave
            Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud
            Available Total Slaves: 2147483646 Available AMI slaves: 2147483646 AMI: ami-ce2d26a4 TemplateDesc: Jenkins - Deployer Slave
            Mar 31, 2016 6:46:45 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo
            Launching ami-ce2d26a4 for template Jenkins - Deployer Slave
            Mar 31, 2016 6:46:46 PM FINE hudson.plugins.ec2.SlaveTemplate
            Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-ce2d26a4]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-xxx]}, {Name: instance.group-id,Values: [sg-xxx]}, {Name: key-name,Values: [dev]}, {Name: instance-type,Values: [t2.small]}, {Name: tag:Name,Values: [DEV-JENKINSSLAVE]}, {Name: tag:ENV,Values: [DEV]}],}
            Mar 31, 2016 6:46:46 PM FINE hudson.plugins.ec2.SlaveTemplate
            checkInstance: {InstanceId: i-3a616ebe,ImageId: snip}
            Mar 31, 2016 6:46:46 PM FINE hudson.plugins.ec2.SlaveTemplate
            Found existing corresponding Jenkins slave: i-3a616ebe
            Mar 31, 2016 6:46:46 PM FINE hudson.plugins.ec2.SlaveTemplate
             true - Node has capacity - can use it
            Mar 31, 2016 6:46:46 PM FINE hudson.plugins.ec2.SlaveTemplate
            Found existing instance: {InstanceId: i-3a616ebe,snip}
            node: i-3a616ebe
            Mar 31, 2016 6:46:46 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo
            Found existing pending or running: running instance: {InstanceId: i-3a616ebe,snip}
            Mar 31, 2016 6:46:46 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo
            Using existing slave: i-3a616ebe
            

            Instance i-3a616ebe in the above example is the temporarily offline node.

            trbaker Trevor Baker added a comment - It is also a problem for nodes provisioned automatically via StandardStrategyImpl Create a 3x3 axis Multi-configuration job. Launch a single slave with the label that can run the job, then mark if offline. It won't start any new nodes and the job will be stuck. Mar 31, 2016 6:46:45 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6437ee5a provisioning strategy with state StrategyState{label=deployer, snapshot=LoadStatisticsSnapshot{definedExecutors=1, onlineExecutors=0, connectingExecutors=0, busyExecutors=0, idleExecutors=0, availableExecutors=0, queueLength=9}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Mar 31, 2016 6:46:45 PM FINE hudson.slaves.NodeProvisioner Excess workload 7.906 detected. (planned capacity=0,connecting capacity=0,Qlen=7.906,available=0.174&0,online=0,m=0.5) Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud Attempting provision, excess workload: 8 Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud Counting current slaves: All AMIS Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud Existing instance found: i-3a616ebe AMI: ami-ce2d26a4 Template: null Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud Counting current slaves: AMI: ami-ce2d26a4 Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud Existing instance found: i-3a616ebe AMI: ami-ce2d26a4 Template: Jenkins - Deployer Slave Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud Available Total Slaves: 2147483646 Available AMI slaves: 2147483646 AMI: ami-ce2d26a4 TemplateDesc: Jenkins - Deployer Slave Mar 31, 2016 6:46:45 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo Launching ami-ce2d26a4 for template Jenkins - Deployer Slave Mar 31, 2016 6:46:46 PM FINE hudson.plugins.ec2.SlaveTemplate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-ce2d26a4]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-xxx]}, {Name: instance.group-id,Values: [sg-xxx]}, {Name: key-name,Values: [dev]}, {Name: instance-type,Values: [t2.small]}, {Name: tag:Name,Values: [DEV-JENKINSSLAVE]}, {Name: tag:ENV,Values: [DEV]}],} Mar 31, 2016 6:46:46 PM FINE hudson.plugins.ec2.SlaveTemplate checkInstance: {InstanceId: i-3a616ebe,ImageId: snip} Mar 31, 2016 6:46:46 PM FINE hudson.plugins.ec2.SlaveTemplate Found existing corresponding Jenkins slave: i-3a616ebe Mar 31, 2016 6:46:46 PM FINE hudson.plugins.ec2.SlaveTemplate true - Node has capacity - can use it Mar 31, 2016 6:46:46 PM FINE hudson.plugins.ec2.SlaveTemplate Found existing instance: {InstanceId: i-3a616ebe,snip} node: i-3a616ebe Mar 31, 2016 6:46:46 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo Found existing pending or running: running instance: {InstanceId: i-3a616ebe,snip} Mar 31, 2016 6:46:46 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo Using existing slave: i-3a616ebe Instance i-3a616ebe in the above example is the temporarily offline node.

            People

              francisu Francis Upton
              trbaker Trevor Baker
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: