-
Bug
-
Resolution: Unresolved
-
Minor
-
Jenkins: 2.204.1
Amazon EC2 plugin (ID: ec2): 1.47
We have two templates for the ec2 plugin:
template1
label: test default
instance cap: 1
usage: NORMAL
template2 label: report instance cap: 3 usage: EXCLUSIVE
Jenkins doesn't create a new instance with template 2 when there are jobs in queue waiting for an available excutor on a running instance created with template 1, for example in our queue:
1. test_job (Restrict where this project can be run unchecked. Waiting for template1:test as all exectuors on template1:test are taken and we only want 1 instance with template1:test)
2. report_job (Restrict where this project can be run checked. Label expression: report. Waiting for template2:report as it does not exist)
report_job will be blocked until test_job is scheduled as Jenkins keeps trying to schedule test_job and ignores the request of an instance with template2 from report_job
UPDATE: I can confirm that this only happens when the first job in the queue has "Restrict where this project can be run" unchecked.
And I believe the problem is from Jenkins core not the ec2 plugin as the NodeProvisioner doesn't not look for all the labels as it should be (https://github.com/jenkinsci/jenkins/blob/jenkins-2.204.1/core/src/main/java/hudson/slaves/NodeProvisioner.java#L815)
Here is the log from hudson.slaves.NodeProvisioner class
Jan 16, 2020 9:38:08 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.plugins.ec2.NoDelayProvisionerStrategy@1cfb55fa provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:09 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6a13a6e5 provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:09 PM FINE hudson.slaves.NodeProvisioner Excess workload 117.81 detected. (planned capacity=0,connecting capacity=0,Qlen=117.81,available=0&0,online=6,m=0.106) Jan 16, 2020 9:38:18 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.plugins.ec2.NoDelayProvisionerStrategy@1cfb55fa provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:18 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6a13a6e5 provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:18 PM FINE hudson.slaves.NodeProvisioner Excess workload 118.729 detected. (planned capacity=0,connecting capacity=0,Qlen=118.729,available=0&0,online=6,m=0.106)
[JENKINS-60657] EC2 plugins with multiple templates: fails to start an instance for job in the middle of the queue
Description |
Original:
We have two templates for the ec2 plugin: {code:java} template1 label: test instance cap: 1 {code} {code:java} template2 label: report instance cap: 3 {code} Jenkins doesn't create a new instance with template 2 when there are jobs in queue waiting for an available excutor on a running instance created with template 1, for example in our queue: 1. test_job (waiting for _template1:test_ as all exectuors on _template1:test_ are taken and we only want 1 instance with _template1:test_) 2. report_job (waiting for _template2:report_ as it does not exist) Job2 will be blocked until job1 is scheduled as Jenkins keeps trying to schedule job1 and ignores the request of an instance with _template2_ from job2 |
New:
We have two templates for the ec2 plugin: {code:java} template1 label: test instance cap: 1 {code} {code:java} template2 label: report instance cap: 3 {code} Jenkins doesn't create a new instance with template 2 when there are jobs in queue waiting for an available excutor on a running instance created with template 1, for example in our queue: 1. test_job (waiting for _template1:test_ as all exectuors on _template1:test_ are taken and we only want 1 instance with _template1:test_) 2. report_job (waiting for _template2:report_ as it does not exist) report_job will be blocked until test_job is scheduled as Jenkins keeps trying to schedule test_job and ignores the request of an instance with _template2_ from report_job |
Description |
Original:
We have two templates for the ec2 plugin: {code:java} template1 label: test instance cap: 1 {code} {code:java} template2 label: report instance cap: 3 {code} Jenkins doesn't create a new instance with template 2 when there are jobs in queue waiting for an available excutor on a running instance created with template 1, for example in our queue: 1. test_job (waiting for _template1:test_ as all exectuors on _template1:test_ are taken and we only want 1 instance with _template1:test_) 2. report_job (waiting for _template2:report_ as it does not exist) report_job will be blocked until test_job is scheduled as Jenkins keeps trying to schedule test_job and ignores the request of an instance with _template2_ from report_job |
New:
We have two templates for the ec2 plugin: {code:java} template1 label: test instance cap: 1 {code} {code:java} template2 label: report instance cap: 3 {code} Jenkins doesn't create a new instance with template 2 when there are jobs in queue waiting for an available excutor on a running instance created with template 1, for example in our queue: 1. test_job (waiting for _template1:test_ as all exectuors on _template1:test_ are taken and we only want 1 instance with _template1:test_) 2. report_job (waiting for _template2:report_ as it does not exist) report_job will be blocked until test_job is scheduled as Jenkins keeps trying to schedule test_job and ignores the request of an instance with _template2_ from report_job |
Description |
Original:
We have two templates for the ec2 plugin: {code:java} template1 label: test instance cap: 1 {code} {code:java} template2 label: report instance cap: 3 {code} Jenkins doesn't create a new instance with template 2 when there are jobs in queue waiting for an available excutor on a running instance created with template 1, for example in our queue: 1. test_job (waiting for _template1:test_ as all exectuors on _template1:test_ are taken and we only want 1 instance with _template1:test_) 2. report_job (waiting for _template2:report_ as it does not exist) report_job will be blocked until test_job is scheduled as Jenkins keeps trying to schedule test_job and ignores the request of an instance with _template2_ from report_job |
New:
We have two templates for the ec2 plugin: {code:java} template1 label: test instance cap: 1 {code} {code:java} template2 label: report instance cap: 3 {code} Jenkins doesn't create a new instance with template 2 when there are jobs in queue waiting for an available excutor on a running instance created with template 1, for example in our queue: 1. test_job (waiting for _template1:test_ as all exectuors on _template1:test_ are taken and we only want 1 instance with _template1:test_) 2. report_job (waiting for _template2:report_ as it does not exist) report_job will be blocked until test_job is scheduled as Jenkins keeps trying to schedule test_job and ignores the request of an instance with _template2_ from report_job Seems like this only happens when there are a lot jobs in the queue (I don't know a definite number.. but I don't experience this issue when there are only two jobs in the queue) And I believe the problem is from Jenkins core not the ec2 plugin as the NodeProvisioner doesn't not look for all the labels. Here is the log from hudson.slaves.NodeProvisioner class {code:java} Jan 16, 2020 9:38:08 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.plugins.ec2.NoDelayProvisionerStrategy@1cfb55fa provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:09 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6a13a6e5 provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:09 PM FINE hudson.slaves.NodeProvisioner Excess workload 117.81 detected. (planned capacity=0,connecting capacity=0,Qlen=117.81,available=0&0,online=6,m=0.106) Jan 16, 2020 9:38:18 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.plugins.ec2.NoDelayProvisionerStrategy@1cfb55fa provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:18 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6a13a6e5 provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:18 PM FINE hudson.slaves.NodeProvisioner Excess workload 118.729 detected. (planned capacity=0,connecting capacity=0,Qlen=118.729,available=0&0,online=6,m=0.106) {code} |
Description |
Original:
We have two templates for the ec2 plugin: {code:java} template1 label: test instance cap: 1 {code} {code:java} template2 label: report instance cap: 3 {code} Jenkins doesn't create a new instance with template 2 when there are jobs in queue waiting for an available excutor on a running instance created with template 1, for example in our queue: 1. test_job (waiting for _template1:test_ as all exectuors on _template1:test_ are taken and we only want 1 instance with _template1:test_) 2. report_job (waiting for _template2:report_ as it does not exist) report_job will be blocked until test_job is scheduled as Jenkins keeps trying to schedule test_job and ignores the request of an instance with _template2_ from report_job Seems like this only happens when there are a lot jobs in the queue (I don't know a definite number.. but I don't experience this issue when there are only two jobs in the queue) And I believe the problem is from Jenkins core not the ec2 plugin as the NodeProvisioner doesn't not look for all the labels. Here is the log from hudson.slaves.NodeProvisioner class {code:java} Jan 16, 2020 9:38:08 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.plugins.ec2.NoDelayProvisionerStrategy@1cfb55fa provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:09 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6a13a6e5 provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:09 PM FINE hudson.slaves.NodeProvisioner Excess workload 117.81 detected. (planned capacity=0,connecting capacity=0,Qlen=117.81,available=0&0,online=6,m=0.106) Jan 16, 2020 9:38:18 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.plugins.ec2.NoDelayProvisionerStrategy@1cfb55fa provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:18 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6a13a6e5 provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:18 PM FINE hudson.slaves.NodeProvisioner Excess workload 118.729 detected. (planned capacity=0,connecting capacity=0,Qlen=118.729,available=0&0,online=6,m=0.106) {code} |
New:
We have two templates for the ec2 plugin: {code:java} template1 label: test default instance cap: 1 usage: NORMAL{code} {code:java} template2 label: report instance cap: 3 usage: EXCLUSIVE{code} Jenkins doesn't create a new instance with template 2 when there are jobs in queue waiting for an available excutor on a running instance created with template 1, for example in our queue: 1. test_job (Restrict where this project can be run unchecked. Waiting for _template1:test_ as all exectuors on _template1:test_ are taken and we only want 1 instance with _template1:test_) 2. report_job (Restrict where this project can be run checked. Label expression: report. Waiting for _template2:report_ as it does not exist) report_job will be blocked until test_job is scheduled as Jenkins keeps trying to schedule test_job and ignores the request of an instance with _template2_ from report_job I suspect that this will only happen when the first job in the queue has "Restrict where this project can be run" unchecked. And I believe the problem is from Jenkins core not the ec2 plugin as the NodeProvisioner doesn't not look for all the labels as it should be ([https://github.com/jenkinsci/jenkins/blob/jenkins-2.204.1/core/src/main/java/hudson/slaves/NodeProvisioner.java#L815]) Here is the log from hudson.slaves.NodeProvisioner class {code:java} Jan 16, 2020 9:38:08 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.plugins.ec2.NoDelayProvisionerStrategy@1cfb55fa provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:09 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6a13a6e5 provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:09 PM FINE hudson.slaves.NodeProvisioner Excess workload 117.81 detected. (planned capacity=0,connecting capacity=0,Qlen=117.81,available=0&0,online=6,m=0.106) Jan 16, 2020 9:38:18 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.plugins.ec2.NoDelayProvisionerStrategy@1cfb55fa provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:18 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6a13a6e5 provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:18 PM FINE hudson.slaves.NodeProvisioner Excess workload 118.729 detected. (planned capacity=0,connecting capacity=0,Qlen=118.729,available=0&0,online=6,m=0.106) {code} |
Description |
Original:
We have two templates for the ec2 plugin: {code:java} template1 label: test default instance cap: 1 usage: NORMAL{code} {code:java} template2 label: report instance cap: 3 usage: EXCLUSIVE{code} Jenkins doesn't create a new instance with template 2 when there are jobs in queue waiting for an available excutor on a running instance created with template 1, for example in our queue: 1. test_job (Restrict where this project can be run unchecked. Waiting for _template1:test_ as all exectuors on _template1:test_ are taken and we only want 1 instance with _template1:test_) 2. report_job (Restrict where this project can be run checked. Label expression: report. Waiting for _template2:report_ as it does not exist) report_job will be blocked until test_job is scheduled as Jenkins keeps trying to schedule test_job and ignores the request of an instance with _template2_ from report_job I suspect that this will only happen when the first job in the queue has "Restrict where this project can be run" unchecked. And I believe the problem is from Jenkins core not the ec2 plugin as the NodeProvisioner doesn't not look for all the labels as it should be ([https://github.com/jenkinsci/jenkins/blob/jenkins-2.204.1/core/src/main/java/hudson/slaves/NodeProvisioner.java#L815]) Here is the log from hudson.slaves.NodeProvisioner class {code:java} Jan 16, 2020 9:38:08 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.plugins.ec2.NoDelayProvisionerStrategy@1cfb55fa provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:09 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6a13a6e5 provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:09 PM FINE hudson.slaves.NodeProvisioner Excess workload 117.81 detected. (planned capacity=0,connecting capacity=0,Qlen=117.81,available=0&0,online=6,m=0.106) Jan 16, 2020 9:38:18 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.plugins.ec2.NoDelayProvisionerStrategy@1cfb55fa provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:18 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6a13a6e5 provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:18 PM FINE hudson.slaves.NodeProvisioner Excess workload 118.729 detected. (planned capacity=0,connecting capacity=0,Qlen=118.729,available=0&0,online=6,m=0.106) {code} |
New:
We have two templates for the ec2 plugin: {code:java} template1 label: test default instance cap: 1 usage: NORMAL{code} {code:java} template2 label: report instance cap: 3 usage: EXCLUSIVE{code} Jenkins doesn't create a new instance with template 2 when there are jobs in queue waiting for an available excutor on a running instance created with template 1, for example in our queue: 1. test_job (Restrict where this project can be run unchecked. Waiting for _template1:test_ as all exectuors on _template1:test_ are taken and we only want 1 instance with _template1:test_) 2. report_job (Restrict where this project can be run checked. Label expression: report. Waiting for _template2:report_ as it does not exist) report_job will be blocked until test_job is scheduled as Jenkins keeps trying to schedule test_job and ignores the request of an instance with _template2_ from report_job UPDATE: I can confirm that this only happens when the first job in the queue has "Restrict where this project can be run" unchecked. And I believe the problem is from Jenkins core not the ec2 plugin as the NodeProvisioner doesn't not look for all the labels as it should be ([https://github.com/jenkinsci/jenkins/blob/jenkins-2.204.1/core/src/main/java/hudson/slaves/NodeProvisioner.java#L815]) Here is the log from hudson.slaves.NodeProvisioner class {code:java} Jan 16, 2020 9:38:08 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.plugins.ec2.NoDelayProvisionerStrategy@1cfb55fa provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:09 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6a13a6e5 provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:09 PM FINE hudson.slaves.NodeProvisioner Excess workload 117.81 detected. (planned capacity=0,connecting capacity=0,Qlen=117.81,available=0&0,online=6,m=0.106) Jan 16, 2020 9:38:18 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.plugins.ec2.NoDelayProvisionerStrategy@1cfb55fa provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:18 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6a13a6e5 provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:18 PM FINE hudson.slaves.NodeProvisioner Excess workload 118.729 detected. (planned capacity=0,connecting capacity=0,Qlen=118.729,available=0&0,online=6,m=0.106) {code} |