[JENKINS-49940] Concurrent build limits not honored on kubernetes container environment with Jenkins 2.89.4 - Jenkins Jira

Details

Type: Bug
Status: Open (View Workflow)
Priority: Critical
Resolution: Unresolved
Component/s: throttle-concurrent-builds-plugin
Labels:
None
Environment:
Jenkins : 2.89.4
Throttle Concurrent Builds : 2.0.1
Kubernetes : 1.3.1

Similar Issues:

Show

Description

After upgrading to Jenkins to 2.89.4 we noticed the throttle plugin 2.0.1 doesn't always prevent jobs from running in parallel as expected.

Steps to reproduce:
1. Create a Throttle category with

Maximum Total Concurrent Builds -> 0
Maximum Concurrent Builds Per Node -> 3
Maximum Concurrent Builds Per Node Labeled As Above -> 3.

2. Created Kubernetes Pod Template with

Max number of instances -> 3
Container Cap -> 3

3. Create 3 jobs using the category, restricted to a node with two executors

4. Request builds of all 3 jobs.

What should happen:
4. Jobs run in sequence; 1 then 2 then 3.

What actually happens:
5. Job 1 starts building, jobs 2 and 3 wait in queue (OK).
6. After job 1 finishes, both job 2 and 3 start running (not OK).

Setup Details:

Jenkins : 2.89.4
Throttle Concurrent Builds : 2.0.1
Kubernetes : 1.3.1

Attachments

Activity

Akshay Chaknalwar added a comment - 2018-03-05 23:38

Tried following scenario

Set limit to 3 for concurrent jobs
Started 6 jobs at once
3 jobs started and 3 jobs were in queue
As soon as 2 jobs from currently running jobs completed, 3 remaining jobs started and total current jobs are 4 now.
Expected was to start only 1 job from queue

Started provisioning Kubernetes Pod Template from k8s with 1 executors. Remaining excess workload: 1.97
Mar 05, 2018 11:27:02 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
Started provisioning Kubernetes Pod Template from k8s with 1 executors. Remaining excess workload: 0.97
Mar 05, 2018 11:27:02 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
Started provisioning Kubernetes Pod Template from k8s with 1 executors. Remaining excess workload: -0.03
Mar 05, 2018 11:27:13 PM INFO hudson.slaves.NodeProvisioner$2 run
Kubernetes Pod Template provisioning successfully completed. We have now 4 computer(s)
Mar 05, 2018 11:27:13 PM INFO hudson.slaves.NodeProvisioner$2 run
Kubernetes Pod Template provisioning successfully completed. We have now 5 computer(s)
Mar 05, 2018 11:27:13 PM INFO hudson.slaves.SlaveComputer tryReconnect
Attempting to reconnect qa-testslave9
Mar 05, 2018 11:27:13 PM INFO hudson.slaves.NodeProvisioner$2 run
Kubernetes Pod Template provisioning successfully completed. We have now 6 computer(s)
Mar 05, 2018 11:27:13 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Kubernetes agents: 0

Akshay Chaknalwar added a comment - 2018-03-05 23:38 Tried following scenario Set limit to 3 for concurrent jobs Started 6 jobs at once 3 jobs started and 3 jobs were in queue As soon as 2 jobs from currently running jobs completed, 3 remaining jobs started and total current jobs are 4 now. Expected was to start only 1 job from queue Started provisioning Kubernetes Pod Template from k8s with 1 executors. Remaining excess workload: 1.97 Mar 05, 2018 11:27:02 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply Started provisioning Kubernetes Pod Template from k8s with 1 executors. Remaining excess workload: 0.97 Mar 05, 2018 11:27:02 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply Started provisioning Kubernetes Pod Template from k8s with 1 executors. Remaining excess workload: -0.03 Mar 05, 2018 11:27:13 PM INFO hudson.slaves.NodeProvisioner$2 run Kubernetes Pod Template provisioning successfully completed. We have now 4 computer(s) Mar 05, 2018 11:27:13 PM INFO hudson.slaves.NodeProvisioner$2 run Kubernetes Pod Template provisioning successfully completed. We have now 5 computer(s) Mar 05, 2018 11:27:13 PM INFO hudson.slaves.SlaveComputer tryReconnect Attempting to reconnect qa-testslave9 Mar 05, 2018 11:27:13 PM INFO hudson.slaves.NodeProvisioner$2 run Kubernetes Pod Template provisioning successfully completed. We have now 6 computer(s) Mar 05, 2018 11:27:13 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision Excess workload after pending Kubernetes agents: 0

Jenkins

Concurrent build limits not honored on kubernetes container environment with Jenkins 2.89.4

Details

Description

Attachments

Activity

People

Dates