-
Improvement
-
Resolution: Fixed
-
Minor
-
jenkins 2.481 java 17 16-core 32 GB memory server
An executor has a significant idle time after completing a task before receiving the next task.
My Jenkins has thousands of items in the queue, and they are all buildable.
An executor has a few seconds of idle time before it gets the next item to execute. When the queue reaches 20,000, there's even a few minutes of idle time.But when the queue is only a few hundred, the time is only a few hundred milliseconds.
I see this class with comments like this.
I can alleviate this problem by splitting the queue across multiple machines, but is there any other way I can reduce this idle time on one machine? This duration may be more appropriate to call scheduling time.
UPDATE 2025-01-23
Through source code analysis I find the lock in Executor.java.
jenkins/core/src/main/java/hudson/model/Executor.java at master · jenkinsci/jenkins · GitHub
Object task uses Queue.withLock(), and it needs lock in Queue.
Recently I added some logs warpped this sentence in Executor.run(), method starts immediately after distribute in Queue.maintain() but it waits the release of Queue.lock to create task.
If the loop in Queue.maintain() is not finished, it will not be able to obtain the timing of execution.
[JENKINS-75152] An executor has a significant idle time after completing a task before receiving the next task when the queue has thousands of items
| Comment |
[
] |
| Description |
Original:
An executor has a significant idle time after completing a task before receiving the next task.
My Jenkins has thousands of items in the queue, and they are all buildable. An executor has a few seconds of idle time before it gets the next item to execute. When the queue reaches 20,000, there's even a few minutes of idle time.But when the queue is only a few hundred, the time is only a few hundred milliseconds. I see this class with comments like this. [https://github.com/jenkinsci/jenkins/blob/3878f0a77b859bf427295ace94762c20d76735a3/core/src/main/java/hudson/model/Queue.java#L1692] I can alleviate this problem by splitting the queue across multiple machines, but is there any other way I can reduce this idle time on one machine? This duration may be more appropriate to call scheduling time. |
New:
An executor has a significant idle time after completing a task before receiving the next task.
My Jenkins has thousands of items in the queue, and they are all buildable. An executor has a few seconds of idle time before it gets the next item to execute. When the queue reaches 20,000, there's even a few minutes of idle time.But when the queue is only a few hundred, the time is only a few hundred milliseconds. I see this class with comments like this. [https://github.com/jenkinsci/jenkins/blob/3878f0a77b859bf427295ace94762c20d76735a3/core/src/main/java/hudson/model/Queue.java#L1692] I can alleviate this problem by splitting the queue across multiple machines, but is there any other way I can reduce this idle time on one machine? This duration may be more appropriate to call scheduling time. _UPDATE 2025-01-23_ Through source code analysis I find the lock in Executor.java. [jenkins/core/src/main/java/hudson/model/Executor.java at master · jenkinsci/jenkins · GitHub|https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/Executor.java#L354] Object task uses Queue.withLock, and it needs lock in Queue. Recently I added some logs warpped this sentence in Executor.run(), method starts immediately after distribute in Queue.maintain() but it waits the release of Queue.lock to create task. |
| Description |
Original:
An executor has a significant idle time after completing a task before receiving the next task.
My Jenkins has thousands of items in the queue, and they are all buildable. An executor has a few seconds of idle time before it gets the next item to execute. When the queue reaches 20,000, there's even a few minutes of idle time.But when the queue is only a few hundred, the time is only a few hundred milliseconds. I see this class with comments like this. [https://github.com/jenkinsci/jenkins/blob/3878f0a77b859bf427295ace94762c20d76735a3/core/src/main/java/hudson/model/Queue.java#L1692] I can alleviate this problem by splitting the queue across multiple machines, but is there any other way I can reduce this idle time on one machine? This duration may be more appropriate to call scheduling time. _UPDATE 2025-01-23_ Through source code analysis I find the lock in Executor.java. [jenkins/core/src/main/java/hudson/model/Executor.java at master · jenkinsci/jenkins · GitHub|https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/Executor.java#L354] Object task uses Queue.withLock, and it needs lock in Queue. Recently I added some logs warpped this sentence in Executor.run(), method starts immediately after distribute in Queue.maintain() but it waits the release of Queue.lock to create task. |
New:
An executor has a significant idle time after completing a task before receiving the next task.
My Jenkins has thousands of items in the queue, and they are all buildable. An executor has a few seconds of idle time before it gets the next item to execute. When the queue reaches 20,000, there's even a few minutes of idle time.But when the queue is only a few hundred, the time is only a few hundred milliseconds. I see this class with comments like this. [https://github.com/jenkinsci/jenkins/blob/3878f0a77b859bf427295ace94762c20d76735a3/core/src/main/java/hudson/model/Queue.java#L1692] I can alleviate this problem by splitting the queue across multiple machines, but is there any other way I can reduce this idle time on one machine? This duration may be more appropriate to call scheduling time. _UPDATE 2025-01-23_ Through source code analysis I find the lock in {_}Executor.java{_}. [jenkins/core/src/main/java/hudson/model/Executor.java at master · jenkinsci/jenkins · GitHub|https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/Executor.java#L354] Object task uses {_}Queue.withLock(){_}, and it needs lock in {_}Queue{_}. Recently I added some logs warpped this sentence in {_}Executor.run(){_}, method starts immediately after distribute in _Queue.maintain()_ but it waits the release of _Queue.lock_ to create task. If the loop in _Queue.maintain()_ is not finished, it will not be able to obtain the timing of execution. |
| Assignee | New: qingyi [ catkingfirst ] |
| Labels | New: improvement |
| Status | Original: Open [ 1 ] | New: In Progress [ 3 ] |
| Resolution | New: Fixed [ 1 ] | |
| Status | Original: In Progress [ 3 ] | New: Resolved [ 5 ] |