-
New Feature
-
Resolution: Unresolved
-
Critical
-
Jenkins 2.45, latest versions of all plugins installed
For Freestyle jobs, the Heavy Job plugin (see https://wiki.jenkins-ci.org/display/JENKINS/Heavy+Job+Plugin) can be used in order to configure the number of executors that a given job occupies. This is necessary in order to avoid CPU oversubscription when a single build consumes multiple HW threads (e.g. in case of "make -jN", "ninja -jN" etc.).
I have done some research and was not able to find a means to specify the number of executors consumed by a Pipeline stage.
Parallel builds are a necessity in case of large C++ builds, and the ability to specify the number of executors is a practical solution that can be used by the scheduler in order to avoid CPU oversubscription on a particular node.
- is duplicated by
-
-
- Resolved
-
-
-
- Resolved
-
-
-
- Resolved
-
-
-
- Resolved
-
- links to
[JENKINS-41940] Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies
Vitali Gontsharuk
created issue -
Vitali Gontsharuk
made changes -
| Summary | Original: Pipeline stages should be configurable with a weight property, corresponding to the number executors the stage occupies | New: Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies |
Vitali Gontsharuk
made changes -
| Rank | New: Ranked higher |
| Component/s | New: pipeline-stage-step-plugin [ 21709 ] | |
| Component/s | Original: pipeline [ 21692 ] |
| Component/s | New: heavy-job-plugin [ 16006 ] | |
| Component/s | Original: pipeline-stage-step-plugin [ 21709 ] | |
| Assignee | Original: Jesse Glick [ jglick ] | |
| Labels | Original: executor pipeline stage weight | New: pipeline |
Vitali Gontsharuk
added a comment - Well, I was using the abstractions from Declarative Pipeline here. You obviously know better how this is mapped to Scripted Pipeline primitives.
Concerning the approach: I have basically proposed the same functionality as currently exists for Freestyle jobs to be supported for Pipeline jobs. This is not a very fancy approach, but it is enough to get the job done in practice. If the same goal could be reached automatically, without any configuration from the user, that would be just perfect.
What do you think, would it be a lot of effort to implement?
Vitali Gontsharuk
added a comment - Well, I was using the abstractions from Declarative Pipeline here. You obviously know better how this is mapped to Scripted Pipeline primitives.
Concerning the approach: I have basically proposed the same functionality as currently exists for Freestyle jobs to be supported for Pipeline jobs. This is not a very fancy approach, but it is enough to get the job done in practice. If the same goal could be reached automatically, without any configuration from the user, that would be just perfect.
What do you think, would it be a lot of effort to implement? Vitali Gontsharuk
added a comment - jglick So who do you think would be an appropriate assignee for this? Thanks!
Vitali Gontsharuk
added a comment - jglick So who do you think would be an appropriate assignee for this? Thanks! As a workaround I'm currently using the following function:
def multiSlotNode(String label, int slots = 1, Closure body) { if (slots == 1) { node(label) { body() } } else if (slots > 1) { node(label) { multiSlotNode(env.NODE_NAME, slots - 1, body) } } else { throw new IllegalArgumentException("Number of slots must be greather than zero") } }
Use as
multiSlotNode('desired-node-label', 2)
{ doSomething }
Thorsten Meinl
added a comment - - edited As a workaround I'm currently using the following function:
def multiSlotNode( String label, int slots = 1, Closure body) {
if (slots == 1) {
node(label) { body() }
} else if (slots > 1) {
node(label) { multiSlotNode(env.NODE_NAME, slots - 1, body) }
} else {
throw new IllegalArgumentException( " Number of slots must be greather than zero" )
}
}
Use as
multiSlotNode('desired-node-label', 2)
{ doSomething }
stage is only for display. I think you are talking about node. Could perhaps be supported somehow in this plugin, TBD.
I think the approach is misconceived, though. Better to have a QueueTaskDispatcher which blocks off executor slots dynamically based on measured system load or something.