This seems like a very elegant workaround. Thanks for sharing it!

That is deadlock-prone.

Since trying to get down job execution time by most possible parallelizing is essential for accepteable turnaround times and hence the user satisfaction about jenkins... it would be realy good to see this resolved.

So I see two usecases for this:

• having a job that uses more resources than others so it needs to allocate more Builders (i.e. starting a clustered test of any software should utilize more resources than a single instance version so it may require at least 2 but if available 3 slots)
• giving a job the possibility to use all available resources (i.e. make -j N) where one would just want to specify `-1` as slots parameter, and get all slots on the node, plus get to know the number of slots that were reserved to pass it on to make; so that a big worker could have 10 build processors or a smaller one only 5 and a pipeline could adopt.

Do you have any workaround ? I'd like to allocate N nodes to run "make -j N" .

To mitigate potential deadlock with Thorsten's solution, we would need to timeout if not possible to get all the nodes allocated, cancel the current job and post it back in the executors queue

To handle deadlock and nodes starving, here a suggestion: running a parallel task with a timer and the job executor.

def multiSlotNode_(String label, int slots = 1, java.util.concurrent.atomic.AtomicInteger nodeAcquired, Closure body) {
	if (slots == 1) {
		node(label) { 
		    nodeAcquired.addAndGet(1)
		    body() }
	} else if (slots > 1) {
		node(label) { 
		    nodeAcquired.addAndGet(1)
		    multiSlotNode_(env.NODE_NAME, slots - 1, nodeAcquired, body)
		}
	} else {
		throw new IllegalArgumentException("Number of slots must be greather than zero")
	}
}

def multiSlotNode(String label, int slots = 1, Closure body) {

    def nodeAcquired = new java.util.concurrent.atomic.AtomicInteger(0)
    
    timestamps {
        def rerun = true
        while (rerun) {
            
            try {
                parallel (
                    failFast: true,
                    
                    "timer" : {
                        timeout(time: 5, unit: 'SECONDS') {
                            waitUntil {
                                return nodeAcquired.get() >= slots
                            }
                        }
                    },
                    
                    "job" : {
                        nodeAcquired.set(0)
                        rerun = false
                        multiSlotNode_(label, slots, nodeAcquired, body)
                    }
                )
            } catch (org.jenkinsci.plugins.workflow.steps.FlowInterruptedException ex) {
                // Timeout to get available nodes. Trying again later
                sleep time: 60, unit: 'SECONDS'
                rerun = true
            }
        }
    }
}

multiSlotNode('', 4, {sleep(time: 10, unit: 'SECONDS'); println "HELLO WORLD"} )

But definitely will appreciate to have a slot parameter in the "node" step to handle it for you.

Just commenting that this is a feature we want as well. We used to use the Heavy Job Plugin, which did exactly what we need. We had to leave it behind when moving to pipelines.

Our Jenkins instance has ~30 jobs, and almost all of them are relatively lightweight and can run in parallel with other jobs. However, 2-3 of them are large C++ compiles, which are very resource intensive and should NOT share the node with other jobs. In this situation, my choices are...

1) Give my nodes only 1 executor. This results in lots of wasted capacity, as I have to spin up LOTS of nodes to ensure I can meet peak demand. 

2) Give my nodes multiple executors. This results in frequent "slow builds" as Jenkins occasionally decides to run 2 compiles on the same node.

Being able to say "this job is heavy and cannot share a node" would mitigate these pain points.

Just adding my comment that something like this would be really useful for us on our end.

We have some large templated codes that eat a lot of ram when building, and it's bad if we get 2 or 3 make -j 30's queued up on the same build slave so being able to set the job weight in our pipeline jobs would enable us to move more of our testing environment into a pipeline system.

If there was a way to do something like:  

node(label: "node_label", weight: 30) { do-my-30-core-build }

  life would be pretty good.

jguigui I will try out your hack, I'm not sure what happens though if your 'node' is really just a label that might include several machines.  For example, we often have labels such as "rhel7" and may have many red hat 7 systems in the build farm under that label so Jenkins can just schedule a build where there are available nodes.
So if I have three 20-core machines under that label and four jobs with a build weight of 15 cores, what I'd want is for the first 3 jobs to run and the fourth to wait until it can fit on a machine. Do you know if your script would do that or would I get 3 machines with 15 executors taken for the first three jobs and then the 4th job taking 5 executors from each machine?

Still would be nice if the pipeline plugin had a weight parameter to the node command

Same issue here. We have a job that makes heavy use of the CPU cores, and being able to declare a specific number of executors required to run this pipeline will make our life way easier.

Hello!

It would be very interesting for me to implement such plugin. I've looked through source code of Heavy Job Plugin and some examples of Pipeline API like lockable-resources-plugin. Unfortunately Pipeline API differs from Freestyle Projects. For now it's clear how to create custom step and wrap another step with it but I can't understand how to create something like subtasks in Heavy Job plugin.

I would be very happy if someone can explain me this moment a bit more or give me a useful link.