So we've identified this to be some sort of race condition that affects pipelines only. We've taken out as much as we can out of the equation. So we setup an almost empty Jenkins environment with docker with just concurrent builds set up. These are the steps to reproduce:

1. Set up a single throttle category (max 1 per node, and max 1 across nodes)
2. Manually create three pipeline jobs called JobA, JobB, JobC with the below pipeline code, and throttle concurrent jobs enabled to with the created category selected.
3. Run JobA, and wait for it to start (wait until the sleep step kicks in)
4. Add all the created jobs (JobA, JobB, and JobC) onto the queue in quick succession (from the list view just click schedule build icon as fast as you can).
5. At this point you should have JobA running (on the sleeping step), and a second run of JobA on the queue in addition to JobB and JobC waiting on the queue.
6. Observe that, once running JobA finished, JobA, JobB and JobC started as soon as the first run of JobA finished.

The pipeline used:

pipeline {
    agent any
    stages {
        stage('Hello') {
            steps {
                echo 'Hello World'
            }
        }
        stage('Wait') {
            steps {
                sh 'sleep 20'
            }
        }
        stage('Goodbye') {
            steps {
                echo 'Goodbye World'
            }
        }
    }
}

Of course this race condition essentially renders this plugin completely useless.

We're currently planning to use lockable-resources as an alternative until this issue gets fixed, but lockable resources isn't ideal as the jobs at this point would take up an executor while waiting for the resources to be unlocked.

basil any thoughts on this? I've been trying to replicate the same behaviour while debugging the plugin in the hopes of fixing the issue, but I'm struggling to do so.

This job-dsl and pipeline seem to reproduce the issue quite nicely:

job-dsl:

for (int i in 0..<10) {
    pipelineJob("job${i}") {
        displayName("Job ${i}")
        triggers {
            cron('H/2 * * * *')
        }
        definition {
            cpsScm {
                scm {
                    git {
                        remote {
                            url('git@github.com:hoshsadiq/parallel-test.git')
                            credentials('git')
                        }
                        branches('*/main')
                    }
                }
                scriptPath('Jenkinsfile')
            }
        }
    }
}

pipeline:

pipeline {
    agent any

    options {
        disableConcurrentBuilds()
        throttleJobProperty(
            categories: ['test_a'],
            throttleEnabled: true,
            throttleOption: 'category'
        )
    }

    stages {
        stage('Hello') {
            steps {
                sh 'echo "hello world"'
                sleep 130
                sh 'echo "bye world"'
            }
        }
    }
} 

Exact same thing is happening. All the jobs get queued up, and a single job ends up running. Once it's

See JENKINS-45140 (comment). ThrottleCategory#maxConcurrentPerNode remains unimplemented for Pipeline jobs using ThrottleJobProperty, unlike ThrottleCategory#maxConcurrentTotal. Note that ThrottleCategory#maxConcurrentPerNode is implemented for Pipeline jobs using the throttle step, although apparently users report that the design of that step is not simpatico with Declarative Pipeline (yet some users have found creative workarounds in the comments of JENKINS-45140, and Scripted Pipeline works just fine). To get a sense for what works and what doesn't work in this plugin, take a look at the test suite. ThrottleJobPropertyPipelineTest#onePerNode demonstrates the problem and is currently marked with @Ignore.

I realize all of this is incredibly confusing. The documentation is long overdue for a rewrite; a pull request to update the documentation would most certainly be appreciated! Also see my last comment in JENKINS-45140 for some ideas about how the underlying problem could be fixed.

I am facing a similar issue. What was the exact fix for this issue?

In my case, jobs in the queue are getting throttled as expected when some jobs are already running. But all the throttled jobs in the queue start together once the running job is finished. We expect only some of those throttled jobs to start together based on the parameter value while the remaining ones should continue to get throttled and wait in the queue. 

Currently, I'm using this in my Jenkinsfile :

throttleJobProperty(maxConcurrentPerNode: 2,   
                   maxConcurrentTotal: 2, 
                   categories: ['sampleCategory'],
                   throttleEnabled: true,
                   throttleOption: 'project',
                   limitOneJobWithMatchingParams: true,
                   paramsToUseForLimit: 'sampleParam')

Our scripted pipeline jobs sort of get throttled, but most of the time end up running more than configured amounts of builds per node. Freestyle jobs outright ignore the existence of scripted pipeline jobs, even though both configured "the same".