Nested lock() statements in a Jenkinsfile pipeline do now work as expected. Here is an example use case (Jenkinsfile with git multi-branch pipeline) of what I tried to do :
Actual outcome is that as soon as one of the two parallel branches of the integration tests stage is done and releases its lock, then any build from the same branch waiting to enter the integration tests stage will acquire the lock and start running integration tests, although the first build is still running the other parallel branch of its integration tests stage.
Workaround is to only use one single lockable resource (the outer one in our case), and making it a global instead of a per-branch lock (same resource label for all branches). This works fine, the only disadvantage is that we now have a global LIFO queue (instead of a per-branch LIFO queue) to enter the integration tests stage. This can cause some unexpected long delays for some builds.
My guess is that the Lockable Resources Plugin" was originally designed for UI-configured Jenkins jobs (not scripted). Then one could only define required lockable resources per job, and all lockable resources would always be acquired at the start of a build, and be released at the end of the build. So releasing one lockable resources in that case is identical to releasing all lockable resources (happens at the same time anyway).
I first thought that the issue has to do with using inversePrecedence: true - see also JENKINS-40787. But removing it didn't help in the example above.