Hello, we've started seeing this odd issue on multiple jenkins masters and it seems to have started since we upgraded them from 2.222.4 to 2.235.1

(we stick with LTS versions).

Jobs using milestones randomly end up in a stuck state where:

• they endlessly wait to progress on a milestone in their job console
• or, they fulfil all of their steps but the job console does not show the job as complete

in both cases, this eventually produces a pile-up of jobs that are shown as still busy with the master having to do something with them, but in effect they are stuck:

I investigated this with the threadDump page and found traces suggesting that these jobs are BLOCKED by other jobs on the milestone step.

All the involved jobs have in common is that they use milestone.

 
Running CpsFlowExecutionOwner[TopFolder/MyJob/trigger-data-release/master/99:TopFolder/MyJob/trigger-data-release/master #99]]
"Running CpsFlowExecutionOwner[TopFolder/MyJob/trigger-data-release/master/99:TopFolder/MyJob/trigger-data-release/master #99]]" Id=460526 Group=main BLOCKED on org.jenkinsci.plugins.pipeline.milestone.MilestoneStep$DescriptorImpl@10b40a8f owned by "Running CpsFlowExecutionOwner[AnotherFolder/AnotherJob/master/429:AnotherFolder/AnotherJob/master #429]]" Id=460528

This is not what I would expect to see, assuming milestone works in the context of the same job (to prevent parallel execution of certain steps); rather, it's something that would suggest cross-jobs locking, which might be ultimately leading to that deadlock state that I observed above. Of course this is entirely my interpretation of the above.

None of the jobs in question are new or have been altered recently, they used to work ok. As mentioned initially the only change we have done is an upgrade to 2.235.1 (and more recently to 2.235.2) so one potential guess here is that this version has changed things in a way that breaks the milestone plugin?

At the moment the workaround we have is to avoid using milestone, and instead use disableConcurrentBuilds() as a replacement (not as granular, but does the job in most cases).