-
Bug
-
Resolution: Unresolved
-
Major
When the JobGlobalBuildDiscarderStrategy is configured (which seems to be the case by default ?) then jobs that have a build discarder configured have their the logRotate performed twice when the build finishes:
- First through https://github.com/jenkinsci/jenkins/blob/jenkins-2.387.3/core/src/main/java/hudson/model/Run.java#L1964
- Then through the RunListener#onFinalized in https://github.com/jenkinsci/jenkins/blob/jenkins-2.387.3/core/src/main/java/jenkins/model/GlobalBuildDiscarderListener.java
This strategy is deduplicating the work on build completion. Which seems like a bug.
In environment with very active jobs and a build discarder that keep a large amount of build (in the 100s or 1000s), this behavior causes severe lock contention (following extract from a 2.414.3 instance:
"Running CpsFlowExecution[Owner[pipeline-1/27636:pipeline-1 #27636]]" id=807 state=BLOCKED - waiting to lock <0x38218367> (a hudson.model.RunMap) owned by Running CpsFlowExecution[Owner[pipeline-1/27624:pipeline-1 #27624]] id=64 at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:566) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:537) at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:77) at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:64) at jenkins.model.lazy.AbstractLazyLoadRunMap$2$1.next(AbstractLazyLoadRunMap.java:211) at hudson.util.RunList.size(RunList.java:136) at hudson.tasks.LogRotator.perform(LogRotator.java:162) at hudson.model.Job.logRotate(Job.java:485) at jenkins.model.JobGlobalBuildDiscarderStrategy.apply(JobGlobalBuildDiscarderStrategy.java:54) at jenkins.model.BackgroundGlobalBuildDiscarder.lambda$processJob$0(BackgroundGlobalBuildDiscarder.java:67) at jenkins.model.BackgroundGlobalBuildDiscarder$$Lambda$1391/0x00000008412b4440.accept(Unknown Source) at java.base@11.0.20/java.lang.Iterable.forEach(Iterable.java:75) at jenkins.model.BackgroundGlobalBuildDiscarder.processJob(BackgroundGlobalBuildDiscarder.java:61) at jenkins.model.GlobalBuildDiscarderListener.onFinalized(GlobalBuildDiscarderListener.java:49) at hudson.model.listeners.RunListener.lambda$fireFinalized$3(RunListener.java:260) at hudson.model.listeners.RunListener$$Lambda$1367/0x00000008412a5040.accept(Unknown Source) at jenkins.util.Listeners.lambda$notify$0(Listeners.java:59) at jenkins.util.Listeners$$Lambda$502/0x0000000840cc7040.run(Unknown Source) at jenkins.util.Listeners.notify(Listeners.java:67) at hudson.model.listeners.RunListener.fireFinalized(RunListener.java:258) at hudson.model.Run.onEndBuilding(Run.java:2008) at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:670) at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:1068) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1535) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:512) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:41) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51) at jenkins.util.ErrorLoggingExecutorService$$Lambda$667/0x0000000840e77440.run(Unknown Source) at java.base@11.0.20/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base@11.0.20/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base@11.0.20/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base@11.0.20/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.20/java.lang.Thread.run(Thread.java:829) Locked synchronizers: count = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@34434bc1 "Running CpsFlowExecution[Owner[pipeline-1/27635:pipeline-1 #27635]]" id=834 state=BLOCKED - waiting to lock <0x38218367> (a hudson.model.RunMap) owned by Running CpsFlowExecution[Owner[pipeline-1/27624:pipeline-1 #27624]] id=64 at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:566) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:537) at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:77) at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:64) at jenkins.model.lazy.AbstractLazyLoadRunMap$2$1.next(AbstractLazyLoadRunMap.java:211) at hudson.util.RunList.size(RunList.java:136) at hudson.tasks.LogRotator.perform(LogRotator.java:162) at hudson.model.Job.logRotate(Job.java:485) at jenkins.model.JobGlobalBuildDiscarderStrategy.apply(JobGlobalBuildDiscarderStrategy.java:54) at jenkins.model.BackgroundGlobalBuildDiscarder.lambda$processJob$0(BackgroundGlobalBuildDiscarder.java:67) at jenkins.model.BackgroundGlobalBuildDiscarder$$Lambda$1391/0x00000008412b4440.accept(Unknown Source) at java.base@11.0.20/java.lang.Iterable.forEach(Iterable.java:75) at jenkins.model.BackgroundGlobalBuildDiscarder.processJob(BackgroundGlobalBuildDiscarder.java:61) at jenkins.model.GlobalBuildDiscarderListener.onFinalized(GlobalBuildDiscarderListener.java:49) at hudson.model.listeners.RunListener.lambda$fireFinalized$3(RunListener.java:260) at hudson.model.listeners.RunListener$$Lambda$1367/0x00000008412a5040.accept(Unknown Source) at jenkins.util.Listeners.lambda$notify$0(Listeners.java:59) at jenkins.util.Listeners$$Lambda$502/0x0000000840cc7040.run(Unknown Source) at jenkins.util.Listeners.notify(Listeners.java:67) at hudson.model.listeners.RunListener.fireFinalized(RunListener.java:258) at hudson.model.Run.onEndBuilding(Run.java:2008) at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:670) at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:1068) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1535) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:512) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:41) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51) at jenkins.util.ErrorLoggingExecutorService$$Lambda$667/0x0000000840e77440.run(Unknown Source) at java.base@11.0.20/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base@11.0.20/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base@11.0.20/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base@11.0.20/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.20/java.lang.Thread.run(Thread.java:829)
