Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
None
-
CentOS 5.4, x86_64, Java 1.6.0_16
Description
This has started happening more frequently since 1.420, and since updating to 1.424, is now pretty much guaranteed to happen after 24 hours of restarting. There have been no major changes to the build architecture, and this seems to happen to all jobs, on all hosts (slaves and master alike). The thread dump is attached, as well as the systemInfo page for one of our build slaves ("Alfred").
Attachments
Activity
amcfague
added a comment - We actually hadn't even initialized it--and the initialization screen would hang forever when the initialize button was clicked in the config.
amcfague
added a comment - We actually hadn't even initialized it--and the initialization screen would hang forever when the initialize button was clicked in the config. 
Frédéric Camblor
added a comment - Global-build-stats 1.2-SNAPSHOT, fixing job hanging on large jenkins instances
Frédéric Camblor
added a comment - Global-build-stats 1.2-SNAPSHOT, fixing job hanging on large jenkins instances Frédéric Camblor
added a comment - Hi all,
Could you mind testing attached hpi, it should solve the problem in 2 ways :
- I generalized kohsuke pull request to delay job results serialization in a separate thread (thus, jobs won't hang anymore after completion)
- I splitted job results into monthly files (instead of one fat global-build-stats.xml file). Thus, when adding a job result, only the current month results will be re-serialized
Before testing it, please, backup your global-build-stats.xml file since v1.2-SNAPSHOT will make changes to your global build stats file organization. To be able to go back to 1.1, you will have to restore your global-build-stats.xml file.
Later, I will provide strategies for job build results retention policy (based on nicolas de loof pull request and my comments on it : https://github.com/jenkinsci/global-build-stats-plugin/pull/1)
Frédéric Camblor
added a comment - Hi all,
Could you mind testing attached hpi, it should solve the problem in 2 ways :
I generalized kohsuke pull request to delay job results serialization in a separate thread (thus, jobs won't hang anymore after completion)
I splitted job results into monthly files (instead of one fat global-build-stats.xml file). Thus, when adding a job result, only the current month results will be re-serialized
Before testing it, please, backup your global-build-stats.xml file since v1.2-SNAPSHOT will make changes to your global build stats file organization. To be able to go back to 1.1, you will have to restore your global-build-stats.xml file.
Later, I will provide strategies for job build results retention policy (based on nicolas de loof pull request and my comments on it : https://github.com/jenkinsci/global-build-stats-plugin/pull/1 ) Frédéric Camblor
added a comment - Fix integrated in global-build-stats v1.2 
Our file is just over 100MB, and has records for builds from May-2010.
We've been talking to Kohsuke as we were tracking down our performance problem to this plugin, and he had some ideas about how the core might better support this kind of data storage for plugins.
One other thing that we noticed is that the xml store contains stats for builds that have been removed from the main storage due to age / count. It might be nice if this plugin had at least an option to mirror that cleanup as well as have other kinds of retention contraints.