-
Bug
-
Resolution: Fixed
-
Minor
-
None
-
Operating System: RHEL 7.4 Linux 64 bit, Kernel 4.4.114-94.11-default
JDK: OpenJDK 1.8.0_161-b14
Jenkins version: 2.150.2
cloud-stats-plugin version: 0.22
-
-
2.23
Hey,
the cloud-stats-plugin keeps track of every provisioning activity in order to create a statistic that is accessible via <Jenkins-Base-URL>/cloud-stats/. The underlying data is persistet in <Jenkins-Home>/org.jenkinsci.plugins.cloudstats.CloudStatistics.xml. However, there seems to be no rotation mechanism, causing that file to grow continually.
In our case that file reached 150 MB which caused various issues:
- Provisioning time took up to 15 minutes, depending on the number of currently requested agents to provision. There are about 50 short-lived (per build) agents connected at any time.
- CPU usage was going through the roof. Jenkins (the master only) is running on a 24 core machine with 128 Gig of RAM and used about 800%-2000% of all CPUs.
I assume that most of that CPU-time is spent on XML serialization as well as sychronization operations.
Workaround: Manually delete the persistence file and restart Jenkins. After pruning that persistence file, our CPU-Usage went down to 20%-200%. Provisioning times went down to about 10 seconds, regardless of load.