-
Bug
-
Resolution: Unresolved
-
Minor
-
Jenkins 2.107.2 + Plot plugin 2.04; master running on openjdk-1.8.0/linux.
After a day or two, our Jenkins master becomes very unresponsive, taking 100% CPU of all available cores. The response time drops to tens of seconds, or sometimes even minutes to load a single page.
Investigation with VisualVM shows much garbage collection taking place, with the heap nearly maxed out (at 1GB use of 1.25GB max). Looking at a heap snapshot, I discovered that the majority of memory is taken up by String[] objects, containing 5-tuples of Strings (referencing char[] arrays in turn). There are over a million of these small string arrays, each having contents similar to ["35300", "Size", "709", "1526866800302", ""].
Tracing the references of the objects reveals that they are held by the rawPlotData of MatrixPlotPublisher objects. rawPlotData is itself an array; a single sample object in our snapshot contains 106710 string arrays like the one above.
The most likely plot that's causing this is a CSVSeries plot tracking the size of ~7000 generated files (per matrix build). That's still several orders of magnitude less than the number of objects being built internally.
We have a Jenkins instance we use to query and plot free space on a number of machines/storage devices (it's currently the only thing that instance is running!) and recently our controller has started manifesting symptoms just like those described here, so it would appear that there's indeed at least one memory leak.
Using /monitoring?part=heaphisto we are indeed seeing a lot of arrays taking up the memory: