-
Bug
-
Resolution: Unresolved
-
Minor
-
Jenkins 2.107.2 + Plot plugin 2.04; master running on openjdk-1.8.0/linux.
After a day or two, our Jenkins master becomes very unresponsive, taking 100% CPU of all available cores. The response time drops to tens of seconds, or sometimes even minutes to load a single page.
Investigation with VisualVM shows much garbage collection taking place, with the heap nearly maxed out (at 1GB use of 1.25GB max). Looking at a heap snapshot, I discovered that the majority of memory is taken up by String[] objects, containing 5-tuples of Strings (referencing char[] arrays in turn). There are over a million of these small string arrays, each having contents similar to ["35300", "Size", "709", "1526866800302", ""].
Tracing the references of the objects reveals that they are held by the rawPlotData of MatrixPlotPublisher objects. rawPlotData is itself an array; a single sample object in our snapshot contains 106710 string arrays like the one above.
The most likely plot that's causing this is a CSVSeries plot tracking the size of ~7000 generated files (per matrix build). That's still several orders of magnitude less than the number of objects being built internally.
[JENKINS-51532] Plot plugin extreme memory leak with matrix jobs
Cameron
created issue -
Cameron
made changes -
| Description |
Original:
After a day or two, our Jenkins master becomes very unresponsive, taking 100% CPU of all available cores. The response time drops to tens of seconds, or sometimes even minutes to load a single page. Investigation with VisualVM shows much garbage collection taking place, with the heap nearly maxed out (at 1GB use of 1.25GB max). Looking at a heap snapshot, I discovered that the majority of memory is taken up by String[] objects, containing 5-tuples of Strings (referencing char[] arrays in turn). There are over a million of these small string arrays, each having contents similar to {{["35300", "Size", "709", "1526866800302", ""]}}. Tracing the references of the objects reveals that they are held by the {{rawPlotData}} of MatrixPlotPublisher objects. {{rawPlotData}} is itself an array; a single sample object in our snapshot contains 106710 string arrays like the one above. |
New:
After a day or two, our Jenkins master becomes very unresponsive, taking 100% CPU of all available cores. The response time drops to tens of seconds, or sometimes even minutes to load a single page. Investigation with VisualVM shows much garbage collection taking place, with the heap nearly maxed out (at 1GB use of 1.25GB max). Looking at a heap snapshot, I discovered that the majority of memory is taken up by String[] objects, containing 5-tuples of Strings (referencing char[] arrays in turn). There are over a million of these small string arrays, each having contents similar to {{["35300", "Size", "709", "1526866800302", ""]}}. Tracing the references of the objects reveals that they are held by the {{rawPlotData}} of MatrixPlotPublisher objects. {{rawPlotData}} is itself an array; a single sample object in our snapshot contains 106710 string arrays like the one above. The most likely plot we have is a CSVSeries plot tracking the size of ~7000 generated files. |
Cameron
made changes -
| Description |
Original:
After a day or two, our Jenkins master becomes very unresponsive, taking 100% CPU of all available cores. The response time drops to tens of seconds, or sometimes even minutes to load a single page. Investigation with VisualVM shows much garbage collection taking place, with the heap nearly maxed out (at 1GB use of 1.25GB max). Looking at a heap snapshot, I discovered that the majority of memory is taken up by String[] objects, containing 5-tuples of Strings (referencing char[] arrays in turn). There are over a million of these small string arrays, each having contents similar to {{["35300", "Size", "709", "1526866800302", ""]}}. Tracing the references of the objects reveals that they are held by the {{rawPlotData}} of MatrixPlotPublisher objects. {{rawPlotData}} is itself an array; a single sample object in our snapshot contains 106710 string arrays like the one above. The most likely plot we have is a CSVSeries plot tracking the size of ~7000 generated files. |
New:
After a day or two, our Jenkins master becomes very unresponsive, taking 100% CPU of all available cores. The response time drops to tens of seconds, or sometimes even minutes to load a single page. Investigation with VisualVM shows much garbage collection taking place, with the heap nearly maxed out (at 1GB use of 1.25GB max). Looking at a heap snapshot, I discovered that the majority of memory is taken up by String[] objects, containing 5-tuples of Strings (referencing char[] arrays in turn). There are over a million of these small string arrays, each having contents similar to {{["35300", "Size", "709", "1526866800302", ""]}}. Tracing the references of the objects reveals that they are held by the {{rawPlotData}} of MatrixPlotPublisher objects. {{rawPlotData}} is itself an array; a single sample object in our snapshot contains 106710 string arrays like the one above. The most likely plot that's causing this is a CSVSeries plot tracking the size of ~7000 generated files (per matrix build). That's still several orders of magnitude less than the number of objects being built internally. |
Olivier Dagenais
made changes -
| Description |
Original:
After a day or two, our Jenkins master becomes very unresponsive, taking 100% CPU of all available cores. The response time drops to tens of seconds, or sometimes even minutes to load a single page. Investigation with VisualVM shows much garbage collection taking place, with the heap nearly maxed out (at 1GB use of 1.25GB max). Looking at a heap snapshot, I discovered that the majority of memory is taken up by String[] objects, containing 5-tuples of Strings (referencing char[] arrays in turn). There are over a million of these small string arrays, each having contents similar to {{["35300", "Size", "709", "1526866800302", ""]}}. Tracing the references of the objects reveals that they are held by the {{rawPlotData}} of MatrixPlotPublisher objects. {{rawPlotData}} is itself an array; a single sample object in our snapshot contains 106710 string arrays like the one above. The most likely plot that's causing this is a CSVSeries plot tracking the size of ~7000 generated files (per matrix build). That's still several orders of magnitude less than the number of objects being built internally. |
New:
After a day or two, our Jenkins master becomes very unresponsive, taking 100% CPU of all available cores. The response time drops to tens of seconds, or sometimes even minutes to load a single page.
Investigation with VisualVM shows much garbage collection taking place, with the heap nearly maxed out (at 1GB use of 1.25GB max). Looking at a heap snapshot, I discovered that the majority of memory is taken up by String[] objects, containing 5-tuples of Strings (referencing char[] arrays in turn). There are over a million of these small string arrays, each having contents similar to {{{}["35300", "Size", "709", "1526866800302", ""]{}}}. Tracing the references of the objects reveals that they are held by the {{rawPlotData}} of MatrixPlotPublisher objects. {{rawPlotData}} is itself an array; a single sample object in our snapshot contains 106710 string arrays like the one above. The most likely plot that's causing this is a CSVSeries plot tracking the size of ~7000 generated files (per matrix build). That's still several orders of magnitude less than the number of objects being built internally. |
Olivier Dagenais
made changes -
| Attachment | New: memory histogram of our Jenkins controller.png [ 61966 ] |
