-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
1.651.3
Hi,
Recently we upgraded from 1.596.3 to 1.651.3 and observing that the later version is consuming lot of heap as soon as it comes up.
WE have got many jobs with huge build history.. in terms of 4000 builds. Each builds's build.xml is in terms of 500k.
Whenever the instance is restarted the heap starts to grow and finally the instance crashed with OEM.
The heap dump shows that values defined under hudson.model.TextParameterValue of each build is loaded into memory!
Refer screenshots: heap-screenshot-0.png
& heap-screenshot-1.png
Here is the contents of build.xml of one such build. refer screenshot build.xml-screenshot.png
Is it this commit SECURITY-170 Store initial parameters list for later use which is loading all values?
[JENKINS-38914] hudson.model.TextParameterValue heavily bloats memory use and size of build.xml by loading all builds TextParameterValue in memorey
Dilip Mahadevappa
created issue -
Dilip Mahadevappa
made changes -
| Summary | Original: hudson.model.ParametersAction heavily bloats memory use and size of build.xml with TextParameterValue | New: hudson.model.ParametersAction heavily bloats memory use and size of build.xml by loading all builds TextParameterValue in memory |
Dilip Mahadevappa
made changes -
| Attachment | New: heap-screenshot-3.png [ 34329 ] |
Dilip Mahadevappa
made changes -
| Attachment | New: heap-screenshot-0.png [ 34330 ] |
Dilip Mahadevappa
made changes -
| Attachment | Original: heap-screenshot-3.png [ 34329 ] |
Dilip Mahadevappa
made changes -
| Attachment | Original: heap-screenshot-0.png [ 34326 ] |
Dilip Mahadevappa
made changes -
| Attachment | New: lazy_BuildReference.MODE.JPG [ 34465 ] | |
| Attachment | New: lazy_BuildReference.MODE-2.JPG [ 34466 ] |
Dilip Mahadevappa
added a comment - dbeckham Thanks for checking. We did some more investigation and here are our findings.
We have a got internal Test Framework which is kicked in with every builds and it is generating large amount of Environment variables. One of the Environment variable is having a large value as seen in heap-screenshot-1.png ;
We have got some 3k jobs like this with each build.xml ~ 500k.
Noting that there is a major code changes between 1.596.3 and 1.609.1, we did some testing by setting jenkins.model.lazy.BuildReference.MODE to none and weak.
With Core 1.651.3
lazy_BuildReference.MODE.JPG
With core 2.19.1 ( jenkins.model.lazy.BuildReference.MODE unset)
lazy_BuildReference.MODE-2.JPG
As we see in the graph, there is significant amount of GC with jenkins.model.lazy.BuildReference.MODE set to weak in core 1.651.3
Looks like core 2.19.1 with default jenkins.model.lazy.BuildReference.MODE (i.e. soft) does perform well with huge build history.
Any insights in very much appreciated.
Dilip Mahadevappa
added a comment - dbeckham Thanks for checking. We did some more investigation and here are our findings.
We have a got internal Test Framework which is kicked in with every builds and it is generating large amount of Environment variables. One of the Environment variable is having a large value as seen in heap-screenshot-1.png ;
We have got some 3k jobs like this with each build.xml ~ 500k.
Noting that there is a major code changes between 1.596.3 and 1.609.1, we did some testing by setting jenkins.model.lazy.BuildReference.MODE to none and weak .
With Core 1.651.3
lazy_BuildReference.MODE.JPG
With core 2.19.1 ( jenkins.model.lazy.BuildReference.MODE unset)
lazy_BuildReference.MODE-2.JPG
As we see in the graph, there is significant amount of GC with jenkins.model.lazy.BuildReference.MODE set to weak in core 1.651.3
Looks like core 2.19.1 with default jenkins.model.lazy.BuildReference.MODE (i.e. soft) does perform well with huge build history.
Any insights in very much appreciated.
Dilip Mahadevappa
made changes -
| Summary | Original: hudson.model.ParametersAction heavily bloats memory use and size of build.xml by loading all builds TextParameterValue in memory | New: hudson.model.TextParamterheavily bloats memory use and size of build.xml by loading all builds TextParameterValue in memory |
Could you clarify, are you using extremely long parameter values for your builds?