Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38914

hudson.model.TextParameterValue heavily bloats memory use and size of build.xml by loading all builds TextParameterValue in memorey

      Hi,

      Recently we upgraded from 1.596.3 to 1.651.3 and observing that the later version is consuming lot of heap as soon as it comes up.

      WE have got many jobs with huge build history.. in terms of 4000 builds. Each builds's build.xml is in terms of 500k.

      Whenever the instance is restarted the heap starts to grow and finally the instance crashed with OEM.

      The heap dump shows that values defined under hudson.model.TextParameterValue of each build is loaded into memory!

      Refer screenshots: heap-screenshot-0.png & heap-screenshot-1.png

      Here is the contents of build.xml of one such build. refer screenshot build.xml-screenshot.png

      Is it this commit SECURITY-170 Store initial parameters list for later use which is loading all values?

          [JENKINS-38914] hudson.model.TextParameterValue heavily bloats memory use and size of build.xml by loading all builds TextParameterValue in memorey

          Daniel Beck added a comment -

          Could you clarify, are you using extremely long parameter values for your builds?

          Daniel Beck added a comment - Could you clarify, are you using extremely long parameter values for your builds?

          dbeckham Thanks for checking. We did some more investigation and here are our findings.

          We have a got internal Test Framework which is kicked in with every builds and it is generating large amount of Environment variables. One of the Environment variable is having a large value as seen in heap-screenshot-1.png ;

          We have got some 3k jobs like this with each build.xml ~ 500k.

          Noting that there is a major code changes between 1.596.3 and 1.609.1, we did some testing by setting jenkins.model.lazy.BuildReference.MODE to none and weak.

          With Core 1.651.3
          lazy_BuildReference.MODE.JPG

          With core 2.19.1 ( jenkins.model.lazy.BuildReference.MODE unset)
          lazy_BuildReference.MODE-2.JPG

          As we see in the graph, there is significant amount of GC with jenkins.model.lazy.BuildReference.MODE set to weak in core 1.651.3

          Looks like core 2.19.1 with default jenkins.model.lazy.BuildReference.MODE (i.e. soft) does perform well with huge build history.

          Any insights in very much appreciated.

          Dilip Mahadevappa added a comment - dbeckham Thanks for checking. We did some more investigation and here are our findings. We have a got internal Test Framework which is kicked in with every builds and it is generating large amount of Environment variables. One of the Environment variable is having a large value as seen in heap-screenshot-1.png ; We have got some 3k jobs like this with each build.xml ~ 500k. Noting that there is a major code changes between 1.596.3 and 1.609.1, we did some testing by setting jenkins.model.lazy.BuildReference.MODE to none and weak . With Core 1.651.3 lazy_BuildReference.MODE.JPG With core 2.19.1 ( jenkins.model.lazy.BuildReference.MODE unset) lazy_BuildReference.MODE-2.JPG As we see in the graph, there is significant amount of GC with jenkins.model.lazy.BuildReference.MODE set to weak in core 1.651.3 Looks like core 2.19.1 with default jenkins.model.lazy.BuildReference.MODE (i.e. soft) does perform well with huge build history. Any insights in very much appreciated.

          Daniel Beck added a comment -

          1.633 introduced build history pagination and search (JENKINS-26554), but a bug resulted in all build records for a project getting loaded when the widget was shown. That was only fixed in 2.17.

          This issue needs to be reproduced on current Jenkins releases to be relevant.

          Daniel Beck added a comment - 1.633 introduced build history pagination and search ( JENKINS-26554 ), but a bug resulted in all build records for a project getting loaded when the widget was shown. That was only fixed in 2.17. This issue needs to be reproduced on current Jenkins releases to be relevant.

            Unassigned Unassigned
            dilipm79 Dilip Mahadevappa
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: