Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19022

GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields

    XMLWordPrintable

Details

    Description

      Hello everyone.

      Months ago, we've noticed a bug/issue with the GIT plug-in. Previously, it was only a minor nuisance but now, it causes each build that we start to use up ~3MB of main memory and ~5MB of disk space in the build.xml.

      The issue is due to the following behaviour of the GIT plug-in:
      For every build that has the GIT SCM defined, it retrieves the list of branches in the remote repository. For each branch, it retrieves the last build in Jenkins that was run against this branch.

      This information is then stored in the Build object in form of the "BuildData" field. This means, that the full list of all branches, plus their last builds is stored in each and every build – thus using up main memory and using up disk space in the "build.xml" file allocated for the build.

      It uses this information to populate a page for the build with the association of branches to builds:
      http://<SERVER>/job/<JOBNAME>/<BUILD-ID>/git/?

      For normal repositories, this data is relatively small, as only a limited number of unmerged branches exist. Unfortunately, we use GIT in an automated manner, where thousands of tags and branches are spawned without merging back into the mainline.

      This means that each build saves several hundred to thousand pointless key-value pairs for GIT branches and Jenkins builds that serve no purpose whatsoever.

      In our case, this means – as outlined above – we waste 3MB of RAM per build and 5 MB of disk space. With 10k builds per day, you can imagine that this is quite a predicament.

      As a workaround, we've written a Jenkins job that removes the tags contained in "<hudson.plugins.git.util.BuildData>" in the "build.xml". This cuts down its size from 5MB down to 16kB (~0.156MB). This of course also greatly boosts the speed of deserealizing the builds from disk.

      Our request would be: Either remove the collections/deserialization of this (from our POV) pointless data, or make its generation optional via a configuration option.

      Best regards,
      Martin Schröder
      Intel Mobile Communications GmbH

      Attachments

        Issue Links

          Activity

            mhschroe Martin Schröder created issue -
            markewaite Mark Waite made changes -
            Field Original Value New Value
            Link This issue is duplicated by JENKINS-30873 [ JENKINS-30873 ]
            jglick Jesse Glick made changes -
            Labels branch build.xml builddata git branch build.xml builddata git performance
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 150469 ] JNJira + In-Review [ 177660 ]
            lwalkera Laine Walker-Avina made changes -
            Labels branch build.xml builddata git performance branch build.xml builddata git multi-job performance
            lwalkera Laine Walker-Avina made changes -
            Labels branch build.xml builddata git multi-job performance branch build.xml builddata disk git multi-job performance
            aheritier Arnaud Héritier made changes -
            Assignee Nicolas De Loof [ ndeloof ]
            aheritier Arnaud Héritier made changes -
            Assignee Nicolas De Loof [ ndeloof ]
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-29482 [ JENKINS-29482 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-32218 [ JENKINS-32218 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-17348 [ JENKINS-17348 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-17348 [ JENKINS-17348 ]
            jglick Jesse Glick made changes -
            Assignee Nicolas De Loof [ ndeloof ]
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            jglick Jesse Glick made changes -
            Link This issue is blocking JENKINS-41074 [ JENKINS-41074 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-18588 [ JENKINS-18588 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "Remove Git Plugin BuildsByBranch BuildData (Web Link)" [ 15422 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 472 (Web Link)" [ 15423 ]
            markewaite Mark Waite made changes -
            Remote Link This issue links to "Removing git plugin BuildData (Web Link)" [ 17982 ]
            markewaite Mark Waite made changes -
            Remote Link This issue links to "Removing git plugin BuildData (Web Link)" [ 17982 ]
            markewaite Mark Waite made changes -
            Status Reopened [ 4 ] Open [ 1 ]
            cloudbees CloudBees Inc. made changes -
            Remote Link This issue links to "CloudBees Internal OSS-724 (Web Link)" [ 18840 ]
            jekeller Jacob Keller made changes -
            Link This issue is duplicated by JENKINS-47789 [ JENKINS-47789 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 568 (Web Link)" [ 22078 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 568 (Web Link)" [ 22078 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 578 (Web Link)" [ 22079 ]
            markewaite Mark Waite made changes -
            Link This issue is related to JENKINS-56838 [ JENKINS-56838 ]
            tsvi Tsvi Mostovicz made changes -
            Link This issue is duplicated by JENKINS-56838 [ JENKINS-56838 ]

            People

              Unassigned Unassigned
              mhschroe Martin Schröder
              Votes:
              38 Vote for this issue
              Watchers:
              88 Start watching this issue

              Dates

                Created:
                Updated: