Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29664

Jenkins java heap full of Strings -> Running out of Memory on Large Jenkins Deploy

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • p4-plugin

    Description

      We run a Jenkins master server with over 3500 jobs defined. We've been running an 8GB heap size for some time, we average arorund 4-6GB used and collections tend to happen reasonably. We recently updated to Jenkins 1.618 (from 1.594) and in the past week our master server has hung in garbage collection twice. Oddly upon restart, it will sometimes immediately hang again.

      We took a heap dump of todays failure to see if we had a reference leak or misbehaving plugin and couldn't find anything obvious. However, we have a huge amount of memory allocated to retaining strings. Java.lang.string/char consume %70 of our memory. While many of the strings seem to be job descriptions, recent changelists etc (things found on the job details page) there is a huge number of strings dedicated to what looks like errors and stack traces I'd normally expect to find in a log (see the attached file for a few examples associated with: org.apache.commons.jelly.JellyTagException).

      I'm not sure if this is normal behavior or not (never analyzed these part of Jenkins before). We're still trying to figure out what retains these strings. Given that we think this may be "normal" we've bumped our heap to 12GB's to see if we've just gotten so big we need more memory.

      I've also attached the monitoring graphs (as images) of relevant monitors. I can, if it's useful, upload the 8GB heap dump somewhere.

      Attachments

        1. jenkins.st.4
          1.05 MB
          Maxfield Stewart
        2. jenkinserr.txt
          14 kB
          Maxfield Stewart
        3. p4changelist.png
          64 kB
          Maxfield Stewart
        4. unnamed.png
          142 kB
          Maxfield Stewart
        5. usedMemory.png
          43 kB
          Maxfield Stewart
        6. usedMemory1week.png
          31 kB
          Maxfield Stewart
        7. usedPhysicalMemorySize.png
          25 kB
          Maxfield Stewart
        8. usedPhysicalMemorySize1week.png
          26 kB
          Maxfield Stewart

        Activity

          People

            Unassigned Unassigned
            maxfields2000 Maxfield Stewart
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: