Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29664

Jenkins java heap full of Strings -> Running out of Memory on Large Jenkins Deploy

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • p4-plugin

      We run a Jenkins master server with over 3500 jobs defined. We've been running an 8GB heap size for some time, we average arorund 4-6GB used and collections tend to happen reasonably. We recently updated to Jenkins 1.618 (from 1.594) and in the past week our master server has hung in garbage collection twice. Oddly upon restart, it will sometimes immediately hang again.

      We took a heap dump of todays failure to see if we had a reference leak or misbehaving plugin and couldn't find anything obvious. However, we have a huge amount of memory allocated to retaining strings. Java.lang.string/char consume %70 of our memory. While many of the strings seem to be job descriptions, recent changelists etc (things found on the job details page) there is a huge number of strings dedicated to what looks like errors and stack traces I'd normally expect to find in a log (see the attached file for a few examples associated with: org.apache.commons.jelly.JellyTagException).

      I'm not sure if this is normal behavior or not (never analyzed these part of Jenkins before). We're still trying to figure out what retains these strings. Given that we think this may be "normal" we've bumped our heap to 12GB's to see if we've just gotten so big we need more memory.

      I've also attached the monitoring graphs (as images) of relevant monitors. I can, if it's useful, upload the 8GB heap dump somewhere.

        1. usedPhysicalMemorySize1week.png
          usedPhysicalMemorySize1week.png
          26 kB
        2. usedPhysicalMemorySize.png
          usedPhysicalMemorySize.png
          25 kB
        3. usedMemory1week.png
          usedMemory1week.png
          31 kB
        4. usedMemory.png
          usedMemory.png
          43 kB
        5. unnamed.png
          unnamed.png
          142 kB
        6. p4changelist.png
          p4changelist.png
          64 kB
        7. jenkinserr.txt
          14 kB
        8. jenkins.st.4
          1.05 MB

            Unassigned Unassigned
            maxfields2000 Maxfield Stewart
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: