Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-69022

Metaspace leak on Jenkins 2.346.1/Java11

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Critical Critical
    • core
    • docker jenkins/jenkins:2.346.1-lts-jdk11 on CentOS 7

      Running Jenkins with hundreds of nodes and jobs, monitoring metaspace and heap with
      jcmd GC.heap_info,
      I got growing used metaspace while used heap is stable (after gc). see attached picture.

      My java options:
      -Xmx400g -Xms200g -XX:MetaspaceSize=8192m -XX:MaxMetaspaceSize=8192m -XX:+ExplicitGCInvokesConcurrent -XX:+UseG1GC

      Jenkins ran out of metaspace and unresponsive finally:
      2022-07-15 16:36:52.921  [Pipeline] End of Pipeline
      2022-07-15 16:36:52.967  java.lang.OutOfMemoryError: Metaspace
      2022-07-15 16:36:52.973  Finished: FAILURE

      top class_stats after 10 hours. the used metaspace increased around 22%, see attached picture.

      metaspace issue disappeared on 7/19 suddenly.

        1. class_stats.png
          class_stats.png
          139 kB
        2. metaspace.png
          metaspace.png
          389 kB
        3. metaspace-issue-gone.png
          metaspace-issue-gone.png
          310 kB
        4. Snipaste_2022-08-15_17-02-34.png
          Snipaste_2022-08-15_17-02-34.png
          91 kB

            Unassigned Unassigned
            loblab Jason Gao
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: