Metaspace leak on Jenkins 2.346.1/Java11

This issue is archived. You can view it, but you can't modify it. Learn more

XMLWordPrintable

      Running Jenkins with hundreds of nodes and jobs, monitoring metaspace and heap with
      jcmd GC.heap_info,
      I got growing used metaspace while used heap is stable (after gc). see attached picture.

      My java options:
      -Xmx400g -Xms200g -XX:MetaspaceSize=8192m -XX:MaxMetaspaceSize=8192m -XX:+ExplicitGCInvokesConcurrent -XX:+UseG1GC

      Jenkins ran out of metaspace and unresponsive finally:
      2022-07-15 16:36:52.921  [Pipeline] End of Pipeline
      2022-07-15 16:36:52.967  java.lang.OutOfMemoryError: Metaspace
      2022-07-15 16:36:52.973  Finished: FAILURE

      top class_stats after 10 hours. the used metaspace increased around 22%, see attached picture.

      metaspace issue disappeared on 7/19 suddenly.

        1. class_stats.png
          class_stats.png
          139 kB
        2. metaspace.png
          metaspace.png
          389 kB
        3. metaspace-issue-gone.png
          metaspace-issue-gone.png
          310 kB
        4. Snipaste_2022-08-15_17-02-34.png
          Snipaste_2022-08-15_17-02-34.png
          91 kB

            Assignee:
            Unassigned
            Reporter:
            Jason Gao
            Archiver:
            Jenkins Service Account

              Created:
              Updated:
              Resolved:
              Archived: