Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-69022

Metaspace leak on Jenkins 2.346.1/Java11

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Critical Critical
    • core
    • docker jenkins/jenkins:2.346.1-lts-jdk11 on CentOS 7

      Running Jenkins with hundreds of nodes and jobs, monitoring metaspace and heap with
      jcmd GC.heap_info,
      I got growing used metaspace while used heap is stable (after gc). see attached picture.

      My java options:
      -Xmx400g -Xms200g -XX:MetaspaceSize=8192m -XX:MaxMetaspaceSize=8192m -XX:+ExplicitGCInvokesConcurrent -XX:+UseG1GC

      Jenkins ran out of metaspace and unresponsive finally:
      2022-07-15 16:36:52.921  [Pipeline] End of Pipeline
      2022-07-15 16:36:52.967  java.lang.OutOfMemoryError: Metaspace
      2022-07-15 16:36:52.973  Finished: FAILURE

      top class_stats after 10 hours. the used metaspace increased around 22%, see attached picture.

      metaspace issue disappeared on 7/19 suddenly.

        1. Snipaste_2022-08-15_17-02-34.png
          Snipaste_2022-08-15_17-02-34.png
          91 kB
        2. metaspace-issue-gone.png
          metaspace-issue-gone.png
          310 kB
        3. class_stats.png
          class_stats.png
          139 kB
        4. metaspace.png
          metaspace.png
          389 kB

          [JENKINS-69022] Metaspace leak on Jenkins 2.346.1/Java11

          Jason Gao created issue -
          Jason Gao made changes -
          Description Original: Running Jenkins with hundreds of nodes and jobs, monitoring metaspace and heap with New: Running Jenkins with hundreds of nodes and jobs, monitoring metaspace and heap with GC.heap_info
          Jason Gao made changes -
          Description Original: Running Jenkins with hundreds of nodes and jobs, monitoring metaspace and heap with GC.heap_info New: Running Jenkins with hundreds of nodes and jobs, monitoring metaspace and heap with
          jcmd GC.heap_info,
          I got growing used metaspace while used heap is plain.
          Jason Gao made changes -
          Attachment New: metaspace.png [ 58486 ]
          Jason Gao made changes -
          Description Original: Running Jenkins with hundreds of nodes and jobs, monitoring metaspace and heap with
          jcmd GC.heap_info,
          I got growing used metaspace while used heap is plain.           		New: !metaspace.png! Running Jenkins with hundreds of nodes and jobs, monitoring metaspace and heap with
          jcmd GC.heap_info,
          I got growing used metaspace while used heap is plain. see attached picture.
          Jason Gao made changes -
          Description Original: !metaspace.png! Running Jenkins with hundreds of nodes and jobs, monitoring metaspace and heap with
          jcmd GC.heap_info,
          I got growing used metaspace while used heap is plain. see attached picture.           		New: !metaspace.png! Running Jenkins with hundreds of nodes and jobs, monitoring metaspace and heap with
          jcmd GC.heap_info,
          I got growing used metaspace while used heap is plain. see attached picture.

          My java options:
          -Xmx400g -Xms200g -XX:MetaspaceSize=8192m -XX:MaxMetaspaceSize=8192m -XX:+ExplicitGCInvokesConcurrent -XX:+UseG1GC

          Jenkins will run out of metaspace and stop response finally.
          Jason Gao made changes -
          Description Original: !metaspace.png! Running Jenkins with hundreds of nodes and jobs, monitoring metaspace and heap with
          jcmd GC.heap_info,
          I got growing used metaspace while used heap is plain. see attached picture.

          My java options:
          -Xmx400g -Xms200g -XX:MetaspaceSize=8192m -XX:MaxMetaspaceSize=8192m -XX:+ExplicitGCInvokesConcurrent -XX:+UseG1GC

          Jenkins will run out of metaspace and stop response finally.           		New: Running Jenkins with hundreds of nodes and jobs, monitoring metaspace and heap with
          jcmd GC.heap_info,
          I got growing used metaspace while used heap is plain. see attached picture.

          My java options:
          -Xmx400g -Xms200g -XX:MetaspaceSize=8192m -XX:MaxMetaspaceSize=8192m -XX:+ExplicitGCInvokesConcurrent -XX:+UseG1GC

          Jenkins will run out of metaspace and stop response finally.
          Jason Gao made changes -
          Description Original: Running Jenkins with hundreds of nodes and jobs, monitoring metaspace and heap with
          jcmd GC.heap_info,
          I got growing used metaspace while used heap is plain. see attached picture.

          My java options:
          -Xmx400g -Xms200g -XX:MetaspaceSize=8192m -XX:MaxMetaspaceSize=8192m -XX:+ExplicitGCInvokesConcurrent -XX:+UseG1GC

          Jenkins will run out of metaspace and stop response finally.           		New: Running Jenkins with hundreds of nodes and jobs, monitoring metaspace and heap with
          jcmd GC.heap_info,
          I got growing used metaspace while used heap is stable (after gc). see attached picture.

          My java options:
          -Xmx400g -Xms200g -XX:MetaspaceSize=8192m -XX:MaxMetaspaceSize=8192m -XX:+ExplicitGCInvokesConcurrent -XX:+UseG1GC

          Jenkins ran out of metaspace and unresponsive finally:
          2022-07-15 16:36:52.921  [Pipeline] End of Pipeline
          2022-07-15 16:36:52.967  java.lang.OutOfMemoryError: Metaspace
          2022-07-15 16:36:52.973  Finished: FAILURE
          Jason Gao made changes -
          Labels New: 2.346.1 leak memory metaspace

          Mark Waite added a comment -

          loblab could you review the comments and discussion in JENKINS-63766 to see if the condition you're seeing is related to a metaspace memory leak bug in JDK 11? A workaround was implemented in Jenkins 2.346.1 and I believe that a fix is expected in Java 11.0.16

          Mark Waite added a comment - loblab could you review the comments and discussion in JENKINS-63766 to see if the condition you're seeing is related to a metaspace memory leak bug in JDK 11? A workaround was implemented in Jenkins 2.346.1 and I believe that a fix is expected in Java 11.0.16

            Unassigned Unassigned
            loblab Jason Gao
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: