Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38503

Jenkins memory leak (Metaspace increases over the period)

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Blocker Blocker
    • None

      Every week we have to restart jenkins due to OOM. Continuously Metaspace increases and the number of classes loaded are also increasing.

      Earlier we had this problem with both master and windows slaves. But we figured out that disabling the JNLP3 protocol on windows slave and not running groovy script on slaves solved the problem. But. it is still a problem on master and restarting a master is not anyone is expecting.

      We recently upgraded our jenkins environment to 2.7.2 and took all the latest plugins.
      We have moved all our flow jobs to pipeline except one.

        1. heap.jpg
          heap.jpg
          349 kB
        2. metaspace.jpg
          metaspace.jpg
          246 kB

          [JENKINS-38503] Jenkins memory leak (Metaspace increases over the period)

          subbarao p added a comment -

          increase your jenkins memory size

          subbarao p added a comment - increase your jenkins memory size

          you can refer to attachment see how metaspace is increasing

          varun shrivastava added a comment - you can refer to attachment see how metaspace is increasing

          i have given 4GB heap

          varun shrivastava added a comment - i have given 4GB heap

          Daniel Beck added a comment -

          Duplicate of JENKINS-33358?

          Daniel Beck added a comment - Duplicate of JENKINS-33358 ?

          varun shrivastava added a comment - - edited

          After generating the heap dump and analyzing those as given in link https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+OutOfMemoryError

          I found the below instances and a lot of them
          this - value: groovy.lang.GroovyClassLoader #28
          <- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #28
          <- <classLoader> - class: get_oas_releases$_run_closure5, value: groovy.lang.GroovyClassLoader$InnerLoader #50
          <- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: get_oas_releases$_run_closure5 class get_oas_releases$_run_closure5
          <- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #2204
          <- [10548] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #2198
          <- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #563500
          <- [9] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #7
          <- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1
          <- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1
          <- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1
          <- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo
          <- elementData - class: java.util.Vector, value: java.lang.Object[] #32667
          <- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14

          this - value: groovy.lang.GroovyClassLoader #39
          <- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #39
          <- <classLoader> - class: Script1, value: groovy.lang.GroovyClassLoader$InnerLoader #61
          <- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: Script1 class Script1
          <- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #987
          <- [1] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #2129
          <- [6906] - class: java.lang.Object[], value: java.lang.Object[] #825722
          <- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #563500
          <- [9] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #7
          <- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1
          <- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1
          <- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1
          <- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo
          <- elementData - class: java.util.Vector, value: java.lang.Object[] #32667
          <- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14

          this - value: groovy.lang.GroovyClassLoader #106
          <- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #106
          <- <classLoader> - class: get_oas_releases$_run_closure2, value: groovy.lang.GroovyClassLoader$InnerLoader #128
          <- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: get_oas_releases$_run_closure2 class get_oas_releases$_run_closure2
          <- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #2146
          <- [14122] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #2113
          <- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #577746
          <- [8] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #8
          <- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1
          <- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1
          <- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1
          <- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo
          <- elementData - class: java.util.Vector, value: java.lang.Object[] #32667
          <- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14

          this - value: groovy.lang.GroovyClassLoader #145
          <- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #145
          <- <classLoader> - class: Script1, value: groovy.lang.GroovyClassLoader$InnerLoader #167
          <- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: Script1 class Script1
          <- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #1099
          <- [5219] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #883
          <- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #557473
          <- [15] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #1
          <- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1
          <- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1
          <- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1
          <- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo
          <- elementData - class: java.util.Vector, value: java.lang.Object[] #32667
          <- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14

          varun shrivastava added a comment - - edited After generating the heap dump and analyzing those as given in link https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+OutOfMemoryError I found the below instances and a lot of them this - value: groovy.lang.GroovyClassLoader #28 <- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #28 <- <classLoader> - class: get_oas_releases$_run_closure5, value: groovy.lang.GroovyClassLoader$InnerLoader #50 <- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: get_oas_releases$_run_closure5 class get_oas_releases$_run_closure5 <- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #2204 <- [10548] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #2198 <- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #563500 <- [9] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #7 <- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1 <- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1 <- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1 <- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo <- elementData - class: java.util.Vector, value: java.lang.Object[] #32667 <- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14 this - value: groovy.lang.GroovyClassLoader #39 <- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #39 <- <classLoader> - class: Script1, value: groovy.lang.GroovyClassLoader$InnerLoader #61 <- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: Script1 class Script1 <- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #987 <- [1] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #2129 <- [6906] - class: java.lang.Object[], value: java.lang.Object[] #825722 <- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #563500 <- [9] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #7 <- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1 <- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1 <- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1 <- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo <- elementData - class: java.util.Vector, value: java.lang.Object[] #32667 <- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14 this - value: groovy.lang.GroovyClassLoader #106 <- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #106 <- <classLoader> - class: get_oas_releases$_run_closure2, value: groovy.lang.GroovyClassLoader$InnerLoader #128 <- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: get_oas_releases$_run_closure2 class get_oas_releases$_run_closure2 <- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #2146 <- [14122] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #2113 <- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #577746 <- [8] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #8 <- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1 <- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1 <- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1 <- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo <- elementData - class: java.util.Vector, value: java.lang.Object[] #32667 <- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14 this - value: groovy.lang.GroovyClassLoader #145 <- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #145 <- <classLoader> - class: Script1, value: groovy.lang.GroovyClassLoader$InnerLoader #167 <- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: Script1 class Script1 <- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #1099 <- [5219] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #883 <- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #557473 <- [15] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #1 <- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1 <- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1 <- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1 <- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo <- elementData - class: java.util.Vector, value: java.lang.Object[] #32667 <- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14

          Daniel Beck added a comment -

          Looks like JENKINS-33358.

          Daniel Beck added a comment - Looks like JENKINS-33358 .

            vjuranek vjuranek
            varun_shrivastava_1987 varun shrivastava
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: