[JENKINS-38503] Jenkins memory leak (Metaspace increases over the period) - Jenkins Jira

Details

Type: Bug
Status: Resolved (View Workflow)
Priority: Blocker
Resolution: Duplicate
Component/s: groovy-plugin, pipeline, remoting
Labels:
None
Environment:

Hide
JDK 1.8.0_51 everywhere
Jenkins version 2.7.2
Jenkins Master - RHEL 7.2 (3.10.0-327.10.1.el7.x86_64)
Jenkins Slaves
1) 15 Windows 7 (64 bit) slaves
2) 2 Linux slaves - RHEL 7.2 (3.10.0-327.10.1.el7.x86_64)
3) 2 Linux slaves - RHEL 6.5 (2.6.32-431.11.2.el6.x86_64)
4) 1 windows slave with Server 2008

Show
JDK 1.8.0_51 everywhere Jenkins version 2.7.2 Jenkins Master - RHEL 7.2 (3.10.0-327.10.1.el7.x86_64) Jenkins Slaves 1) 15 Windows 7 (64 bit) slaves 2) 2 Linux slaves - RHEL 7.2 (3.10.0-327.10.1.el7.x86_64) 3) 2 Linux slaves - RHEL 6.5 (2.6.32-431.11.2.el6.x86_64) 4) 1 windows slave with Server 2008

Similar Issues:

Show

Description

Every week we have to restart jenkins due to OOM. Continuously Metaspace increases and the number of classes loaded are also increasing.

Earlier we had this problem with both master and windows slaves. But we figured out that disabling the JNLP3 protocol on windows slave and not running groovy script on slaves solved the problem. But. it is still a problem on master and restarting a master is not anyone is expecting.

We recently upgraded our jenkins environment to 2.7.2 and took all the latest plugins.
We have moved all our flow jobs to pipeline except one.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

heap.jpg
349 kB
2016-09-27 12:02
metaspace.jpg
246 kB
2016-09-27 12:01

Issue Links

duplicates

JENKINS-33358 Groovy and PermGen memory leak

Resolved

Activity

Ascending order - Click to sort in descending order

subbarao p added a comment - 2016-09-27 12:05

increase your jenkins memory size

subbarao p added a comment - 2016-09-27 12:05 increase your jenkins memory size

varun shrivastava added a comment - 2016-09-27 12:06

you can refer to attachment see how metaspace is increasing

varun shrivastava added a comment - 2016-09-27 12:06 you can refer to attachment see how metaspace is increasing

varun shrivastava added a comment - 2016-09-27 12:07

i have given 4GB heap

varun shrivastava added a comment - 2016-09-27 12:07 i have given 4GB heap

Daniel Beck added a comment - 2016-09-27 20:44

Duplicate of ~~JENKINS-33358~~?

Daniel Beck added a comment - 2016-09-27 20:44 Duplicate of JENKINS-33358 ?

varun shrivastava added a comment - 2016-09-29 11:14 - edited

After generating the heap dump and analyzing those as given in link https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+OutOfMemoryError

I found the below instances and a lot of them
this - value: groovy.lang.GroovyClassLoader #28
<- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #28
<- <classLoader> - class: get_oas_releases$_run_closure5, value: groovy.lang.GroovyClassLoader$InnerLoader #50
<- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: get_oas_releases$_run_closure5 class get_oas_releases$_run_closure5
<- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #2204
<- [10548] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #2198
<- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #563500
<- [9] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #7
<- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1
<- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1
<- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1
<- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo
<- elementData - class: java.util.Vector, value: java.lang.Object[] #32667
<- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14

this - value: groovy.lang.GroovyClassLoader #39
<- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #39
<- <classLoader> - class: Script1, value: groovy.lang.GroovyClassLoader$InnerLoader #61
<- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: Script1 class Script1
<- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #987
<- [1] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #2129
<- [6906] - class: java.lang.Object[], value: java.lang.Object[] #825722
<- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #563500
<- [9] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #7
<- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1
<- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1
<- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1
<- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo
<- elementData - class: java.util.Vector, value: java.lang.Object[] #32667
<- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14

this - value: groovy.lang.GroovyClassLoader #106
<- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #106
<- <classLoader> - class: get_oas_releases$_run_closure2, value: groovy.lang.GroovyClassLoader$InnerLoader #128
<- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: get_oas_releases$_run_closure2 class get_oas_releases$_run_closure2
<- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #2146
<- [14122] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #2113
<- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #577746
<- [8] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #8
<- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1
<- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1
<- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1
<- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo
<- elementData - class: java.util.Vector, value: java.lang.Object[] #32667
<- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14

this - value: groovy.lang.GroovyClassLoader #145
<- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #145
<- <classLoader> - class: Script1, value: groovy.lang.GroovyClassLoader$InnerLoader #167
<- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: Script1 class Script1
<- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #1099
<- [5219] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #883
<- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #557473
<- [15] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #1
<- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1
<- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1
<- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1
<- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo
<- elementData - class: java.util.Vector, value: java.lang.Object[] #32667
<- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14

varun shrivastava added a comment - 2016-09-29 11:14 - edited After generating the heap dump and analyzing those as given in link https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+OutOfMemoryError I found the below instances and a lot of them this - value: groovy.lang.GroovyClassLoader #28 <- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #28 <- <classLoader> - class: get_oas_releases$_run_closure5, value: groovy.lang.GroovyClassLoader$InnerLoader #50 <- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: get_oas_releases$_run_closure5 class get_oas_releases$_run_closure5 <- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #2204 <- [10548] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #2198 <- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #563500 <- [9] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #7 <- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1 <- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1 <- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1 <- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo <- elementData - class: java.util.Vector, value: java.lang.Object[] #32667 <- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14 this - value: groovy.lang.GroovyClassLoader #39 <- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #39 <- <classLoader> - class: Script1, value: groovy.lang.GroovyClassLoader$InnerLoader #61 <- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: Script1 class Script1 <- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #987 <- [1] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #2129 <- [6906] - class: java.lang.Object[], value: java.lang.Object[] #825722 <- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #563500 <- [9] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #7 <- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1 <- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1 <- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1 <- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo <- elementData - class: java.util.Vector, value: java.lang.Object[] #32667 <- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14 this - value: groovy.lang.GroovyClassLoader #106 <- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #106 <- <classLoader> - class: get_oas_releases$_run_closure2, value: groovy.lang.GroovyClassLoader$InnerLoader #128 <- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: get_oas_releases$_run_closure2 class get_oas_releases$_run_closure2 <- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #2146 <- [14122] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #2113 <- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #577746 <- [8] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #8 <- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1 <- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1 <- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1 <- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo <- elementData - class: java.util.Vector, value: java.lang.Object[] #32667 <- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14 this - value: groovy.lang.GroovyClassLoader #145 <- delegate - class: groovy.lang.GroovyClassLoader$InnerLoader, value: groovy.lang.GroovyClassLoader #145 <- <classLoader> - class: Script1, value: groovy.lang.GroovyClassLoader$InnerLoader #167 <- klazz - class: org.codehaus.groovy.reflection.ClassInfo, value: Script1 class Script1 <- value - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue, value: org.codehaus.groovy.reflection.ClassInfo #1099 <- [5219] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue #883 <- table - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment, value: java.lang.Object[] #557473 <- [15] - class: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[], value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment #1 <- segments - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map, value: org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[] #1 <- map - class: org.codehaus.groovy.reflection.GroovyClassValuePreJava7, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map #1 <- globalClassValue - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.GroovyClassValuePreJava7 #1 <- [1632] - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo class ClassInfo <- elementData - class: java.util.Vector, value: java.lang.Object[] #32667 <- classes (Java frame) - class: org.eclipse.jetty.webapp.WebAppClassLoader, value: java.util.Vector #14

Daniel Beck added a comment - 2016-09-29 11:24

Looks like ~~JENKINS-33358~~.

Daniel Beck added a comment - 2016-09-29 11:24 Looks like JENKINS-33358 .

Jenkins

Jenkins memory leak (Metaspace increases over the period)

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates