Not sure if this will help, but in our /etc/sysconfig/jenkins you can added:
JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Dgroovy.use.classvalue=true -Xms4096m -Xmx4096m -XX:MaxPermSize=1024m -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -Dhudson.model.ParametersAction.keepUndefinedParameters=true"
We also run some crazy groovy code in some of jenkins jobs and we tend to run out of memory too, so we've installed this plugin to help us track java resources:
https://wiki.jenkins-ci.org/display/JENKINS/Monitoring
And finally, we periodically (once per hour i think) run this groovy script to clean things up:
import net.bull.javamelody.*;
before = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();
System.gc();
after = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();
println I18N.getFormattedString("ramasse_miette_execute", Math.round((before - after) / 1024));
And we launch it like this:
java -jar jenkins-cli.jar -noCertificateCheck -i id_rsa -s JENKINS_URL groovy my_groovy_script.groovy'
You can probably put that groovy script as a jenkins job have it run periodically.
Ever since we we run that System.gc() command, we've never run out of memory. We run 100s of jobs a day a AWS t2.medium without any downtime for months at a time. Before I did these things, we were running on a huge instance and we had to restart it every week or so.
Hope this helps!
Memory dump might be partly or broken:
/usr/java/latest_jdk8/bin/jmap -F -dump:format=b,file=jenkins_stuck.hprof 21247
Attaching to process ID 21247, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.45-b02
Dumping heap to jenkins_stuck.hprof ...
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at sun.tools.jmap.JMap.runTool(JMap.java:201)
at sun.tools.jmap.JMap.main(JMap.java:130)
Caused by: sun.jvm.hotspot.utilities.AssertionFailure: can not get class data for java/lang/UNIXProcess$$Lambda$50x00000007c0119428
at sun.jvm.hotspot.utilities.Assert.that(Assert.java:32)
at sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeInstance(HeapHprofBinWriter.java:803)
at sun.jvm.hotspot.utilities.AbstractHeapGraphWriter$1.doObj(AbstractHeapGraphWriter.java:95)
at sun.jvm.hotspot.oops.ObjectHeap.iterateLiveRegions(ObjectHeap.java:353)
at sun.jvm.hotspot.oops.ObjectHeap.iterate(ObjectHeap.java:171)
at sun.jvm.hotspot.utilities.AbstractHeapGraphWriter.write(AbstractHeapGraphWriter.java:51)
at sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:433)
at sun.jvm.hotspot.tools.HeapDumper.run(HeapDumper.java:62)
at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
at sun.jvm.hotspot.tools.HeapDumper.main(HeapDumper.java:83)
... 6 more