-
Bug
-
Resolution: Won't Do
-
Major
-
Jenkins version: 2.62
Java version "1.8.0_131" (build 1.8.0_131-b11)
Ubuntu 11.10
Command to start Jenkins:
/usr/bin/java -Xmx1024m -Xms1024m -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.rmi.port=1099 -Djava.rmi.server.hostname=10.0.0.66 -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080 --ajp13Port=-1
Log error:
Jun 20, 2017 9:06:48 AM org.eclipse.jetty.server.session.Session beginInvalidate
INFO: Session node0o10k68itvv7217ppw6c9hmqa841 already being invalidated
Jun 20, 2017 9:34:09 AM com.cloudbees.hudson.plugins.folder.computed.FolderCron doRun
WARNING: Cron thread throw an exception
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.regex.Matcher.<init>(Matcher.java:225)
at java.util.regex.Pattern.matcher(Pattern.java:1093)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$2.perform(RoleMap.java:329)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$RoleWalker.walk(RoleMap.java:387)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$RoleWalker.<init>(RoleMap.java:376)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$2.<init>(RoleMap.java:327)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.getMatchingRoles(RoleMap.java:327)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.newMatchingRoleMap(RoleMap.java:283)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:124)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:141)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:136)
at hudson.model.Job.getACL(Job.java:1599)
at hudson.model.AbstractItem.hasPermission(AbstractItem.java:506)
at com.cloudbees.hudson.plugins.folder.AbstractFolder.getItems(AbstractFolder.java:1146)
at hudson.model.Items.getAllItems(Items.java:407)
at hudson.model.Items.getAllItems(Items.java:417)
at hudson.model.Items.getAllItems(Items.java:403)
at jenkins.model.Jenkins.getAllItems(Jenkins.java:1757)
at com.cloudbees.hudson.plugins.folder.computed.FolderCron.checkTriggers(FolderCron.java:117)
at com.cloudbees.hudson.plugins.folder.computed.FolderCron.doRun(FolderCron.java:99)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Jun 20, 2017 9:34:20 AM hudson.triggers.Trigger$Cron doRun
WARNING: Cron thread throw an exception
java.lang.OutOfMemoryError: GC overhead limit exceeded
at com.google.common.cache.AbstractCache$SimpleStatsCounter.<init>(AbstractCache.java:210)
at com.google.common.cache.CacheBuilder$2.get(CacheBuilder.java:170)
at com.google.common.cache.CacheBuilder$2.get(CacheBuilder.java:167)
at com.google.common.cache.LocalCache.<init>(LocalCache.java:296)
at com.google.common.cache.LocalCache$LocalManualCache.<init>(LocalCache.java:4750)
at com.google.common.cache.LocalCache$LocalManualCache.<init>(LocalCache.java:4745)
at com.google.common.cache.CacheBuilder.build(CacheBuilder.java:757)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.<init>(RoleMap.java:86)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.newMatchingRoleMap(RoleMap.java:288)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:124)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:141)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:136)
at hudson.model.Job.getACL(Job.java:1599)
at hudson.model.AbstractItem.hasPermission(AbstractItem.java:506)
at com.cloudbees.hudson.plugins.folder.AbstractFolder.getItems(AbstractFolder.java:1146)
at hudson.model.Items$AllItemsIterable$AllItemsIterator.hasNext(Items.java:583)
at hudson.triggers.Trigger.checkTriggers(Trigger.java:269)
at hudson.triggers.Trigger$Cron.doRun(Trigger.java:226)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.regex.Matcher.<init>(Matcher.java:225)
at java.util.regex.Pattern.matcher(Pattern.java:1093)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$2.perform(RoleMap.java:329)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$RoleWalker.walk(RoleMap.java:387)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$RoleWalker.<init>(RoleMap.java:376)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap$2.<init>(RoleMap.java:327)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.getMatchingRoles(RoleMap.java:327)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleMap.newMatchingRoleMap(RoleMap.java:283)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:124)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:141)
at com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy.getACL(RoleBasedAuthorizationStrategy.java:136)
at hudson.model.Job.getACL(Job.java:1599)
at hudson.model.AbstractItem.hasPermission(AbstractItem.java:506)
at com.cloudbees.hudson.plugins.folder.AbstractFolder.getItems(AbstractFolder.java:1146)
at hudson.model.Items.getAllItems(Items.java:407)
at hudson.model.Items.getAllItems(Items.java:417)
at hudson.model.Items.getAllItems(Items.java:403)
at jenkins.model.Jenkins.getAllItems(Jenkins.java:1757)
at com.cloudbees.hudson.plugins.folder.computed.FolderCron.checkTriggers(FolderCron.java:117)
at com.cloudbees.hudson.plugins.folder.computed.FolderCron.doRun(FolderCron.java:99)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Jun 20, 2017 9:34:32 AM net.bull.javamelody.JavaLogger warn
WARNING: exception while collecting data: java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
at net.bull.javamelody.CounterRequest.clone(CounterRequest.java:420)
at net.bull.javamelody.Collector.collectCounterRequestData(Collector.java:813)
at net.bull.javamelody.Collector.collectCounterRequestsAndErrorsData(Collector.java:759)
at net.bull.javamelody.Collector.collectCounterData(Collector.java:747)
at net.bull.javamelody.Collector.collect(Collector.java:380)
at net.bull.javamelody.Collector.collectWithoutErrors(Collector.java:352)
at net.bull.javamelody.Collector.collectLocalContextWithoutErrors(Collector.java:341)
at net.bull.javamelody.FilterContext$CollectTimerTask.run(FilterContext.java:64)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Jun 20, 2017 9:34:42 AM net.bull.javamelody.JavaLogger warn
WARNING: exception while collecting data
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at hudson.remoting.Channel$2.adapt(Channel.java:862)
at hudson.remoting.Channel$2.adapt(Channel.java:857)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
at net.bull.javamelody.RemoteCallHelper.collectDataByNodeName(RemoteCallHelper.java:172)
at net.bull.javamelody.RemoteCallHelper.collectJavaInformationsListByName(RemoteCallHelper.java:182)
at net.bull.javamelody.NodesCollector.collectWithoutErrorsNow(NodesCollector.java:153)
at net.bull.javamelody.NodesCollector.collectWithoutErrors(NodesCollector.java:142)
at net.bull.javamelody.NodesCollector$1.run(NodesCollector.java:86)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.getDeclaredMethod(Class.java:2128)
at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1475)
at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:72)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:498)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at hudson.remoting.UserRequest.deserialize(UserRequest.java:217)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:250)
at hudson.remoting.Channel$2.adapt(Channel.java:860)
... 9 more
giselasqa Have you tried configuring your Garbage Collector? Or diagnosing this issue more, e.g. by using this guide: https://plumbr.eu/outofmemoryerror/gc-overhead-limit-exceeded
Maybe you just need to increase the memory allowance in your JVM settings
Gisela Nogueira
added a comment - oleg_nenashev I'll do settings on my GC.
Today the problem occurred again and I made a Thread Dump with VisualVM (attached) and printscreen from my environment too (attached).
Gisela Nogueira
added a comment - oleg_nenashev I'll do settings on my GC.
Today the problem occurred again and I made a Thread Dump with VisualVM (attached) and printscreen from my environment too (attached). Gisela Nogueira
added a comment - I've created a heap dump but it is too long for attach them.
Can I see the job executions list in "heap dump" or "thread dump"?
Gisela Nogueira
added a comment - oleg_nenashev ,
I've created a heap dump but it is too long for attach them.
Can I see the job executions list in "heap dump" or "thread dump"? Gisela Nogueira
added a comment - The performance problem with Jenkins occurred again today. All jobs was stopped at the peak moment. I sow my Jenkins and didn't have anyone information about de problem, anything Exception and anything error.
My VisualVm image error:
Gisela Nogueira
added a comment - oleg_nenashev ,
The performance problem with Jenkins occurred again today. All jobs was stopped at the peak moment. I sow my Jenkins and didn't have anyone information about de problem, anything Exception and anything error.
My VisualVm image error:
Gisela Nogueira
added a comment - I had updated my environment to the last Jenkins version (2.66) and the max memory heap to 2048M.
Gisela Nogueira
added a comment - I had updated my environment to the last Jenkins version (2.66) and the max memory heap to 2048M. Gisela Nogueira
added a comment - Hi oleg_nenashev,
I've changed my JVM parameters. I increased JVM memory from default to 2024 MBs and configured the garbage collector.
Jenkins log information:
The problema was started at 04:41 pm but in the jenkins.log haven't any information about the performance problems.
Jun 27, 2017 4:35:39 PM hudson.model.Run execute
INFO: Immobile/IMMOBILE-FINANCEIRO #3 main build action completed: FAILURE
Jun 27, 2017 4:46:45 PM org.eclipse.jetty.server.session.Session beginInvalidate
INFO: Session node01pbn3vxmf3g60180w3ga3638yl24339 already being invalidated
Attached files:
I'll attach more files to help you.
- jenkins_VisualVm_2017.06.27_0441pm.png: Print screen of VisualVm at the problem moment.
- threaddump-2017.06.27_0519pm.tdump: Thread dump of Jenkins service
Follows my JVM arguments:
-Xmx2048m
-Xms1024m
-Djava.awt.headless=true
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.port=1099
-Dcom.sun.management.jmxremote.rmi.port=1099
-Djava.rmi.server.hostname=10.0.0.66
-verbose:gc
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/log/jenkins/
Xloggc:$/var/log/jenkins/gc%t.log
-XX:NumberOfGCLogFiles=5
-XX:+UseGCLogFileRotation
-XX:GCLogFileSize=20m
-XX:+PrintGC
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+PrintHeapAtGC
-XX:+PrintGCCause
-XX:+PrintTenuringDistribution
-XX:+PrintReferenceGC
-XX:+PrintAdaptiveSizePolicy
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:+ParallelRefProcEnabled
-XX:+CMSClassUnloadingEnabled
-XX:+ScavengeBeforeFullGC
-XX:+CMSScavengeBeforeRemark
-XX:NewSize=512m
-XX:MaxNewSize=512m
-XX:NewRatio=2
If you need for more information, you can ask to me.
Gisela Nogueira
added a comment - Hi oleg_nenashev ,
I've changed my JVM parameters. I increased JVM memory from default to 2024 MBs and configured the garbage collector.
Jenkins log information :
The problema was started at 04:41 pm but in the jenkins.log haven't any information about the performance problems.
Part of jenkins.log
Jun 27, 2017 4:35:39 PM hudson.model.Run execute
INFO: Immobile/IMMOBILE-FINANCEIRO #3 main build action completed: FAILURE
Jun 27, 2017 4:46:45 PM org.eclipse.jetty.server.session.Session beginInvalidate
INFO: Session node01pbn3vxmf3g60180w3ga3638yl24339 already being invalidated
Attached files :
I'll attach more files to help you.
jenkins_VisualVm_2017.06.27_0441pm.png: Print screen of VisualVm at the problem moment.
threaddump-2017.06.27_0519pm.tdump: Thread dump of Jenkins service
Follows my JVM arguments :
-Xmx2048m
-Xms1024m
-Djava.awt.headless=true
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.port=1099
-Dcom.sun.management.jmxremote.rmi.port=1099
-Djava.rmi.server.hostname=10.0.0.66
-verbose:gc
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/log/jenkins/
Xloggc:$/var/log/jenkins/gc %t.log
-XX:NumberOfGCLogFiles=5
-XX:+UseGCLogFileRotation
-XX:GCLogFileSize=20m
-XX:+PrintGC
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+PrintHeapAtGC
-XX:+PrintGCCause
-XX:+PrintTenuringDistribution
-XX:+PrintReferenceGC
-XX:+PrintAdaptiveSizePolicy
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:+ParallelRefProcEnabled
-XX:+CMSClassUnloadingEnabled
-XX:+ScavengeBeforeFullGC
-XX:+CMSScavengeBeforeRemark
-XX:NewSize=512m
-XX:MaxNewSize=512m
-XX:NewRatio=2
If you need for more information, you can ask to me. Gisela Nogueira
added a comment - We identified the problem and solved them. 
This problem occurred today at 09:34:09 AM (reported on jenkins.log) and occurred to on 16 June at 5:24:29 PM (reported on jenkins.log.1)