-
Bug
-
Resolution: Fixed
-
Critical
-
None
-
Running on Solaris 10 (sparc). Tomcat 6.0.29. Jenkins 401 and 404. This was not an issue in Hudson 380. JVM is set to 1.5GB on one system and 2GB on another.
-
Powered by SuggestiMate
Run a build that generates a large console output. Say around 10MB or more. View the console log for the build. It should work just fine. Add the timestamper and run the build again (version 0.6). View the console log. Select to view full log. This frequently crashes Jenkins/Tomcat. Sometimes I actually get the whole log. This has occurred after a restart as well as after it has been running for a bit. I don't see the issue when view logs without the timestamper enabled (at least not yet).
[JENKINS-9349] Viewing large console logs with timestamper plugin cause Jenkins to crash
Thanks for the quick reply. The following is in the catalina.out.
#
- A fatal error has been detected by the Java Runtime Environment:
# - java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?
# - Internal Error (allocation.cpp:166), pid=2009, tid=11
- Error: ChunkPool::allocate
# - JRE version: 6.0_24-b07
- Java VM: Java HotSpot(TM) Server VM (19.1-b02 mixed mode solaris-sparc )
- An error report file with more information is saved as:
- /work1/SFCM/sandbox/apache-tomcat-6.0.29/bin/hs_err_pid2009.log
# - If you would like to submit a bug report, please visit:
- http://java.sun.com/webapps/bugreport/crash.jsp
#
Here is a crash stack trace I got in the browser when just viewing a Console output:
Status Code: 500 Exception: Stacktrace: java.lang.OutOfMemoryError at java.util.zip.Deflater.init(Native Method) at java.util.zip.Deflater.(Deflater.java:124) at java.util.zip.GZIPOutputStream.(GZIPOutputStream.java:46) at java.util.zip.GZIPOutputStream.(GZIPOutputStream.java:58) at hudson.console.AnnotatedLargeText.writeHtmlTo(AnnotatedLargeText.java:160) at hudson.console.AnnotatedLargeText.writeLogTo(AnnotatedLargeText.java:141) at org.kohsuke.stapler.framework.io.LargeText.doProgressText(LargeText.java:214) at hudson.console.AnnotatedLargeText.doProgressiveHtml(AnnotatedLargeText.java:91) at sun.reflect.GeneratedMethodAccessor521.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:282) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:149) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:88) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:104) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:196) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:196) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:234) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:234) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:234) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:234) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:477) at org.kohsuke.stapler.Stapler.service(Stapler.java:159) at javax.servlet.http.HttpServlet.service(HttpServlet.java:45) at winstone.ServletConfiguration.execute(ServletConfiguration.java:249) at winstone.RequestDispatcher.forward(RequestDispatcher.java:335) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:378) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:157) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:131) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:83) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:74) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at winstone.FilterConfiguration.execute(FilterConfiguration.java:195) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47) at winstone.FilterConfiguration.execute(FilterConfiguration.java:195) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:61) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:195) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:195) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368) at winstone.RequestDispatcher.forward(RequestDispatcher.java:333) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:244) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150) at java.lang.Thread.run(Thread.java:619)
And the following message was displayed in the jenkins log:
# # A fatal error has been detected by the Java Runtime Environment: # # java.lang.OutOfMemoryError: requested 59208 bytes for Chunk::new. Out of swap space? # # Internal Error (allocation.cpp:215), pid=9617, tid=11 # Error: Chunk::new # # JRE version: 6.0_19-b04 # Java VM: Java HotSpot(TM) Server VM (16.2-b04 mixed mode solaris-sparc ) # An error report file with more information is saved as: # /opt/hudson/hs_err_pid9617.log # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp #
The Server is a Solaris 10 Sparc with Jenkins 1.432 with Winstone, with timestamper plugin 1.2.2
Irrespective of whether the timestamper plugin is causing jenkins to use too much memory the error in the log is that the operating system has run out of memory to give to Jenkins
# A fatal error has been detected by the Java Runtime Environment: # # java.lang.OutOfMemoryError: requested 59208 bytes for Chunk::new. Out of swap space?
You need to ensure that your operating system has enough physical memory (or swap space) to allocate the memory that you have told Java it can use.
That's the thing I don't think that is the problem because this is running on a dedicated SunFire v445 server with 16Gb of Ram + 16Gb of Swap.
Jenkins was started with -Xmx3500m, but there was just one build running.
I do have many slaves (15) with 2-3 threads each but they were all idle.
Right now I just removed the -Xmx flag but I think the default is less thant the limit I've set.
Attachment | New: hs_err_pid9617.log [ 20907 ] |
Well the problem was that the JVM could not allocate any more memory. The error you posted was the JVM crashing because it could not allocate memory. It maybe that the OS is limiting the amount of memory that the JVM can use (do you have any ulimits or other process limits) set?
You need to fix that problem before Jenkins can be blamed
Does this information helps?
-bash-3.00$ ulimit -a core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited open files (-n) 256 pipe size (512 bytes, -p) 10 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 29995 virtual memory (kbytes, -v) unlimited
None of those limits seem to be a problem.
It is likely to be a real virtual memory shortage in that case. You say you are using Solaris so I presume that you will be using tmpfs for your /tmp storage. In that case it might be that /tmp is getting filled up (does your job use /tmp for any storage? Jenkins may be storing in /tmp too). You likely need to monitor virtual memory/swap usage and /tmp free space and work out what is using the space.
Until you eliminate local machine configuration issues there isn't a lot that can be attributed to Jenkins.
Any errors in the command-line output?