Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-9349

Viewing large console logs with timestamper plugin cause Jenkins to crash

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: timestamper-plugin
    • Labels:
      None
    • Environment:
      Running on Solaris 10 (sparc). Tomcat 6.0.29. Jenkins 401 and 404. This was not an issue in Hudson 380. JVM is set to 1.5GB on one system and 2GB on another.
    • Similar Issues:

      Description

      Run a build that generates a large console output. Say around 10MB or more. View the console log for the build. It should work just fine. Add the timestamper and run the build again (version 0.6). View the console log. Select to view full log. This frequently crashes Jenkins/Tomcat. Sometimes I actually get the whole log. This has occurred after a restart as well as after it has been running for a bit. I don't see the issue when view logs without the timestamper enabled (at least not yet).

        Attachments

          Activity

          ford30066 Matthew Ford created issue -
          Hide
          stevengbrown Steven G Brown added a comment -

          Any errors in the command-line output?

          Show
          stevengbrown Steven G Brown added a comment - Any errors in the command-line output?
          Hide
          ford30066 Matthew Ford added a comment -

          Thanks for the quick reply. The following is in the catalina.out.

          #

          1. A fatal error has been detected by the Java Runtime Environment:
            #
          2. java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?
            #
          3. Internal Error (allocation.cpp:166), pid=2009, tid=11
          4. Error: ChunkPool::allocate
            #
          5. JRE version: 6.0_24-b07
          6. Java VM: Java HotSpot(TM) Server VM (19.1-b02 mixed mode solaris-sparc )
          7. An error report file with more information is saved as:
          8. /work1/SFCM/sandbox/apache-tomcat-6.0.29/bin/hs_err_pid2009.log
            #
          9. If you would like to submit a bug report, please visit:
          10. http://java.sun.com/webapps/bugreport/crash.jsp
            #
          Show
          ford30066 Matthew Ford added a comment - Thanks for the quick reply. The following is in the catalina.out. # A fatal error has been detected by the Java Runtime Environment: # java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space? # Internal Error (allocation.cpp:166), pid=2009, tid=11 Error: ChunkPool::allocate # JRE version: 6.0_24-b07 Java VM: Java HotSpot(TM) Server VM (19.1-b02 mixed mode solaris-sparc ) An error report file with more information is saved as: /work1/SFCM/sandbox/apache-tomcat-6.0.29/bin/hs_err_pid2009.log # If you would like to submit a bug report, please visit: http://java.sun.com/webapps/bugreport/crash.jsp #
          Hide
          josesa Jose Sa added a comment -

          Here is a crash stack trace I got in the browser when just viewing a Console output:

          Status Code: 500
          
          Exception: 
          Stacktrace: 
          java.lang.OutOfMemoryError
          	at java.util.zip.Deflater.init(Native Method)
          	at java.util.zip.Deflater.(Deflater.java:124)
          	at java.util.zip.GZIPOutputStream.(GZIPOutputStream.java:46)
          	at java.util.zip.GZIPOutputStream.(GZIPOutputStream.java:58)
          	at hudson.console.AnnotatedLargeText.writeHtmlTo(AnnotatedLargeText.java:160)
          	at hudson.console.AnnotatedLargeText.writeLogTo(AnnotatedLargeText.java:141)
          	at org.kohsuke.stapler.framework.io.LargeText.doProgressText(LargeText.java:214)
          	at hudson.console.AnnotatedLargeText.doProgressiveHtml(AnnotatedLargeText.java:91)
          	at sun.reflect.GeneratedMethodAccessor521.invoke(Unknown Source)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:282)
          	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:149)
          	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:88)
          	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:104)
          	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561)
          	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646)
          	at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:196)
          	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561)
          	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646)
          	at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:196)
          	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561)
          	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646)
          	at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:234)
          	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561)
          	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646)
          	at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:234)
          	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561)
          	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646)
          	at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:234)
          	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561)
          	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646)
          	at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:234)
          	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561)
          	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646)
          	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:477)
          	at org.kohsuke.stapler.Stapler.service(Stapler.java:159)
          	at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
          	at winstone.ServletConfiguration.execute(ServletConfiguration.java:249)
          	at winstone.RequestDispatcher.forward(RequestDispatcher.java:335)
          	at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:378)
          	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
          	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:157)
          	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:131)
          	at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:83)
          	at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84)
          	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
          	at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:74)
          	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
          	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
          	at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
          	at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
          	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
          	at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
          	at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
          	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:61)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
          	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
          	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
          	at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
          	at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
          	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
          	at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
          	at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
          	at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
          	at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:244)
          	at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)
          	at java.lang.Thread.run(Thread.java:619)
          

          And the following message was displayed in the jenkins log:

          #
          # A fatal error has been detected by the Java Runtime Environment:
          #
          # java.lang.OutOfMemoryError: requested 59208 bytes for Chunk::new. Out of swap space?
          #
          #  Internal Error (allocation.cpp:215), pid=9617, tid=11
          #  Error: Chunk::new
          #
          # JRE version: 6.0_19-b04
          # Java VM: Java HotSpot(TM) Server VM (16.2-b04 mixed mode solaris-sparc )
          # An error report file with more information is saved as:
          # /opt/hudson/hs_err_pid9617.log
          #
          # If you would like to submit a bug report, please visit:
          #   http://java.sun.com/webapps/bugreport/crash.jsp
          #
          

          The Server is a Solaris 10 Sparc with Jenkins 1.432 with Winstone, with timestamper plugin 1.2.2

          Show
          josesa Jose Sa added a comment - Here is a crash stack trace I got in the browser when just viewing a Console output: Status Code: 500 Exception: Stacktrace: java.lang.OutOfMemoryError at java.util.zip.Deflater.init(Native Method) at java.util.zip.Deflater.(Deflater.java:124) at java.util.zip.GZIPOutputStream.(GZIPOutputStream.java:46) at java.util.zip.GZIPOutputStream.(GZIPOutputStream.java:58) at hudson.console.AnnotatedLargeText.writeHtmlTo(AnnotatedLargeText.java:160) at hudson.console.AnnotatedLargeText.writeLogTo(AnnotatedLargeText.java:141) at org.kohsuke.stapler.framework.io.LargeText.doProgressText(LargeText.java:214) at hudson.console.AnnotatedLargeText.doProgressiveHtml(AnnotatedLargeText.java:91) at sun.reflect.GeneratedMethodAccessor521.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:282) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:149) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:88) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:104) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:196) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:196) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:234) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:234) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:234) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:234) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:477) at org.kohsuke.stapler.Stapler.service(Stapler.java:159) at javax.servlet.http.HttpServlet.service(HttpServlet.java:45) at winstone.ServletConfiguration.execute(ServletConfiguration.java:249) at winstone.RequestDispatcher.forward(RequestDispatcher.java:335) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:378) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:157) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:131) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:83) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:74) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at winstone.FilterConfiguration.execute(FilterConfiguration.java:195) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47) at winstone.FilterConfiguration.execute(FilterConfiguration.java:195) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:61) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:195) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:195) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368) at winstone.RequestDispatcher.forward(RequestDispatcher.java:333) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:244) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150) at java.lang.Thread.run(Thread.java:619) And the following message was displayed in the jenkins log: # # A fatal error has been detected by the Java Runtime Environment: # # java.lang.OutOfMemoryError: requested 59208 bytes for Chunk::new. Out of swap space? # # Internal Error (allocation.cpp:215), pid=9617, tid=11 # Error: Chunk::new # # JRE version: 6.0_19-b04 # Java VM: Java HotSpot(TM) Server VM (16.2-b04 mixed mode solaris-sparc ) # An error report file with more information is saved as: # /opt/hudson/hs_err_pid9617.log # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # The Server is a Solaris 10 Sparc with Jenkins 1.432 with Winstone, with timestamper plugin 1.2.2
          Hide
          oldelvet Richard Mortimer added a comment -

          Irrespective of whether the timestamper plugin is causing jenkins to use too much memory the error in the log is that the operating system has run out of memory to give to Jenkins

          # A fatal error has been detected by the Java Runtime Environment:
          #
          # java.lang.OutOfMemoryError: requested 59208 bytes for Chunk::new. Out of swap space?
          

          You need to ensure that your operating system has enough physical memory (or swap space) to allocate the memory that you have told Java it can use.

          Show
          oldelvet Richard Mortimer added a comment - Irrespective of whether the timestamper plugin is causing jenkins to use too much memory the error in the log is that the operating system has run out of memory to give to Jenkins # A fatal error has been detected by the Java Runtime Environment: # # java.lang.OutOfMemoryError: requested 59208 bytes for Chunk:: new . Out of swap space? You need to ensure that your operating system has enough physical memory (or swap space) to allocate the memory that you have told Java it can use.
          Hide
          josesa Jose Sa added a comment -

          That's the thing I don't think that is the problem because this is running on a dedicated SunFire v445 server with 16Gb of Ram + 16Gb of Swap.

          Jenkins was started with -Xmx3500m, but there was just one build running.

          I do have many slaves (15) with 2-3 threads each but they were all idle.

          Right now I just removed the -Xmx flag but I think the default is less thant the limit I've set.

          Show
          josesa Jose Sa added a comment - That's the thing I don't think that is the problem because this is running on a dedicated SunFire v445 server with 16Gb of Ram + 16Gb of Swap. Jenkins was started with -Xmx3500m, but there was just one build running. I do have many slaves (15) with 2-3 threads each but they were all idle. Right now I just removed the -Xmx flag but I think the default is less thant the limit I've set.
          josesa Jose Sa made changes -
          Field Original Value New Value
          Attachment hs_err_pid9617.log [ 20907 ]
          Hide
          oldelvet Richard Mortimer added a comment -

          Well the problem was that the JVM could not allocate any more memory. The error you posted was the JVM crashing because it could not allocate memory. It maybe that the OS is limiting the amount of memory that the JVM can use (do you have any ulimits or other process limits) set?

          You need to fix that problem before Jenkins can be blamed

          Show
          oldelvet Richard Mortimer added a comment - Well the problem was that the JVM could not allocate any more memory. The error you posted was the JVM crashing because it could not allocate memory. It maybe that the OS is limiting the amount of memory that the JVM can use (do you have any ulimits or other process limits) set? You need to fix that problem before Jenkins can be blamed
          Hide
          josesa Jose Sa added a comment -

          Does this information helps?

          -bash-3.00$ ulimit -a
          core file size        (blocks, -c) unlimited
          data seg size         (kbytes, -d) unlimited
          file size             (blocks, -f) unlimited
          open files                    (-n) 256
          pipe size          (512 bytes, -p) 10
          stack size            (kbytes, -s) 8192
          cpu time             (seconds, -t) unlimited
          max user processes            (-u) 29995
          virtual memory        (kbytes, -v) unlimited
          
          Show
          josesa Jose Sa added a comment - Does this information helps? -bash-3.00$ ulimit -a core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited open files (-n) 256 pipe size (512 bytes, -p) 10 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 29995 virtual memory (kbytes, -v) unlimited
          Hide
          oldelvet Richard Mortimer added a comment -

          None of those limits seem to be a problem.

          It is likely to be a real virtual memory shortage in that case. You say you are using Solaris so I presume that you will be using tmpfs for your /tmp storage. In that case it might be that /tmp is getting filled up (does your job use /tmp for any storage? Jenkins may be storing in /tmp too). You likely need to monitor virtual memory/swap usage and /tmp free space and work out what is using the space.

          Until you eliminate local machine configuration issues there isn't a lot that can be attributed to Jenkins.

          Show
          oldelvet Richard Mortimer added a comment - None of those limits seem to be a problem. It is likely to be a real virtual memory shortage in that case. You say you are using Solaris so I presume that you will be using tmpfs for your /tmp storage. In that case it might be that /tmp is getting filled up (does your job use /tmp for any storage? Jenkins may be storing in /tmp too). You likely need to monitor virtual memory/swap usage and /tmp free space and work out what is using the space. Until you eliminate local machine configuration issues there isn't a lot that can be attributed to Jenkins.
          Hide
          kohsuke Kohsuke Kawaguchi added a comment -

          I have a hunch. timestamper uses an exntesion point called console annotation which stores a small compressed gzip stream of data into the log file. It uses a lot of those, because each timestamp it's adding, I believe, is a new annotation.

          If there's a race condition in the write operations, it could corrupt the gzip encoded stream, and I wonder if it can happen just in the right away that breaks the decompressor like this.

          If my hypothesis is correct, trying to look at the problematic build log will consistently reproduce the problem (as opposed to "sometimes I can see build #15 but some times it crashes when looking at the same build #15.)

          If this fits the issue you are seeing, please attach your console output (or send it personally to me if it's sensitive), and that'd greatly simplify our efforts to fix the issue.

          Show
          kohsuke Kohsuke Kawaguchi added a comment - I have a hunch. timestamper uses an exntesion point called console annotation which stores a small compressed gzip stream of data into the log file. It uses a lot of those, because each timestamp it's adding, I believe, is a new annotation. If there's a race condition in the write operations, it could corrupt the gzip encoded stream, and I wonder if it can happen just in the right away that breaks the decompressor like this. If my hypothesis is correct, trying to look at the problematic build log will consistently reproduce the problem (as opposed to "sometimes I can see build #15 but some times it crashes when looking at the same build #15.) If this fits the issue you are seeing, please attach your console output (or send it personally to me if it's sensitive), and that'd greatly simplify our efforts to fix the issue.
          Hide
          kohsuke Kohsuke Kawaguchi added a comment -

          I take it back. Neither 32756 nor 59208 is a big memory chunk.

          Show
          kohsuke Kohsuke Kawaguchi added a comment - I take it back. Neither 32756 nor 59208 is a big memory chunk.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          changelog.html
          core/src/main/java/hudson/console/AnnotatedLargeText.java
          core/src/main/java/hudson/console/ConsoleNote.java
          http://jenkins-ci.org/commit/jenkins/313fb5940f63ca7e13281f4bba1cdbcfcfd8f2c3
          Log:
          [FIXED JENKINS-9349] let GZipInputStream release its native memory
          eagerly.

          Compare: https://github.com/jenkinsci/jenkins/compare/e7cc141...313fb59

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/src/main/java/hudson/console/AnnotatedLargeText.java core/src/main/java/hudson/console/ConsoleNote.java http://jenkins-ci.org/commit/jenkins/313fb5940f63ca7e13281f4bba1cdbcfcfd8f2c3 Log: [FIXED JENKINS-9349] let GZipInputStream release its native memory eagerly. Compare: https://github.com/jenkinsci/jenkins/compare/e7cc141...313fb59
          scm_issue_link SCM/JIRA link daemon made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]
          Hide
          dogfood dogfood added a comment -
          Show
          dogfood dogfood added a comment - Integrated in jenkins_main_trunk #1267
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          changelog.html
          core/src/main/java/hudson/console/AnnotatedLargeText.java
          core/src/main/java/hudson/console/ConsoleNote.java
          http://jenkins-ci.org/commit/jenkins/7aab7ebf317d8ad1c81b72e813c339fbab05dfca
          Log:
          [FIXED JENKINS-9349] let GZipInputStream release its native memory
          eagerly.
          (cherry picked from commit 313fb5940f63ca7e13281f4bba1cdbcfcfd8f2c3)

          Conflicts:

          changelog.html

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/src/main/java/hudson/console/AnnotatedLargeText.java core/src/main/java/hudson/console/ConsoleNote.java http://jenkins-ci.org/commit/jenkins/7aab7ebf317d8ad1c81b72e813c339fbab05dfca Log: [FIXED JENKINS-9349] let GZipInputStream release its native memory eagerly. (cherry picked from commit 313fb5940f63ca7e13281f4bba1cdbcfcfd8f2c3) Conflicts: changelog.html
          stevengbrown Steven G Brown made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 139510 ] JNJira + In-Review [ 205020 ]

            People

            Assignee:
            stevengbrown Steven G Brown
            Reporter:
            ford30066 Matthew Ford
            Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: