Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-5119

Winstone: Memory leak due to default session timeout of -1

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • core
    • None
    • Hudson 1.337 (Winstone), Solaris 10 SPARC, Sun JDK 1.6.0_16

      We are running Hudson with its embedded Winstone servlet engine. User authentication within Hudson is done via Hudsons LDAP security realm using our corporate ActiveDirectory via LDAP.

      After running Hudson a few weeks we're running into OutOfMemory exceptions. Here the culprit is a single winstone.WebAppConfiguration instance holding a lot of winstone.WinStoneSession instances each holding an Acegi RememberMeAuthenticationToken instance with 36k data from the authenticated users AD record.

      It seems that Hudson isn't setting Winstones session timeout. Without specifying a session timeout winstone.WebAppConfiguration.makeNewSession() is using a session timeout of -1.

          [JENKINS-5119] Winstone: Memory leak due to default session timeout of -1

          tjuerge created issue -

          tjuerge added a comment -

          This can be fixed by specifying a session timeout in hudson/main/war/resources/WEB-INF/web.xml:

          Index: resources/WEB-INF/web.xml
          ===================================================================
          --- resources/WEB-INF/web.xml	(revision 24689)
          +++ resources/WEB-INF/web.xml	(working copy)
          @@ -30,6 +30,10 @@
             <display-name>Hudson</display-name>
             <description>Build management system</description>
           
          +  <session-config>
          +    <session-timeout>60</session-timeout>
          +  </session-config>
          +
             <servlet>
               <servlet-name>Stapler</servlet-name>
               <servlet-class>org.kohsuke.stapler.Stapler</servlet-class>
          

          tjuerge added a comment - This can be fixed by specifying a session timeout in hudson/main/war/resources/WEB-INF/web.xml : Index: resources/WEB-INF/web.xml =================================================================== --- resources/WEB-INF/web.xml (revision 24689) +++ resources/WEB-INF/web.xml (working copy) @@ -30,6 +30,10 @@ <display-name>Hudson</display-name> <description>Build management system</description> + <session-config> + <session-timeout>60</session-timeout> + </session-config> + <servlet> <servlet-name>Stapler</servlet-name> <servlet-class>org.kohsuke.stapler.Stapler</servlet-class>

          tjuerge added a comment -

          Answered my question: winstone.HostConfiguration.run() is responsible for calling WebAppCOnfiguration.invalidateExpiredSessions()

          tjuerge added a comment - Answered my question: winstone.HostConfiguration.run() is responsible for calling WebAppCOnfiguration.invalidateExpiredSessions()
          tjuerge made changes -
          Description Original: We are running Hudson with its embedded Winstone servlet engine. User authentication within Hudson is done via Hudsons LDAP security realm using our corporate ActiveDirectory via LDAP.

          After running Hudson a few weeks we're running into OutOfMemory exceptions. Here the culprit is a single {{winstone.WebAppConfiguration}} instance holding a lot of {{winstone.WinStoneSession}} instances each holding an Acegi {{RememberMeAuthenticationToken}} instance with 36k data from the authenticated users AD record.

          It seems that Hudson isn't setting Winstones session timeout. Without specifying a session timeout {{winstone.WebAppConfiguration.makeNewSession()}} is using a session timeout of -1 via {{ws.setMaxInactiveInterval(-1)}}.

          Btw. who is responsible for calling {{winstone.WebAppConfiguration.invalidateExpiredSessions()}}?
          New: We are running Hudson with its embedded Winstone servlet engine. User authentication within Hudson is done via Hudsons LDAP security realm using our corporate ActiveDirectory via LDAP.

          After running Hudson a few weeks we're running into OutOfMemory exceptions. Here the culprit is a single {{winstone.WebAppConfiguration}} instance holding a lot of {{winstone.WinStoneSession}} instances each holding an Acegi {{RememberMeAuthenticationToken}} instance with 36k data from the authenticated users AD record.

          It seems that Hudson isn't setting Winstones session timeout. Without specifying a session timeout {{winstone.WebAppConfiguration.makeNewSession()}} is using a session timeout of -1.

          slurdge added a comment -

          We have also the same issue.
          I tried tjuerge patch to web.xml but it did not change anything (by the way I had to remove a sha1 - computed file to do this).
          We have a ActiveDirectory that is configured via LDAP. Every connection leaks a little bit. The problem is worsened by the fact that we use a tray applet that checks the build status. This tray applet authenticates itself with Hudson. I believe that is creating a WinStoneSession each time, leaking memory slowly.
          I've also looked at the memory dump and my findings are the same as tjuerge.
          In one week, Hudson takes the maximum of 512Mb allocated and starts to fail builds.

          Any solution is welcomed, I would be happy to try a build with a fix for this.
          Also, if there is a workaround in forcing the flush of expired sessions, this would allow us to periodically do it and not interrupt operation by restarting Hudson (the only available option so far, but not very good for us).

          slurdge added a comment - We have also the same issue. I tried tjuerge patch to web.xml but it did not change anything (by the way I had to remove a sha1 - computed file to do this). We have a ActiveDirectory that is configured via LDAP. Every connection leaks a little bit. The problem is worsened by the fact that we use a tray applet that checks the build status. This tray applet authenticates itself with Hudson. I believe that is creating a WinStoneSession each time, leaking memory slowly. I've also looked at the memory dump and my findings are the same as tjuerge. In one week, Hudson takes the maximum of 512Mb allocated and starts to fail builds. Any solution is welcomed, I would be happy to try a build with a fix for this. Also, if there is a workaround in forcing the flush of expired sessions, this would allow us to periodically do it and not interrupt operation by restarting Hudson (the only available option so far, but not very good for us).

          We're also seeing a large number of objects 6,143 sessions with 17.9Mb. Can this timeout be added to the standard distribution?

          We're going OOM as often as 3 times a day. When monit restarts Hudson we blow away the expanded WAR directory which messing with the web.xml a bit much pain.

          We are also using the Hudson tray applet. Our primary OOM problem seems to be related to the Cobertura reports, by the way, but this is not a small leak either.

          Hudson ver. 1.353 on Linux, Sun JDK 1.6.0_18

          StevenTheAmusing added a comment - We're also seeing a large number of objects 6,143 sessions with 17.9Mb. Can this timeout be added to the standard distribution? We're going OOM as often as 3 times a day. When monit restarts Hudson we blow away the expanded WAR directory which messing with the web.xml a bit much pain. We are also using the Hudson tray applet. Our primary OOM problem seems to be related to the Cobertura reports, by the way, but this is not a small leak either. Hudson ver. 1.353 on Linux, Sun JDK 1.6.0_18

          VisualVM showing a large number of sessions.

          StevenTheAmusing added a comment - VisualVM showing a large number of sessions.
          StevenTheAmusing made changes -
          Attachment New: VisualVM 1.2.2 4192010 12616 PM.jpg [ 19349 ]

          Fixed in Winstone 0.9.10-hudson-17 to have the finite default session timeout value. This change still needs to be integrated into Hudson.

          Kohsuke Kawaguchi added a comment - Fixed in Winstone 0.9.10-hudson-17 to have the finite default session timeout value. This change still needs to be integrated into Hudson.

          amrangan added a comment -

          Will this change be integrated into the next release of Hudson (i.e. 1.356) ?

          amrangan added a comment - Will this change be integrated into the next release of Hudson (i.e. 1.356) ?

            Unassigned Unassigned
            tjuerge tjuerge
            Votes:
            5 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: