The latest version of Jenkins core uses Jetty 9.4, this version seems changes the way to manage the session cache and now it is infinite by default, this makes that the session cache is not cleaned with the time and cause a memory leak.

      This script gets the EvictionPolicy value and returns -1

      import net.bull.javamelody.* 
      def sessionMapField = SessionListener.class.getDeclaredField('SESSION_MAP_BY_ID') 
      sessionMapField.setAccessible(true) 
      def sessions = sessionMapField.get(null) 
      for (def sessionKV : sessions) { 
      def session = sessionKV.value 
        println session.getSessionHandler().getSessionCache().getEvictionPolicy() 
      }
      

      Jetty was upgraded from 9.2 to 9.4 in Jenkins 2.61, The old Jetty uses a different caching mechanism, which seems to have a non-infinite default eviction policy

      The workaround is to place this script at JENKINS_HOME/init.groovy.d to set the eviction policy to 30 min.

      import net.bull.javamelody.* 
      def sessionMapField = SessionListener.class.getDeclaredField('SESSION_MAP_BY_ID') 
      sessionMapField.setAccessible(true) 
      def sessions = sessionMapField.get(null) 
      for (def sessionKV : sessions) { 
      def session = sessionKV.value 
         session.getSessionHandler().getSessionCache().setEvictionPolicy(1800) 
      }
      

      These are other utility scripts to check the leak.

      This script list the sessions

      import net.bull.javamelody.* 
      println SessionListener.getSessionCount() + " sessions:" 
      def sessioninfos = SessionListener.getAllSessionsInformations() 
      for (sessioninfo in sessioninfos) { 
        println sessioninfo 
      }
      

      This script returns the number of sessions in cache

      import net.bull.javamelody.*
      
      def sessionMapField = SessionListener.class.getDeclaredField('SESSION_MAP_BY_ID')
      sessionMapField.setAccessible(true)
      def sessions = sessionMapField.get(null)
      for (def sessionKV: sessions) {
        def session = sessionKV.value
        def sessionCache = session.getSessionHandler().getSessionCache()
        def sessionsField = sessionCache.class.getDeclaredField('_sessions')
        sessionsField.setAccessible(true)
        println sessionsField.get(sessionCache).size()
      }
      

      This script force to clear the session cache

      import net.bull.javamelody.*
        
      def sessionMapField = SessionListener.class.getDeclaredField('SESSION_MAP_BY_ID')
      sessionMapField.setAccessible(true)
      def sessions = sessionMapField.get(null)
      for (def sessionKV: sessions) {
        def session = sessionKV.value
        def sessionCache = session.getSessionHandler().getSessionCache()
        def sessionsField = sessionCache.class.getDeclaredField('_sessions')
        sessionsField.setAccessible(true)
        def sessionKeys = sessionsField.get(sessionCache).keys()
        for (def sessionKey: sessionKeys) {
          sessionCache.delete(sessionKey)
        }
      }
      

          [JENKINS-49596] User session memory leak

          Oleg Nenashev added a comment -

          https://github.com/jenkinsci/winstone/pull/44 pulls in newer relese of Jetty. Let's see what olamy responds about this ticket

          Oleg Nenashev added a comment - https://github.com/jenkinsci/winstone/pull/44 pulls in newer relese of Jetty. Let's see what olamy responds about this ticket

          Olivier Lamy added a comment -

          yup good catch sounds something to change in winstone and sessionCache configuration.
          I have updated https://github.com/jenkinsci/winstone/pull/44

          Olivier Lamy added a comment - yup good catch sounds something to change in winstone and sessionCache configuration. I have updated https://github.com/jenkinsci/winstone/pull/44

          Code changed in jenkins
          User: olivier lamy
          Path:
          src/java/winstone/HostConfiguration.java
          src/java/winstone/cmdline/Option.java
          http://jenkins-ci.org/commit/winstone/12474923c40149530094823ecbe80ff2d7f6f216
          Log:
          JENKINS-49596 session cache is never cleaned

          Signed-off-by: olivier lamy <olamy@apache.org>

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: olivier lamy Path: src/java/winstone/HostConfiguration.java src/java/winstone/cmdline/Option.java http://jenkins-ci.org/commit/winstone/12474923c40149530094823ecbe80ff2d7f6f216 Log: JENKINS-49596 session cache is never cleaned Signed-off-by: olivier lamy <olamy@apache.org>

          Oleg Nenashev added a comment -

          ifernandezcalvo olamy I would like to highlight that the integrated pull request is not backportable. So it won't get into 2.107.x unless we prepare a separate patch-release for that.

          My proposal would be to...
          1) Create a separate branch and backport the issue fix to the Winstone baseline with old Jetty
          2) Release that version and integrate it into the next weekly. Mark this issue for backporting
          3) After that release the current master baseline and integrate it into a later weekly release.

          Oleg Nenashev added a comment - ifernandezcalvo olamy I would like to highlight that the integrated pull request is not backportable. So it won't get into 2.107.x unless we prepare a separate patch-release for that. My proposal would be to... 1) Create a separate branch and backport the issue fix to the Winstone baseline with old Jetty 2) Release that version and integrate it into the next weekly. Mark this issue for backporting 3) After that release the current master baseline and integrate it into a later weekly release.

          Olivier Lamy added a comment -

          a bit conservative... but fair enough I can do that

          Olivier Lamy added a comment - a bit conservative... but fair enough I can do that

          Oleg Nenashev added a comment -

          Well, 2.107.1 is already on the crashlanding course due to JEP-200. I do not want to add more risks like fresh Jetty release there.
          Thanks for doing that! I am happy to share the release legwork if needed

          Oleg Nenashev added a comment - Well, 2.107.1 is already on the crashlanding course due to JEP-200. I do not want to add more risks like fresh Jetty release there. Thanks for doing that! I am happy to share the release legwork if needed

          Olivier Lamy added a comment -

          well I'm doing more check before releasing anything.
          Do you have the issue without using javamelody?

          Olivier Lamy added a comment - well I'm doing more check before releasing anything. Do you have the issue without using javamelody?

          Code changed in jenkins
          User: olivier lamy
          Path:
          src/java/winstone/HostConfiguration.java
          src/java/winstone/cmdline/Option.java
          http://jenkins-ci.org/commit/winstone/36ab8d1ec18a118d4a0f17a1e996eb294cc44589
          Log:
          JENKINS-49596 session cache is never cleaned

          Signed-off-by: olivier lamy <olamy@apache.org>

          Compare: https://github.com/jenkinsci/winstone/compare/4e556fcaa53e^...36ab8d1ec18a

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: olivier lamy Path: src/java/winstone/HostConfiguration.java src/java/winstone/cmdline/Option.java http://jenkins-ci.org/commit/winstone/36ab8d1ec18a118d4a0f17a1e996eb294cc44589 Log: JENKINS-49596 session cache is never cleaned Signed-off-by: olivier lamy <olamy@apache.org> Compare: https://github.com/jenkinsci/winstone/compare/4e556fcaa53e ^...36ab8d1ec18a

          Olivier Lamy added a comment - - edited

          for my understanding SessionListener is a javamelody class?
          why does it have references to the sessions?

          Olivier Lamy added a comment - - edited for my understanding SessionListener is a javamelody class? why does it have references to the sessions?

          Devin Nusbaum added a comment - - edited

          olamy Yes it is a JavaMelody class, here is the source code. I am not super familiar with JavaMelody, but it looks like it provides monitoring of various information, including http sessions, and allows http sessions to be invalidated manually. There is a JavaMelody Jenkins plugin with some details here.

          Thanks for preparing the winstone-4.1.x branch so the fix can be backported! Is there anything I can do to help you get that branch released?

          Devin Nusbaum added a comment - - edited olamy Yes it is a JavaMelody class, here is the source code. I am not super familiar with JavaMelody, but it looks like it provides monitoring of various information, including http sessions, and allows http sessions to be invalidated manually. There is a JavaMelody Jenkins plugin with some details here . Thanks for preparing the winstone-4.1.x branch so the fix can be backported! Is there anything I can do to help you get that branch released?

          Olivier Lamy added a comment -

          Hi dnusbaum I'd like to do some testing with and without JavaMelody because the issue is not really clear for me. Do you know which version of the monitoring plugin have you tested with to reproduce the problem?

          Olivier Lamy added a comment - Hi dnusbaum I'd like to do some testing with and without JavaMelody because the issue is not really clear for me. Do you know which version of the monitoring plugin have you tested with to reproduce the problem?

          Oleg Nenashev added a comment -

          For now I have released 4.1.1 so that we can do some testing on the core's side, but actually I messed up it a bit. It should have been a snapshot release: https://github.com/jenkinsci/winstone/pull/45

          Oleg Nenashev added a comment - For now I have released 4.1.1 so that we can do some testing on the core's side, but actually I messed up it a bit. It should have been a snapshot release: https://github.com/jenkinsci/winstone/pull/45

          Olivier Lamy added a comment -

          oops sorry I was a bit busy today. But I will do some testing tomorrow.
          There is still something I don't understand with this issue here and especially with the use of javamelody.
          for details on Jetty session in 9.4.x please read here https://www.eclipse.org/jetty/documentation/9.4.x/session-management.html

          Olivier Lamy added a comment - oops sorry I was a bit busy today. But I will do some testing tomorrow. There is still something I don't understand with this issue here and especially with the use of javamelody. for details on Jetty session in 9.4.x please read here https://www.eclipse.org/jetty/documentation/9.4.x/session-management.html

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          CHANGELOG.md
          README.md
          http://jenkins-ci.org/commit/winstone/15941f8e757eb8e0614b30ac5326724a7820dca0
          Log:
          JENKINS-49596 - noting and documenting 4.1.1

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: CHANGELOG.md README.md http://jenkins-ci.org/commit/winstone/15941f8e757eb8e0614b30ac5326724a7820dca0 Log: JENKINS-49596 - noting and documenting 4.1.1

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          README.md
          src/java/winstone/cmdline/Option.java
          http://jenkins-ci.org/commit/winstone/a18027d72ab9e38652e25920caead985dac24e39
          Log:
          JENKINS-49596 - Increase the default session eviction timeout to 30min

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: README.md src/java/winstone/cmdline/Option.java http://jenkins-ci.org/commit/winstone/a18027d72ab9e38652e25920caead985dac24e39 Log: JENKINS-49596 - Increase the default session eviction timeout to 30min

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          CHANGELOG.md
          README.md
          src/java/winstone/cmdline/Option.java
          http://jenkins-ci.org/commit/winstone/7cb373b79083d6fdb721c6a0d69ef83ed176ab90
          Log:
          Merge pull request #45 from oleg-nenashev/4.1.1-changelog

          JENKINS-49596 - Increase timeout to 30 min, Noting 4.1.1 and adding it to documentation

          Compare: https://github.com/jenkinsci/winstone/compare/0e58e07876af...7cb373b79083

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: CHANGELOG.md README.md src/java/winstone/cmdline/Option.java http://jenkins-ci.org/commit/winstone/7cb373b79083d6fdb721c6a0d69ef83ed176ab90 Log: Merge pull request #45 from oleg-nenashev/4.1.1-changelog JENKINS-49596 - Increase timeout to 30 min, Noting 4.1.1 and adding it to documentation Compare: https://github.com/jenkinsci/winstone/compare/0e58e07876af...7cb373b79083

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          src/java/winstone/cmdline/Option.java
          http://jenkins-ci.org/commit/winstone/9c448bee191aa078dbb6cc3e3cf0b04087fb4e6f
          Log:
          JENKINS-49596 - Backport the default session eviction timeout value fix to 4.1.2

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/java/winstone/cmdline/Option.java http://jenkins-ci.org/commit/winstone/9c448bee191aa078dbb6cc3e3cf0b04087fb4e6f Log: JENKINS-49596 - Backport the default session eviction timeout value fix to 4.1.2

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          CHANGELOG.md
          http://jenkins-ci.org/commit/winstone/9137572c6fe1bdb3ef510be5ee0ee9fbc9f56d43
          Log:
          JENKINS-49596 - Noting 4.1.2

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: CHANGELOG.md http://jenkins-ci.org/commit/winstone/9137572c6fe1bdb3ef510be5ee0ee9fbc9f56d43 Log: JENKINS-49596 - Noting 4.1.2

          Oleg Nenashev added a comment -

          The fix has been integrated towards 2.110.
          It may be considered as LTS candidate since we decoupled the meaty Jetty update part

          Oleg Nenashev added a comment - The fix has been integrated towards 2.110. It may be considered as LTS candidate since we decoupled the meaty Jetty update part

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          war/pom.xml
          http://jenkins-ci.org/commit/jenkins/805a7ab57d4d25f8c5565c5af0b16b36a780a8c9
          Log:
          JENKINS-49596 - Update Winstone to 4.1.2 (#3307)

          • JENKINS-49596 - Pick 4.1.2 with extended session eviction defaults

          (cherry picked from commit 852ecdc5f2a7802fa74f425bd17ba028f787b2c1)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: war/pom.xml http://jenkins-ci.org/commit/jenkins/805a7ab57d4d25f8c5565c5af0b16b36a780a8c9 Log: JENKINS-49596 - Update Winstone to 4.1.2 (#3307) JENKINS-49596 - Update Winstone to 4.1.1 JENKINS-49596 - Pick 4.1.2 with extended session eviction defaults (cherry picked from commit 852ecdc5f2a7802fa74f425bd17ba028f787b2c1)

          Prior to this fix, I'd been setting --sessionTimeout=NNNNN in my java options to extend the session.  Is there an equivalent way to set the eviction policy to longer than 30 minutes, or is my only option to add a groovy script to JENKINS_HOME/init.groovy.d?

          marc macintyre added a comment - Prior to this fix, I'd been setting --sessionTimeout=NNNNN in my java options to extend the session.  Is there an equivalent way to set the eviction policy to longer than 30 minutes, or is my only option to add a groovy script to JENKINS_HOME/init.groovy.d?

          Devin Nusbaum added a comment - - edited

          marcmac Based on the README I think you want to pass --sessionEviction=foo in addition to --sessionTimeout=foo.

          Devin Nusbaum added a comment - - edited marcmac Based on the  README I think you want to pass --sessionEviction=foo in addition to --sessionTimeout=foo .

            olamy Olivier Lamy
            ifernandezcalvo Ivan Fernandez Calvo
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: