• In larger installations of Jenkins the complex cycles of object references containing classloaders containing references to the channel which contains references back to the classloaders can confuse the garbage collector.
      • The confusion arrises because classloaders are considered live while there are objects from the classloader live and the RemoteClassLoader and RemoteInvocationHandler's proxy both need a reference to the Channel which typically can be storing either some of the objects from the RemoteClassLoader or the RemoteInvocationHandler's proxy instances in the Channel properties. Thus although the entire subgraph is no longer live, the cycle is not detected as ClassLoaders are treated separately. The issue is worse in Java 6/7 where the ClassLoaders are on the PermGen heap though it is also present in Java 8.
      • Typically multiple out of memory errors can cause the cycle to ultimately get cleaned up as one half gets cleared out breaking the cycle enough so that a subsequent out of memory removes the second half leaving a third out of memory to clear out the ClassLoaders... but after three out of memory errors in a row, Jenkins itself is typically well dead.
      • I suspect this issue is related to the eventual OOM that JNLP slaves die from after many reconnects

      CloudBees Ref: ZD-26018

          [JENKINS-28844] The curious case of the Channel memory cycles

          Stephen Connolly added a comment - See https://github.com/jenkinsci/remoting/pull/54

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          pom.xml
          src/main/java/hudson/remoting/Channel.java
          src/main/java/hudson/remoting/JarCacheSupport.java
          src/main/java/hudson/remoting/Launcher.java
          src/main/java/hudson/remoting/NamingThreadFactory.java
          src/main/java/hudson/remoting/RemoteClassLoader.java
          src/main/java/hudson/remoting/RemoteInvocationHandler.java
          src/test/java/hudson/remoting/ChannelTest.java
          src/test/java/hudson/remoting/util/GCTask.java
          http://jenkins-ci.org/commit/remoting/bf81838a2c6ab8fcc96f54fedff003c168433688
          Log:
          Merge pull request #54 from jenkinsci/fix-memory-leak

          JENKINS-28844 Fix memory leak

          Compare: https://github.com/jenkinsci/remoting/compare/05329f599148...bf81838a2c6a

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: pom.xml src/main/java/hudson/remoting/Channel.java src/main/java/hudson/remoting/JarCacheSupport.java src/main/java/hudson/remoting/Launcher.java src/main/java/hudson/remoting/NamingThreadFactory.java src/main/java/hudson/remoting/RemoteClassLoader.java src/main/java/hudson/remoting/RemoteInvocationHandler.java src/test/java/hudson/remoting/ChannelTest.java src/test/java/hudson/remoting/util/GCTask.java http://jenkins-ci.org/commit/remoting/bf81838a2c6ab8fcc96f54fedff003c168433688 Log: Merge pull request #54 from jenkinsci/fix-memory-leak JENKINS-28844 Fix memory leak Compare: https://github.com/jenkinsci/remoting/compare/05329f599148...bf81838a2c6a

          Daniel Beck added a comment -

          stephenconnolly Should this be an lts-candidate?

          Daniel Beck added a comment - stephenconnolly Should this be an lts-candidate ?

          danielbeck yes, I have added the label

          Stephen Connolly added a comment - danielbeck yes, I have added the label

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          pom.xml
          http://jenkins-ci.org/commit/jenkins/aac20468ff5511ea6c58de07f3bcf3ae108b9cb0
          Log:
          [FIXED JENKINS-28844] The curious case of the Channel memory cycles

          Pick up the remoting 2.52 release

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: pom.xml http://jenkins-ci.org/commit/jenkins/aac20468ff5511ea6c58de07f3bcf3ae108b9cb0 Log: [FIXED JENKINS-28844] The curious case of the Channel memory cycles Pick up the remoting 2.52 release

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          pom.xml
          http://jenkins-ci.org/commit/jenkins/2906718ff53dd781a1248fff5dd56ac729438558
          Log:
          [FIXED JENKINS-28844] The curious case of the Channel memory cycles

          Pick up the remoting 2.52 release
          (cherry picked from commit aac20468ff5511ea6c58de07f3bcf3ae108b9cb0)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: pom.xml http://jenkins-ci.org/commit/jenkins/2906718ff53dd781a1248fff5dd56ac729438558 Log: [FIXED JENKINS-28844] The curious case of the Channel memory cycles Pick up the remoting 2.52 release (cherry picked from commit aac20468ff5511ea6c58de07f3bcf3ae108b9cb0)

          dogfood added a comment -

          Integrated in jenkins_main_trunk #4292
          [FIXED JENKINS-28844] The curious case of the Channel memory cycles (Revision 2906718ff53dd781a1248fff5dd56ac729438558)

          Result = UNSTABLE
          ogondza : 2906718ff53dd781a1248fff5dd56ac729438558
          Files :

          • pom.xml

          dogfood added a comment - Integrated in jenkins_main_trunk #4292 [FIXED JENKINS-28844] The curious case of the Channel memory cycles (Revision 2906718ff53dd781a1248fff5dd56ac729438558) Result = UNSTABLE ogondza : 2906718ff53dd781a1248fff5dd56ac729438558 Files : pom.xml

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          pom.xml
          http://jenkins-ci.org/commit/jenkins/9cd1929cfc80145657ad51c2d7d98d16b9f62195
          Log:
          Merge pull request #1766 from jenkinsci/jenkins-28844

          [FIXED JENKINS-28844] The curious case of the Channel memory cycles

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: pom.xml http://jenkins-ci.org/commit/jenkins/9cd1929cfc80145657ad51c2d7d98d16b9f62195 Log: Merge pull request #1766 from jenkinsci/jenkins-28844 [FIXED JENKINS-28844] The curious case of the Channel memory cycles

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          changelog.html
          http://jenkins-ci.org/commit/jenkins/1edb51cb919a3270b26e8782a0426381385df9b9
          Log:
          JENKINS-28844 Noting merge of #1766

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: changelog.html http://jenkins-ci.org/commit/jenkins/1edb51cb919a3270b26e8782a0426381385df9b9 Log: JENKINS-28844 Noting merge of #1766

            stephenconnolly Stephen Connolly
            stephenconnolly Stephen Connolly
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: