• In larger installations of Jenkins the complex cycles of object references containing classloaders containing references to the channel which contains references back to the classloaders can confuse the garbage collector.
      • The confusion arrises because classloaders are considered live while there are objects from the classloader live and the RemoteClassLoader and RemoteInvocationHandler's proxy both need a reference to the Channel which typically can be storing either some of the objects from the RemoteClassLoader or the RemoteInvocationHandler's proxy instances in the Channel properties. Thus although the entire subgraph is no longer live, the cycle is not detected as ClassLoaders are treated separately. The issue is worse in Java 6/7 where the ClassLoaders are on the PermGen heap though it is also present in Java 8.
      • Typically multiple out of memory errors can cause the cycle to ultimately get cleaned up as one half gets cleared out breaking the cycle enough so that a subsequent out of memory removes the second half leaving a third out of memory to clear out the ClassLoaders... but after three out of memory errors in a row, Jenkins itself is typically well dead.
      • I suspect this issue is related to the eventual OOM that JNLP slaves die from after many reconnects

      CloudBees Ref: ZD-26018

          [JENKINS-28844] The curious case of the Channel memory cycles

          Stephen Connolly created issue -
          Stephen Connolly made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]

          Stephen Connolly added a comment - See https://github.com/jenkinsci/remoting/pull/54
          Jesse Glick made changes -
          Labels New: memory-leak

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          pom.xml
          src/main/java/hudson/remoting/Channel.java
          src/main/java/hudson/remoting/JarCacheSupport.java
          src/main/java/hudson/remoting/Launcher.java
          src/main/java/hudson/remoting/NamingThreadFactory.java
          src/main/java/hudson/remoting/RemoteClassLoader.java
          src/main/java/hudson/remoting/RemoteInvocationHandler.java
          src/test/java/hudson/remoting/ChannelTest.java
          src/test/java/hudson/remoting/util/GCTask.java
          http://jenkins-ci.org/commit/remoting/bf81838a2c6ab8fcc96f54fedff003c168433688
          Log:
          Merge pull request #54 from jenkinsci/fix-memory-leak

          JENKINS-28844 Fix memory leak

          Compare: https://github.com/jenkinsci/remoting/compare/05329f599148...bf81838a2c6a

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: pom.xml src/main/java/hudson/remoting/Channel.java src/main/java/hudson/remoting/JarCacheSupport.java src/main/java/hudson/remoting/Launcher.java src/main/java/hudson/remoting/NamingThreadFactory.java src/main/java/hudson/remoting/RemoteClassLoader.java src/main/java/hudson/remoting/RemoteInvocationHandler.java src/test/java/hudson/remoting/ChannelTest.java src/test/java/hudson/remoting/util/GCTask.java http://jenkins-ci.org/commit/remoting/bf81838a2c6ab8fcc96f54fedff003c168433688 Log: Merge pull request #54 from jenkinsci/fix-memory-leak JENKINS-28844 Fix memory leak Compare: https://github.com/jenkinsci/remoting/compare/05329f599148...bf81838a2c6a

          Daniel Beck added a comment -

          stephenconnolly Should this be an lts-candidate?

          Daniel Beck added a comment - stephenconnolly Should this be an lts-candidate ?
          Stephen Connolly made changes -
          Labels Original: memory-leak New: lts-candidate memory-leak

          danielbeck yes, I have added the label

          Stephen Connolly added a comment - danielbeck yes, I have added the label

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          pom.xml
          http://jenkins-ci.org/commit/jenkins/aac20468ff5511ea6c58de07f3bcf3ae108b9cb0
          Log:
          [FIXED JENKINS-28844] The curious case of the Channel memory cycles

          Pick up the remoting 2.52 release

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: pom.xml http://jenkins-ci.org/commit/jenkins/aac20468ff5511ea6c58de07f3bcf3ae108b9cb0 Log: [FIXED JENKINS-28844] The curious case of the Channel memory cycles Pick up the remoting 2.52 release
          SCM/JIRA link daemon made changes -
          Resolution New: Fixed [ 1 ]
          Status Original: In Progress [ 3 ] New: Resolved [ 5 ]

            stephenconnolly Stephen Connolly
            stephenconnolly Stephen Connolly
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: