• In larger installations of Jenkins the complex cycles of object references containing classloaders containing references to the channel which contains references back to the classloaders can confuse the garbage collector.
      • The confusion arrises because classloaders are considered live while there are objects from the classloader live and the RemoteClassLoader and RemoteInvocationHandler's proxy both need a reference to the Channel which typically can be storing either some of the objects from the RemoteClassLoader or the RemoteInvocationHandler's proxy instances in the Channel properties. Thus although the entire subgraph is no longer live, the cycle is not detected as ClassLoaders are treated separately. The issue is worse in Java 6/7 where the ClassLoaders are on the PermGen heap though it is also present in Java 8.
      • Typically multiple out of memory errors can cause the cycle to ultimately get cleaned up as one half gets cleared out breaking the cycle enough so that a subsequent out of memory removes the second half leaving a third out of memory to clear out the ClassLoaders... but after three out of memory errors in a row, Jenkins itself is typically well dead.
      • I suspect this issue is related to the eventual OOM that JNLP slaves die from after many reconnects

      CloudBees Ref: ZD-26018

          [JENKINS-28844] The curious case of the Channel memory cycles

          Stephen Connolly created issue -
          Stephen Connolly made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]
          Jesse Glick made changes -
          Labels New: memory-leak
          Stephen Connolly made changes -
          Labels Original: memory-leak New: lts-candidate memory-leak
          SCM/JIRA link daemon made changes -
          Resolution New: Fixed [ 1 ]
          Status Original: In Progress [ 3 ] New: Resolved [ 5 ]
          Oliver Gondža made changes -
          Labels Original: lts-candidate memory-leak New: 1.609.3-fixed memory-leak
          Jesse Glick made changes -
          Link New: This issue is duplicated by JENKINS-28429 [ JENKINS-28429 ]
          Oleg Nenashev made changes -
          Link New: This issue is related to JENKINS-26855 [ JENKINS-26855 ]
          Oleg Nenashev made changes -
          Link New: This issue is duplicated by JENKINS-26855 [ JENKINS-26855 ]
          R. Tyler Croy made changes -
          Workflow Original: JNJira [ 163704 ] New: JNJira + In-Review [ 197284 ]
          Stephen Connolly made changes -
          Status Original: Resolved [ 5 ] New: Closed [ 6 ]

            stephenconnolly Stephen Connolly
            stephenconnolly Stephen Connolly
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: