- In larger installations of Jenkins the complex cycles of object references containing classloaders containing references to the channel which contains references back to the classloaders can confuse the garbage collector.
- The confusion arrises because classloaders are considered live while there are objects from the classloader live and the RemoteClassLoader and RemoteInvocationHandler's proxy both need a reference to the Channel which typically can be storing either some of the objects from the RemoteClassLoader or the RemoteInvocationHandler's proxy instances in the Channel properties. Thus although the entire subgraph is no longer live, the cycle is not detected as ClassLoaders are treated separately. The issue is worse in Java 6/7 where the ClassLoaders are on the PermGen heap though it is also present in Java 8.
- Typically multiple out of memory errors can cause the cycle to ultimately get cleaned up as one half gets cleared out breaking the cycle enough so that a subsequent out of memory removes the second half leaving a third out of memory to clear out the ClassLoaders... but after three out of memory errors in a row, Jenkins itself is typically well dead.
- I suspect this issue is related to the eventual OOM that JNLP slaves die from after many reconnects
CloudBees Ref: ZD-26018