• 2.312

      When running on Java 7, Jenkins should ensure that all its ClassLoader instances are marked parallel-capable. This would improve class loading speed in some circumstances—mainly during startup or when first using new features, but also when searching for Groovy imports and the like.

      At least UberClassLoader, DependencyClassLoader, AntClassLoader2, and PluginFirstClassLoader should be so marked. (PluginFirstClassLoader ought to be made to extend AntClassLoader2 since we control it; all superclasses must be marked.) To work on Java 7, can use this idiom:

      try {
          Method m = ClassLoader.class.getDeclaredMethod("registerAsParallelCapable");
          m.setAccessible(true);
          boolean b = (Boolean) m.invoke(null);
          if (!b) {
              ...warn...
          }
      } catch (NoSuchMethodException x) {
          // fine, Java 6
      } catch (Exception x) {
          ...warn...
      }
      

      (Unfortunately this pattern must be repeated in such class, since registerAsParallelCapable is caller-sensitive.)

          [JENKINS-23784] Allow parallel class loading

          Jesse Glick added a comment -

          I can make CpsVmExecutorService set a more appropriate context loader, which perhaps helps unblock some of these, but others are apparently initiated on the application class loader and so still block. (This is not Jenkins code, so making Jenkins plugin class loaders parallel-capable would not help.)

          Regardless, after a while the system becomes slower and slower and memory usage grows to extreme levels (for example, over 3m instances of java.lang.reflect.Method such as X058.super$1$notify, consuming 268Mb excluding retained objects, after running a few hundred builds). Even after stopping all builds and letting the system go idle, manual GC does not help much. Unfortunately I have not had any success identifying root references.

          Jesse Glick added a comment - I can make CpsVmExecutorService set a more appropriate context loader, which perhaps helps unblock some of these, but others are apparently initiated on the application class loader and so still block. (This is not Jenkins code, so making Jenkins plugin class loaders parallel-capable would not help.) Regardless, after a while the system becomes slower and slower and memory usage grows to extreme levels (for example, over 3m instances of java.lang.reflect.Method such as X058.super$1$notify , consuming 268Mb excluding retained objects, after running a few hundred builds). Even after stopping all builds and letting the system go idle, manual GC does not help much. Unfortunately I have not had any success identifying root references.

          Jesse Glick added a comment -

          I suspect ClassInfo.globalClassSet is failing to relinquish metadata after builds. If true, it is possible Jenkins 2 (which uses Groovy 2) behaves better—need to check.

          Jesse Glick added a comment - I suspect ClassInfo.globalClassSet is failing to relinquish metadata after builds. If true, it is possible Jenkins 2 (which uses Groovy 2) behaves better—need to check.

          Jesse Glick added a comment -

          Filed workflow-cps PR 41 to deal with some memory leak issues I could reproduce in tests—but not this one, which analysis tools still fail to diagnose: the only references seem to be from Introspector.declaredMethodCache, which uses weak references and so should be quickly cleared during normal GC cycles, to say nothing of repeated forced GC.

          These are side issues—not part of the issue reported here, which is lock contention—but the slowdown (whether due to the memory leak or something else) makes it harder to verify that a workaround of this issue is actually effective.

          Jesse Glick added a comment - Filed workflow-cps PR 41 to deal with some memory leak issues I could reproduce in tests—but not this one, which analysis tools still fail to diagnose: the only references seem to be from Introspector.declaredMethodCache , which uses weak references and so should be quickly cleared during normal GC cycles, to say nothing of repeated forced GC. These are side issues—not part of the issue reported here, which is lock contention—but the slowdown (whether due to the memory leak or something else) makes it harder to verify that a workaround of this issue is actually effective.

          Jesse Glick added a comment -

          Indeed, after stopping new builds, while Introspector.flushCaches() had little effect, I managed to get heap to drop from over 3Gb to around 110Mb, by running in /script

          int x = 1024;
          xs = []
          while (true) {
            xs += new Object[x]
            x *= 1.3
          }
          

          until I got an OutOfMemoryError, and then forcing another GC. So this is a SoftReference collection issue. IIRC Groovy 2 eschews the soft references, so Jenkins 2 may scale better after PR 41 is merged. (Not testing that now since I have an independent patch to the same plugin associated with this actual issue.)

          Jesse Glick added a comment - Indeed, after stopping new builds, while Introspector.flushCaches() had little effect, I managed to get heap to drop from over 3Gb to around 110Mb, by running in /script int x = 1024; xs = [] while ( true ) { xs += new Object [x] x *= 1.3 } until I got an OutOfMemoryError , and then forcing another GC. So this is a SoftReference collection issue. IIRC Groovy 2 eschews the soft references, so Jenkins 2 may scale better after PR 41 is merged. (Not testing that now since I have an independent patch to the same plugin associated with this actual issue.)

          Jesse Glick added a comment -

          A related issue I just found while looking at a thread dump from an unresponsive instance: some threads such as

          at hudson.util.MaskingClassLoader.getResource(MaskingClassLoader.java:80)
          - waiting to lock <…> (a hudson.util.MaskingClassLoader)
          at java.lang.ClassLoader.getResource(ClassLoader.java:1091)
          at jenkins.util.AntClassLoader.getResource(AntClassLoader.java:888)
          at org.kohsuke.stapler.ClassDescriptor$ASM.loadParametersFromAsm(ClassDescriptor.java:258)
          at org.kohsuke.stapler.ClassDescriptor$ASM.access$100(ClassDescriptor.java:212)
          at org.kohsuke.stapler.ClassDescriptor.loadParameterNames(ClassDescriptor.java:145)
          at org.kohsuke.stapler.ClassDescriptor.loadConstructorParamNames(ClassDescriptor.java:179)
          at org.jenkinsci.plugins.workflow.cps.DSL.loadSoleArgumentKey(DSL.java:220)
          at org.jenkinsci.plugins.workflow.cps.DSL.parseArgs(DSL.java:356)
          at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:152)
          at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:124)
          at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:117)
          at …
          

          or

          at hudson.util.MaskingClassLoader.loadClass(MaskingClassLoader.java:70)
          - waiting to lock <…> (a hudson.util.MaskingClassLoader)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
          - locked <…> (a hudson.ClassicPluginStrategy$DependencyClassLoader)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
          at jenkins.util.AntClassLoader.findBaseClass(AntClassLoader.java:1398)
          at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1075)
          - locked <…> (a hudson.ClassicPluginStrategy$AntClassLoader2)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
          at hudson.util.XStream2$AssociatedConverterImpl.findConverter(XStream2.java:318)
          at hudson.util.XStream2$AssociatedConverterImpl.canConvert(XStream2.java:357)
          at com.thoughtworks.xstream.core.DefaultConverterLookup.lookupConverterForType(DefaultConverterLookup.java:59)
          - locked <…> (a com.thoughtworks.xstream.core.DefaultConverterLookup)
          at com.thoughtworks.xstream.XStream$1.lookupConverterForType(XStream.java:498)
          at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:56)
          at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
          at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
          at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
          at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
          at hudson.util.XStream2.unmarshal(XStream2.java:114)
          at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
          at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1044)
          at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1035)
          at …
          

          were blocked on one thread waiting for a lower-level lock:

          at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:390)
          - waiting to lock <…> (a org.eclipse.jetty.webapp.WebAppClassLoader)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
          - locked <…> (a hudson.util.MaskingClassLoader)
          at hudson.util.MaskingClassLoader.loadClass(MaskingClassLoader.java:75)
          - locked <…> (a hudson.util.MaskingClassLoader)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
          - locked <…> (a hudson.ClassicPluginStrategy$DependencyClassLoader)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
          at jenkins.util.AntClassLoader.findBaseClass(AntClassLoader.java:1398)
          at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1075)
          - locked <…> (a hudson.ClassicPluginStrategy$AntClassLoader2)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
          at …
          

          It is clearly wrong for MaskingClassLoader to be adding its own synchronization here, especially when the only apparent reason for the synchronized statements in this case was to guard access to some rarely-written lists. I will propose a patch.

          Jesse Glick added a comment - A related issue I just found while looking at a thread dump from an unresponsive instance: some threads such as at hudson.util.MaskingClassLoader.getResource(MaskingClassLoader.java:80) - waiting to lock <…> (a hudson.util.MaskingClassLoader) at java.lang.ClassLoader.getResource(ClassLoader.java:1091) at jenkins.util.AntClassLoader.getResource(AntClassLoader.java:888) at org.kohsuke.stapler.ClassDescriptor$ASM.loadParametersFromAsm(ClassDescriptor.java:258) at org.kohsuke.stapler.ClassDescriptor$ASM.access$100(ClassDescriptor.java:212) at org.kohsuke.stapler.ClassDescriptor.loadParameterNames(ClassDescriptor.java:145) at org.kohsuke.stapler.ClassDescriptor.loadConstructorParamNames(ClassDescriptor.java:179) at org.jenkinsci.plugins.workflow.cps.DSL.loadSoleArgumentKey(DSL.java:220) at org.jenkinsci.plugins.workflow.cps.DSL.parseArgs(DSL.java:356) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:152) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:124) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:117) at … or at hudson.util.MaskingClassLoader.loadClass(MaskingClassLoader.java:70) - waiting to lock <…> (a hudson.util.MaskingClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) - locked <…> (a hudson.ClassicPluginStrategy$DependencyClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at jenkins.util.AntClassLoader.findBaseClass(AntClassLoader.java:1398) at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1075) - locked <…> (a hudson.ClassicPluginStrategy$AntClassLoader2) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at hudson.util.XStream2$AssociatedConverterImpl.findConverter(XStream2.java:318) at hudson.util.XStream2$AssociatedConverterImpl.canConvert(XStream2.java:357) at com.thoughtworks.xstream.core.DefaultConverterLookup.lookupConverterForType(DefaultConverterLookup.java:59) - locked <…> (a com.thoughtworks.xstream.core.DefaultConverterLookup) at com.thoughtworks.xstream.XStream$1.lookupConverterForType(XStream.java:498) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:56) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50) at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189) at hudson.util.XStream2.unmarshal(XStream2.java:114) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173) at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1044) at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1035) at … were blocked on one thread waiting for a lower-level lock: at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:390) - waiting to lock <…> (a org.eclipse.jetty.webapp.WebAppClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) - locked <…> (a hudson.util.MaskingClassLoader) at hudson.util.MaskingClassLoader.loadClass(MaskingClassLoader.java:75) - locked <…> (a hudson.util.MaskingClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) - locked <…> (a hudson.ClassicPluginStrategy$DependencyClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at jenkins.util.AntClassLoader.findBaseClass(AntClassLoader.java:1398) at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1075) - locked <…> (a hudson.ClassicPluginStrategy$AntClassLoader2) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at … It is clearly wrong for MaskingClassLoader to be adding its own synchronization here, especially when the only apparent reason for the synchronized statements in this case was to guard access to some rarely-written lists. I will propose a patch.

          Jesse Glick added a comment -

          Unfortunately it seems that Jetty’s WebAppClassLoader does not call ClassLoader.registerAsParallelCapable

          Fixed in bug 464442 and available as of 9.3.0. Currently we bundle 9.2.x.

          Jesse Glick added a comment - Unfortunately it seems that Jetty’s WebAppClassLoader does not call ClassLoader.registerAsParallelCapable Fixed in bug 464442 and available as of 9.3.0. Currently we bundle 9.2.x .

          Code changed in jenkins
          User: Jesse Glick
          Path:
          core/src/main/java/hudson/util/MaskingClassLoader.java
          http://jenkins-ci.org/commit/jenkins/a04e2f9836263f7ae5e68617f5bafde18e594446
          Log:
          JENKINS-23784 Avoid acquiring ClassLoader locks. (#2581)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/util/MaskingClassLoader.java http://jenkins-ci.org/commit/jenkins/a04e2f9836263f7ae5e68617f5bafde18e594446 Log: JENKINS-23784 Avoid acquiring ClassLoader locks. (#2581)

          Oleg Nenashev added a comment -

          > Fixed in bug 464442 and available as of 9.3.0. Currently we bundle 9.2.x.

          IMHO we should schedule the Jetty upgrade to 9.3.x or 9.4.x as a part of the Java 7 deprecation. The problem is that 9.3.x and 9.2.x are not fully binary compatible IIRC

          Oleg Nenashev added a comment - > Fixed in bug 464442 and available as of 9.3.0. Currently we bundle 9.2.x. IMHO we should schedule the Jetty upgrade to 9.3.x or 9.4.x as a part of the Java 7 deprecation. The problem is that 9.3.x and 9.2.x are not fully binary compatible IIRC

          James Dumay added a comment - - edited

          jglickI saw that there is a bunch of PRs for this ticket that have now been merged. Whats the current status of this ticket?

          James Dumay added a comment - - edited jglick I saw that there is a bunch of PRs for this ticket that have now been merged. Whats the current status of this ticket?

          Jesse Glick added a comment -

          jamesdumay it is open. I think a fix would be easy enough, though verifying its effectiveness is another matter.

          Jesse Glick added a comment - jamesdumay it is open. I think a fix would be easy enough, though verifying its effectiveness is another matter.

            Unassigned Unassigned
            jglick Jesse Glick
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: