Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37213

java.lang.OutOfMemoryError: Compressed class space in Job DSL

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • job-dsl-plugin
    • None

      # using OpenJDK Java 8
      # and Jenkins 2.7.1
      /usr/bin/java -Xms1024m -Xmx3072m -Dorg.apache.commons.jelly.tags.fmt.timeZone=US/Mountain -jar /usr/share/jenkins/jenkins.war --webroot=/var/run/jenkins/war --httpListenAddress=127.0.0.1 --httpPort=8080 --ajp13Port=-1
      

      After a long time, I encountered the error

      FATAL: Compressed class space
      java.lang.OutOfMemoryError: Compressed class space
          at java.lang.ClassLoader.defineClass1(Native Method)
          at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
          at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts.define(ClassLoaderForClassArtifacts.java:44)
          at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts$1.run(ClassLoaderForClassArtifacts.java:77)
          at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts$1.run(ClassLoaderForClassArtifacts.java:75)
          at java.security.AccessController.doPrivileged(Native Method)
          at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts.defineClassAndGetConstructor(ClassLoaderForClassArtifacts.java:75)
          at org.codehaus.groovy.runtime.callsite.CallSiteGenerator.compilePogoMethod(CallSiteGenerator.java:222)
          at org.codehaus.groovy.reflection.CachedMethod.createPogoMetaMethodSite(CachedMethod.java:233)
          at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.createCachedMethodSite(PogoMetaMethodSite.java:150)
          at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.createPogoMetaMethodSite(PogoMetaMethodSite.java:126)
          at groovy.lang.MetaClassImpl.createPogoCallSite(MetaClassImpl.java:3405)
          at org.codehaus.groovy.runtime.callsite.CallSiteArray.createPogoSite(CallSiteArray.java:150)
          at org.codehaus.groovy.runtime.callsite.CallSiteArray.createCallSite(CallSiteArray.java:164)
          at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
          at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
          at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
          at javaposse.jobdsl.dsl.helpers.publisher.PublisherContext.archiveJunit(PublisherContext.groovy:192)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:169)
          at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
          at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52)
          at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154)
          at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:174)
          at javaposse.jobdsl.dsl.helpers.publisher.PublisherContext.archiveJunit(PublisherContext.groovy)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
      

          [JENKINS-37213] java.lang.OutOfMemoryError: Compressed class space in Job DSL

          Paul Draper created issue -
          Paul Draper made changes -
          Description Original: {code}
          # using OpenJDK Java 8
          /usr/bin/java -Xms1024m -Xmx3072m -Dorg.apache.commons.jelly.tags.fmt.timeZone=US/Mountain -jar /usr/share/jenkins/jenkins.war --webroot=/var/run/jenkins/war --httpListenAddress=127.0.0.1 --httpPort=8080 --ajp13Port=-1
          {code}

          After a long time, I encountered the error

          {code}
          FATAL: Compressed class space
          java.lang.OutOfMemoryError: Compressed class space
              at java.lang.ClassLoader.defineClass1(Native Method)
              at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
              at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts.define(ClassLoaderForClassArtifacts.java:44)
              at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts$1.run(ClassLoaderForClassArtifacts.java:77)
              at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts$1.run(ClassLoaderForClassArtifacts.java:75)
              at java.security.AccessController.doPrivileged(Native Method)
              at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts.defineClassAndGetConstructor(ClassLoaderForClassArtifacts.java:75)
              at org.codehaus.groovy.runtime.callsite.CallSiteGenerator.compilePogoMethod(CallSiteGenerator.java:222)
              at org.codehaus.groovy.reflection.CachedMethod.createPogoMetaMethodSite(CachedMethod.java:233)
              at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.createCachedMethodSite(PogoMetaMethodSite.java:150)
              at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.createPogoMetaMethodSite(PogoMetaMethodSite.java:126)
              at groovy.lang.MetaClassImpl.createPogoCallSite(MetaClassImpl.java:3405)
              at org.codehaus.groovy.runtime.callsite.CallSiteArray.createPogoSite(CallSiteArray.java:150)
              at org.codehaus.groovy.runtime.callsite.CallSiteArray.createCallSite(CallSiteArray.java:164)
              at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
              at javaposse.jobdsl.dsl.helpers.publisher.PublisherContext.archiveJunit(PublisherContext.groovy:192)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:169)
              at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
              at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:174)
              at javaposse.jobdsl.dsl.helpers.publisher.PublisherContext.archiveJunit(PublisherContext.groovy)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
          {code}
          New: {code}
          # using OpenJDK Java 8
          # and Jenkins 2.7.1
          /usr/bin/java -Xms1024m -Xmx3072m -Dorg.apache.commons.jelly.tags.fmt.timeZone=US/Mountain -jar /usr/share/jenkins/jenkins.war --webroot=/var/run/jenkins/war --httpListenAddress=127.0.0.1 --httpPort=8080 --ajp13Port=-1
          {code}

          After a long time, I encountered the error

          {code}
          FATAL: Compressed class space
          java.lang.OutOfMemoryError: Compressed class space
              at java.lang.ClassLoader.defineClass1(Native Method)
              at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
              at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts.define(ClassLoaderForClassArtifacts.java:44)
              at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts$1.run(ClassLoaderForClassArtifacts.java:77)
              at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts$1.run(ClassLoaderForClassArtifacts.java:75)
              at java.security.AccessController.doPrivileged(Native Method)
              at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts.defineClassAndGetConstructor(ClassLoaderForClassArtifacts.java:75)
              at org.codehaus.groovy.runtime.callsite.CallSiteGenerator.compilePogoMethod(CallSiteGenerator.java:222)
              at org.codehaus.groovy.reflection.CachedMethod.createPogoMetaMethodSite(CachedMethod.java:233)
              at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.createCachedMethodSite(PogoMetaMethodSite.java:150)
              at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.createPogoMetaMethodSite(PogoMetaMethodSite.java:126)
              at groovy.lang.MetaClassImpl.createPogoCallSite(MetaClassImpl.java:3405)
              at org.codehaus.groovy.runtime.callsite.CallSiteArray.createPogoSite(CallSiteArray.java:150)
              at org.codehaus.groovy.runtime.callsite.CallSiteArray.createCallSite(CallSiteArray.java:164)
              at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
              at javaposse.jobdsl.dsl.helpers.publisher.PublisherContext.archiveJunit(PublisherContext.groovy:192)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:169)
              at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
              at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:174)
              at javaposse.jobdsl.dsl.helpers.publisher.PublisherContext.archiveJunit(PublisherContext.groovy)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
          {code}

          Is this reproducible? Can you analyze the heap to see what's consuming memory?

          Daniel Spilker added a comment - Is this reproducible? Can you analyze the heap to see what's consuming memory?

          Paul Draper added a comment -

          Yes, apparently. It happens almost once a week for me since upgrading from 1.42 to 1.45.

          Next time, I'll get a heap dump and metaspace info.

          Paul Draper added a comment - Yes, apparently. It happens almost once a week for me since upgrading from 1.42 to 1.45. Next time, I'll get a heap dump and metaspace info.

          mcrooney added a comment -

          We've seen this as well a few times with Job DSL 1.48 and Jenkins 2.7.2. I've got a heap dump (2.6GB), daspilker, I can give you an S3 link or if it is trivial to analyze myself, let me know what to look for.

          mcrooney added a comment - We've seen this as well a few times with Job DSL 1.48 and Jenkins 2.7.2. I've got a heap dump (2.6GB), daspilker , I can give you an S3 link or if it is trivial to analyze myself, let me know what to look for.
          Daniel Spilker made changes -
          Link New: This issue is related to JENKINS-33358 [ JENKINS-33358 ]

          draperp mcrooney Can you try to set the -Dgroovy.use.classvalue=true system property as suggested in JENKINS-33358 and report the results?

          Daniel Spilker added a comment - draperp mcrooney Can you try to set the -Dgroovy.use.classvalue=true system property as suggested in JENKINS-33358 and report the results?

          mcrooney added a comment - - edited

          We had our third outage from this so I've now restarted it with that option, would "java -Dgroovy.use.classvalue=true -jar jenkins.war" be a valid invocation to use this? I'll report back the results However, should this matter in Java 8 (Hotspot) where PermGen is gone? https://www.infoq.com/articles/Java-PERMGEN-Removed

          mcrooney added a comment - - edited We had our third outage from this so I've now restarted it with that option, would "java -Dgroovy.use.classvalue=true -jar jenkins.war" be a valid invocation to use this? I'll report back the results However, should this matter in Java 8 (Hotspot) where PermGen is gone? https://www.infoq.com/articles/Java-PERMGEN-Removed
          Paul Draper made changes -
          Attachment New: class_stats.gz [ 33895 ]

          Paul Draper added a comment - - edited

          I attached the gzipped results of

          jcmd <pid> GC.class_stats
          

          You'll notice 1 million ..._run_closure... classes.

          To give some background for our setup...."mainJobs" is run in parameterized job that run on each branch push. It creates a different folder for each branch, and creates and triggers a "jobs" job in that folder that runs and produces the jobs in that directory. Periodically, folders for deleted branches are removed.

          Paul Draper added a comment - - edited I attached the gzipped results of jcmd <pid> GC.class_stats You'll notice 1 million ..._run_closure... classes. To give some background for our setup...."mainJobs" is run in parameterized job that run on each branch push. It creates a different folder for each branch, and creates and triggers a "jobs" job in that folder that runs and produces the jobs in that directory. Periodically, folders for deleted branches are removed.

            daspilker Daniel Spilker
            draperp Paul Draper
            Votes:
            3 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: