Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63766

Metaspace memory leak in Pipeline when running on Java 11 (but not 8 or 17)

    • 2.350, 2.346.1

      Hello,

      If the following pipeline script is run as a pipeline job in Jenkins, The instantiated class is not collected by the garbage collector:

      class asdf {
        void run () {}
      }
      
      new asdf().run()
      

      Even after a garbage collection is run using jcmd, the instantiated class is still there:

      $ jcmd <PID> GC.run
      
      # wait for garbage collection to complete
      
      $ jcmd <PID> GC.class_stats | grep asdf
      11804    13         0        640          48     3344          13       470      2472     1904      5008      6912 asdf
      

      With one class or a few classes, this issue might not lead to any problems. But our pipelines have a lot of classes and we're encountering a steady increase in the MetaSpace and OutOfMemoryExceptions (CompressedClassSpace) after every 2 to 3 weeks. Additionally the Metaspace increases by several GBs per week to about 10 GB in 2 to 3 weeks.

       

      Some more information about the environment:

      • Operating System: Jenkins-Master runs in a debian-slim docker container
      • JRE/JDK vendors and versions: OpenJDK 11.0.6
      • Jenkins and plugin versions
        • Jenkins 2.235.3
        • Plugins
          ace-editor:1.1 - https://plugins.jenkins.io/ace-editor
          apache-httpcomponents-client-4-api:4.5.10-2.0 - https://plugins.jenkins.io/apache-httpcomponents-client-4-api
          authentication-tokens:1.4 - https://plugins.jenkins.io/authentication-tokens
          authorize-project:1.3.0 - https://plugins.jenkins.io/authorize-project
          blueocean-autofavorite:1.2.4 - https://plugins.jenkins.io/blueocean-autofavorite
          blueocean-bitbucket-pipeline:1.24.0 - https://plugins.jenkins.io/blueocean-bitbucket-pipeline
          blueocean-commons:1.24.0 - https://plugins.jenkins.io/blueocean-commons
          blueocean-config:1.24.0 - https://plugins.jenkins.io/blueocean-config
          blueocean-core-js:1.24.0 - https://plugins.jenkins.io/blueocean-core-js
          blueocean-dashboard:1.24.0 - https://plugins.jenkins.io/blueocean-dashboard
          blueocean-display-url:2.4.0 - https://plugins.jenkins.io/blueocean-display-url
          blueocean-events:1.24.0 - https://plugins.jenkins.io/blueocean-events
          blueocean-git-pipeline:1.24.0 - https://plugins.jenkins.io/blueocean-git-pipeline
          blueocean-github-pipeline:1.24.0 - https://plugins.jenkins.io/blueocean-github-pipeline
          blueocean-i18n:1.24.0 - https://plugins.jenkins.io/blueocean-i18n
          blueocean-jira:1.24.0 - https://plugins.jenkins.io/blueocean-jira
          blueocean-jwt:1.24.0 - https://plugins.jenkins.io/blueocean-jwt
          blueocean-personalization:1.24.0 - https://plugins.jenkins.io/blueocean-personalization
          blueocean-pipeline-api-impl:1.24.0 - https://plugins.jenkins.io/blueocean-pipeline-api-impl
          blueocean-pipeline-editor:1.24.0 - https://plugins.jenkins.io/blueocean-pipeline-editor
          blueocean-pipeline-scm-api:1.24.0 - https://plugins.jenkins.io/blueocean-pipeline-scm-api
          blueocean-rest-impl:1.24.0 - https://plugins.jenkins.io/blueocean-rest-impl
          blueocean-rest:1.24.0 - https://plugins.jenkins.io/blueocean-rest
          blueocean-web:1.24.0 - https://plugins.jenkins.io/blueocean-web
          blueocean:1.23.2 - https://plugins.jenkins.io/blueocean
          branch-api:2.6.0 - https://plugins.jenkins.io/branch-api
          cloudbees-bitbucket-branch-source:2.9.1 - https://plugins.jenkins.io/cloudbees-bitbucket-branch-source
          cloudbees-folder:6.14 - https://plugins.jenkins.io/cloudbees-folder
          conditional-buildstep:1.3.6 - https://plugins.jenkins.io/conditional-buildstep
          configuration-as-code:1.42 - https://plugins.jenkins.io/configuration-as-code
          credentials-binding:1.23 - https://plugins.jenkins.io/credentials-binding
          credentials:2.3.13 - https://plugins.jenkins.io/credentials
          display-url-api:2.3.3 - https://plugins.jenkins.io/display-url-api
          docker-commons:1.17 - https://plugins.jenkins.io/docker-commons
          docker-workflow:1.24 - https://plugins.jenkins.io/docker-workflow
          durable-task:1.35 - https://plugins.jenkins.io/durable-task
          echarts-api:4.8.0-2 - https://plugins.jenkins.io/echarts-api
          favorite:2.3.2 - https://plugins.jenkins.io/favorite
          git-client:3.5.0 - https://plugins.jenkins.io/git-client
          git-server:1.9 - https://plugins.jenkins.io/git-server
          git:4.4.3 - https://plugins.jenkins.io/git
          github-api:1.116 - https://plugins.jenkins.io/github-api
          github-branch-source:2.9.0 - https://plugins.jenkins.io/github-branch-source
          github:1.31.0 - https://plugins.jenkins.io/github
          handlebars:1.1.1 - https://plugins.jenkins.io/handlebars
          handy-uri-templates-2-api:2.1.8-1.0 - https://plugins.jenkins.io/handy-uri-templates-2-api
          htmlpublisher:1.23 - https://plugins.jenkins.io/htmlpublisher
          http_request:1.8.26 - https://plugins.jenkins.io/http_request
          jackson2-api:2.11.2 - https://plugins.jenkins.io/jackson2-api
          javadoc:1.6 - https://plugins.jenkins.io/javadoc
          jenkins-design-language:1.24.0 - https://plugins.jenkins.io/jenkins-design-language
          jira:3.1.1 - https://plugins.jenkins.io/jira
          job-dsl:1.77 - https://plugins.jenkins.io/job-dsl
          jobConfigHistory:2.26 - https://plugins.jenkins.io/jobConfigHistory
          jquery-detached:1.2.1 - https://plugins.jenkins.io/jquery-detached
          jquery3-api:3.5.1-1 - https://plugins.jenkins.io/jquery3-api
          jsch:0.1.55.2 - https://plugins.jenkins.io/jsch
          junit:1.35 - https://plugins.jenkins.io/junit
          lockable-resources:2.8 - https://plugins.jenkins.io/lockable-resources
          mailer:1.32 - https://plugins.jenkins.io/mailer
          matrix-auth:2.6.2 - https://plugins.jenkins.io/matrix-auth
          matrix-project:1.17 - https://plugins.jenkins.io/matrix-project
          maven-plugin:3.7 - https://plugins.jenkins.io/maven-plugin
          mercurial:2.10 - https://plugins.jenkins.io/mercurial
          metrics:4.0.2.6 - https://plugins.jenkins.io/metrics
          momentjs:1.1.1 - https://plugins.jenkins.io/momentjs
          okhttp-api:3.14.9 - https://plugins.jenkins.io/okhttp-api
          parameterized-trigger:2.37 - https://plugins.jenkins.io/parameterized-trigger
          pipeline-build-step:2.13 - https://plugins.jenkins.io/pipeline-build-step
          pipeline-graph-analysis:1.10 - https://plugins.jenkins.io/pipeline-graph-analysis
          pipeline-input-step:2.12 - https://plugins.jenkins.io/pipeline-input-step
          pipeline-milestone-step:1.3.1 - https://plugins.jenkins.io/pipeline-milestone-step
          pipeline-model-api:1.7.2 - https://plugins.jenkins.io/pipeline-model-api
          pipeline-model-definition:1.7.2 - https://plugins.jenkins.io/pipeline-model-definition
          pipeline-model-extensions:1.7.2 - https://plugins.jenkins.io/pipeline-model-extensions
          pipeline-rest-api:2.15 - https://plugins.jenkins.io/pipeline-rest-api
          pipeline-stage-step:2.5 - https://plugins.jenkins.io/pipeline-stage-step
          pipeline-stage-tags-metadata:1.7.2 - https://plugins.jenkins.io/pipeline-stage-tags-metadata
          pipeline-stage-view:2.15 - https://plugins.jenkins.io/pipeline-stage-view
          pipeline-utility-steps:2.6.1 - https://plugins.jenkins.io/pipeline-utility-steps
          plain-credentials:1.7 - https://plugins.jenkins.io/plain-credentials
          plugin-util-api:1.2.5 - https://plugins.jenkins.io/plugin-util-api
          pubsub-light:1.13 - https://plugins.jenkins.io/pubsub-light
          resource-disposer:0.14 - https://plugins.jenkins.io/resource-disposer
          run-condition:1.3 - https://plugins.jenkins.io/run-condition
          scm-api:2.6.3 - https://plugins.jenkins.io/scm-api
          script-security:1.74 - https://plugins.jenkins.io/script-security
          snakeyaml-api:1.27.0 - https://plugins.jenkins.io/snakeyaml-api
          sse-gateway:1.23 - https://plugins.jenkins.io/sse-gateway
          ssh-credentials:1.18.1 - https://plugins.jenkins.io/ssh-credentials
          ssh-steps:2.0.0 - https://plugins.jenkins.io/ssh-steps
          structs:1.20 - https://plugins.jenkins.io/structs
          swarm:3.22 - https://plugins.jenkins.io/swarm
          thinBackup:1.9 - https://plugins.jenkins.io/thinBackup
          token-macro:2.12 - https://plugins.jenkins.io/token-macro
          trilead-api:1.0.10 - https://plugins.jenkins.io/trilead-api
          variant:1.3 - https://plugins.jenkins.io/variant
          webhook-step:1.4 - https://plugins.jenkins.io/webhook-step
          workflow-aggregator:2.6 - https://plugins.jenkins.io/workflow-aggregator
          workflow-api:2.40 - https://plugins.jenkins.io/workflow-api
          workflow-basic-steps:2.21 - https://plugins.jenkins.io/workflow-basic-steps
          workflow-cps-global-lib:2.17 - https://plugins.jenkins.io/workflow-cps-global-lib
          workflow-cps:2.83 - https://plugins.jenkins.io/workflow-cps
          workflow-durable-task-step:2.35 - https://plugins.jenkins.io/workflow-durable-task-step
          workflow-job:2.40 - https://plugins.jenkins.io/workflow-job
          workflow-multibranch:2.22 - https://plugins.jenkins.io/workflow-multibranch
          workflow-scm-step:2.11 - https://plugins.jenkins.io/workflow-scm-step
          workflow-step-api:2.22 - https://plugins.jenkins.io/workflow-step-api
          workflow-support:3.5 - https://plugins.jenkins.io/workflow-support
          ws-cleanup:0.38 - https://plugins.jenkins.io/ws-cleanup
          bouncycastle-api:2.16.0 - https://plugins.jenkins.io/bouncycastle-api
          command-launcher:1.2 - https://plugins.jenkins.io/command-launcher
          jdk-tool:1.0 - https://plugins.jenkins.io/jdk-tool
          jaxb:2.3.0 - https://plugins.jenkins.io/jaxb
          configuration-as-code-groovy:1.1 - https://plugins.jenkins.io/configuration-as-code-groovy
          bootstrap4-api:4.5.2-1 - https://plugins.jenkins.io/bootstrap4-api
          font-awesome-api:5.14.0-1 - https://plugins.jenkins.io/font-awesome-api
          popper-api:1.16.0-6 - https://plugins.jenkins.io/popper-api
          

          [JENKINS-63766] Metaspace memory leak in Pipeline when running on Java 11 (but not 8 or 17)

          Jesse Glick added a comment -

          Tried to reproduce using MemoryAssert.assertGC without success. If reproducible, someone would need to look at a heap dump and see what is actually being held, and why https://github.com/jenkinsci/workflow-cps-plugin/blob/52d3de6044d0d3373ce513a5ff2f1de15d572d5d/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L1281-L1436 does not take care of it.

          Jesse Glick added a comment - Tried to reproduce using MemoryAssert.assertGC without success. If reproducible, someone would need to look at a heap dump and see what is actually being held, and why https://github.com/jenkinsci/workflow-cps-plugin/blob/52d3de6044d0d3373ce513a5ff2f1de15d572d5d/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L1281-L1436 does not take care of it.

          Basil Crow added a comment -

          I did a clean installation of Jenkins with nothing but the suggested plugins and ran the Pipeline script provided by the reporter. I confirmed (by increasing logging) that https://github.com/jenkinsci/workflow-cps-plugin/blob/52d3de6044d0d3373ce513a5ff2f1de15d572d5d/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L1281-L1436 found the asdf class as expected. I then took a heap dump and searched for the asdf class in the heap dump. I found asdf in the list of objects, with a count of 0 and a retained size of 0 bytes. Based on the above, I conclude the original issue as submitted by the reporter is no longer valid. I am closing this bug as Cannot Reproduce.

          Basil Crow added a comment - I did a clean installation of Jenkins with nothing but the suggested plugins and ran the Pipeline script provided by the reporter. I confirmed (by increasing logging) that https://github.com/jenkinsci/workflow-cps-plugin/blob/52d3de6044d0d3373ce513a5ff2f1de15d572d5d/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L1281-L1436 found the asdf class as expected. I then took a heap dump and searched for the asdf class in the heap dump. I found asdf in the list of objects, with a count of 0 and a retained size of 0 bytes. Based on the above, I conclude the original issue as submitted by the reporter is no longer valid. I am closing this bug as Cannot Reproduce.

          Stefan Hölzle added a comment - - edited

          basil,
          Please note that the Metaspace is not part of the Java Heap Memory.
          And It's not the Java Heap that grows from that Memory leak, but the Metaspace, more specifically the CompressedClassSpace part of the Metaspace.
          So it's plausible for me that you don't see any space blocked in the Heap Memory.

          The issue ist still present in Jenkins 2.344 run in a Java 11 JVM.

          Stefan Hölzle added a comment - - edited basil , Please note that the Metaspace is not part of the Java Heap Memory. And It's not the Java Heap that grows from that Memory leak, but the Metaspace, more specifically the CompressedClassSpace part of the Metaspace. So it's plausible for me that you don't see any space blocked in the Heap Memory. The issue ist still present in Jenkins 2.344 run in a Java 11 JVM.

          Basil Crow added a comment -

          Thank you shoelzle for the clarification. I can now reproduce the problem. I created a new Pipeline job with

          class asdf {
            void run () {}
          }
          
          new asdf().run()
          

          When running the above Pipeline under Java 8 with -verbose:class:

          [Loaded asdf from file:/groovy/shell]
          [Loaded WorkflowScript from file:/groovy/shell]
          […] // jcmd ${PID} GC.run
          [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor33 0x00000007c0878428]
          [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor32 0x00000007c0878028]
          [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor31 0x00000007c0877c28]
          [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor30 0x00000007c0877828]
          [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor29 0x00000007c0877428]
          [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor28 0x00000007c0877028]
          [Unloading class WorkflowScript 0x00000007c084b290]
          [Unloading class asdf 0x00000007c084b028]
          [Unloading class org.eclipse.jetty.server.session.DefaultSessionCache$$Lambda$302/1809148827 0x00000007c07ec828]
          [Loaded sun.reflect.GeneratedConstructorAccessor84 from __JVM_DefineClass__]
          

          When running the same Pipeline under Java 11 with -verbose:class:

          [10.332s][info][class,load  ] asdf source: file:/groovy/shell
          [10.337s][info][class,load  ] WorkflowScript source: file:/groovy/shell
          […] // jcmd ${PID} GC.run
          [33.509s][info][class,load  ] jdk.internal.reflect.GeneratedConstructorAccessor81 source: __JVM_DefineClass__
          [38.551s][info][class,load  ] jdk.internal.reflect.GeneratedConstructorAccessor82 source: __JVM_DefineClass__
          [38.552s][info][class,load  ] jdk.internal.reflect.GeneratedMethodAccessor27 source: __JVM_DefineClass__
          [38.553s][info][class,load  ] jdk.internal.reflect.GeneratedMethodAccessor28 source: __JVM_DefineClass__
          

          It can clearly be seen that the asdf class is loaded in both cases but is only unloaded from the Metaspace in Java 8.

          Basil Crow added a comment - Thank you shoelzle for the clarification. I can now reproduce the problem. I created a new Pipeline job with class asdf { void run () {} } new asdf().run() When running the above Pipeline under Java 8 with -verbose:class : [Loaded asdf from file:/groovy/shell] [Loaded WorkflowScript from file:/groovy/shell] […] // jcmd ${PID} GC.run [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor33 0x00000007c0878428] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor32 0x00000007c0878028] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor31 0x00000007c0877c28] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor30 0x00000007c0877828] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor29 0x00000007c0877428] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor28 0x00000007c0877028] [Unloading class WorkflowScript 0x00000007c084b290] [Unloading class asdf 0x00000007c084b028] [Unloading class org.eclipse.jetty.server.session.DefaultSessionCache$$Lambda$302/1809148827 0x00000007c07ec828] [Loaded sun.reflect.GeneratedConstructorAccessor84 from __JVM_DefineClass__] When running the same Pipeline under Java 11 with -verbose:class : [10.332s][info][class,load ] asdf source: file:/groovy/shell [10.337s][info][class,load ] WorkflowScript source: file:/groovy/shell […] // jcmd ${PID} GC.run [33.509s][info][class,load ] jdk.internal.reflect.GeneratedConstructorAccessor81 source: __JVM_DefineClass__ [38.551s][info][class,load ] jdk.internal.reflect.GeneratedConstructorAccessor82 source: __JVM_DefineClass__ [38.552s][info][class,load ] jdk.internal.reflect.GeneratedMethodAccessor27 source: __JVM_DefineClass__ [38.553s][info][class,load ] jdk.internal.reflect.GeneratedMethodAccessor28 source: __JVM_DefineClass__ It can clearly be seen that the asdf class is loaded in both cases but is only unloaded from the Metaspace in Java 8.

          Basil Crow added a comment - - edited

          The problem appears to be resolved when running under Java 17:

          [38.700s][info][class,load] asdf source: file:/groovy/shell
          [38.702s][info][class,load] WorkflowScript source: file:/groovy/shell
          […] // jcmd ${PID} GC.run
          [63.316s][info][class,load] jdk.internal.reflect.GeneratedMethodAccessor76 source: __ClassDefiner__
          [63.317s][info][class,load] jdk.internal.reflect.GeneratedMethodAccessor77 source: __ClassDefiner__
          [64.378s][info][class,unload] unloading class jdk.internal.reflect.GeneratedSerializationConstructorAccessor33 0x00000008013b4400
          [64.378s][info][class,unload] unloading class jdk.internal.reflect.GeneratedSerializationConstructorAccessor32 0x00000008013b4000
          [64.378s][info][class,unload] unloading class jdk.internal.reflect.GeneratedSerializationConstructorAccessor31 0x00000008013b3c00
          [64.378s][info][class,unload] unloading class jdk.internal.reflect.GeneratedSerializationConstructorAccessor30 0x00000008013b3800
          [64.378s][info][class,unload] unloading class jdk.internal.reflect.GeneratedSerializationConstructorAccessor29 0x00000008013b3400
          [64.378s][info][class,unload] unloading class jdk.internal.reflect.GeneratedSerializationConstructorAccessor28 0x00000008013b3000
          [64.378s][info][class,unload] unloading class WorkflowScript 0x0000000801394280
          [64.378s][info][class,unload] unloading class asdf 0x0000000801394000
          

          Basil Crow added a comment - - edited The problem appears to be resolved when running under Java 17: [38.700s][info][class,load] asdf source: file:/groovy/shell [38.702s][info][class,load] WorkflowScript source: file:/groovy/shell […] // jcmd ${PID} GC.run [63.316s][info][class,load] jdk.internal.reflect.GeneratedMethodAccessor76 source: __ClassDefiner__ [63.317s][info][class,load] jdk.internal.reflect.GeneratedMethodAccessor77 source: __ClassDefiner__ [64.378s][info][class,unload] unloading class jdk.internal.reflect.GeneratedSerializationConstructorAccessor33 0x00000008013b4400 [64.378s][info][class,unload] unloading class jdk.internal.reflect.GeneratedSerializationConstructorAccessor32 0x00000008013b4000 [64.378s][info][class,unload] unloading class jdk.internal.reflect.GeneratedSerializationConstructorAccessor31 0x00000008013b3c00 [64.378s][info][class,unload] unloading class jdk.internal.reflect.GeneratedSerializationConstructorAccessor30 0x00000008013b3800 [64.378s][info][class,unload] unloading class jdk.internal.reflect.GeneratedSerializationConstructorAccessor29 0x00000008013b3400 [64.378s][info][class,unload] unloading class jdk.internal.reflect.GeneratedSerializationConstructorAccessor28 0x00000008013b3000 [64.378s][info][class,unload] unloading class WorkflowScript 0x0000000801394280 [64.378s][info][class,unload] unloading class asdf 0x0000000801394000

          Basil Crow added a comment -

          I ran the above test 10 times (all on Linux x86_64) and found the following results:

          Temurin 1.8.0_332-b09 - PASS
          AdoptOpenJDK 9.0.4+11 - FAIL
          AdoptOpenJDK 10.0.2+13 - FAIL
          Temurin 11.0.15+10 - FAIL
          AdoptOpenJDK 12.0.2+10 - FAIL
          AdoptOpenJDK 13.0.2+8 - FAIL
          AdoptOpenJDK 14.0.2+12 - FAIL
          AdoptOpenJDK 15.0.2+7 - FAIL
          Temurin 16.0.2+7 - PASS
          Temurin 17.0.3+7 - PASS

          So Java versions 9 through 15 are affected.

          Basil Crow added a comment - I ran the above test 10 times (all on Linux x86_64) and found the following results: Temurin 1.8.0_332-b09 - PASS AdoptOpenJDK 9.0.4+11 - FAIL AdoptOpenJDK 10.0.2+13 - FAIL Temurin 11.0.15+10 - FAIL AdoptOpenJDK 12.0.2+10 - FAIL AdoptOpenJDK 13.0.2+8 - FAIL AdoptOpenJDK 14.0.2+12 - FAIL AdoptOpenJDK 15.0.2+7 - FAIL Temurin 16.0.2+7 - PASS Temurin 17.0.3+7 - PASS So Java versions 9 through 15 are affected.

          Basil Crow added a comment -

          The last failing commit is openjdk/jdk@42a6eadb853. The first passing commit is openjdk/jdk@2ee2b4ae19d. I confirmed that backporting openjdk/jdk@2ee2b4ae19d to jdk11u-dev resolves the problem on Java 11.

          Basil Crow added a comment - The last failing commit is openjdk/jdk@ 42a6eadb853 . The first passing commit is openjdk/jdk@ 2ee2b4ae19d . I confirmed that backporting openjdk/jdk@ 2ee2b4ae19d to jdk11u-dev resolves the problem on Java 11.

          Basil Crow added a comment -

          The upstream bug is JDK-8231454. As described in the upstream bug, a memory memory leak can occur when java.beans.Introspector#getBeanInfo is invoked against a class, even if java.beans.Introspector#flushFromCaches is later called on that class.

          In our case, java.beans.Introspector#getBeanInfo is invoked from the following stack trace:

          getBeanInfo:196, Introspector (java.beans)
          run:3328, MetaClassImpl$15 (groovy.lang)
          doPrivileged:-1, AccessController (java.security)
          addProperties:3326, MetaClassImpl (groovy.lang)
          initialize:3303, MetaClassImpl (groovy.lang)
          getMetaClassUnderLock:289, ClassInfo (org.codehaus.groovy.reflection)
          getMetaClass:331, ClassInfo (org.codehaus.groovy.reflection)
          getMetaClass:277, MetaClassRegistryImpl (org.codehaus.groovy.runtime.metaclass)
          getMetaClass:905, InvokerHelper (org.codehaus.groovy.runtime)
          createCallConstructorSite:86, CallSiteArray (org.codehaus.groovy.runtime.callsite)
          defaultCallConstructor:59, CallSiteArray (org.codehaus.groovy.runtime.callsite)
          callConstructor:238, AbstractCallSite (org.codehaus.groovy.runtime.callsite)
          call:208, Checker$3 (org.kohsuke.groovy.sandbox.impl)
          onNewInstance:42, GroovyInterceptor (org.kohsuke.groovy.sandbox)
          onNewInstance:173, SandboxInterceptor (org.jenkinsci.plugins.scriptsecurity.sandbox.groovy)
          call:205, Checker$3 (org.kohsuke.groovy.sandbox.impl)
          checkedConstructor:210, Checker (org.kohsuke.groovy.sandbox.impl)
          constructorCall:21, SandboxInvoker (com.cloudbees.groovy.cps.sandbox)
          dispatchOrArg:97, FunctionCallBlock$ContinuationImpl (com.cloudbees.groovy.cps.impl)
          fixName:78, FunctionCallBlock$ContinuationImpl (com.cloudbees.groovy.cps.impl)
          invoke0:-1, NativeMethodAccessorImpl (jdk.internal.reflect)
          invoke:62, NativeMethodAccessorImpl (jdk.internal.reflect)
          invoke:43, DelegatingMethodAccessorImpl (jdk.internal.reflect)
          invoke:566, Method (java.lang.reflect)
          receive:72, ContinuationPtr$ContinuationImpl (com.cloudbees.groovy.cps.impl)
          eval:21, ConstantBlock (com.cloudbees.groovy.cps.impl)
          step:83, Next (com.cloudbees.groovy.cps)
          call:174, Continuable$1 (com.cloudbees.groovy.cps)
          call:163, Continuable$1 (com.cloudbees.groovy.cps)
          use:136, GroovyCategorySupport$ThreadCategoryInfo (org.codehaus.groovy.runtime)
          use:275, GroovyCategorySupport (org.codehaus.groovy.runtime)
          run0:163, Continuable (com.cloudbees.groovy.cps)
          access$001:18, SandboxContinuable (org.jenkinsci.plugins.workflow.cps)
          run0:51, SandboxContinuable (org.jenkinsci.plugins.workflow.cps)
          runNextChunk:187, CpsThread (org.jenkinsci.plugins.workflow.cps)
          run:420, CpsThreadGroup (org.jenkinsci.plugins.workflow.cps)
          access$400:95, CpsThreadGroup (org.jenkinsci.plugins.workflow.cps)
          call:330, CpsThreadGroup$2 (org.jenkinsci.plugins.workflow.cps)
          call:294, CpsThreadGroup$2 (org.jenkinsci.plugins.workflow.cps)
          call:67, CpsVmExecutorService$2 (org.jenkinsci.plugins.workflow.cps)
          run:264, FutureTask (java.util.concurrent)
          run:139, SingleLaneExecutorService$1 (hudson.remoting)
          run:28, ContextResettingExecutorService$1 (jenkins.util)
          run:68, ImpersonatingExecutorService$1 (jenkins.security)
          call:515, Executors$RunnableAdapter (java.util.concurrent)
          run:264, FutureTask (java.util.concurrent)
          runWorker:1128, ThreadPoolExecutor (java.util.concurrent)
          run:628, ThreadPoolExecutor$Worker (java.util.concurrent)
          run:829, Thread (java.lang)
          

          java.beans.Introspector#flushFromCaches is later called on that class, but to no avail:

          cleanUpLoader:1300, CpsFlowExecution (org.jenkinsci.plugins.workflow.cps)
          cleanUpHeap:1269, CpsFlowExecution (org.jenkinsci.plugins.workflow.cps)
          run:464, CpsThreadGroup (org.jenkinsci.plugins.workflow.cps)
          access$400:95, CpsThreadGroup (org.jenkinsci.plugins.workflow.cps)
          call:330, CpsThreadGroup$2 (org.jenkinsci.plugins.workflow.cps)
          call:294, CpsThreadGroup$2 (org.jenkinsci.plugins.workflow.cps)
          call:67, CpsVmExecutorService$2 (org.jenkinsci.plugins.workflow.cps)
          run:264, FutureTask (java.util.concurrent)
          run:139, SingleLaneExecutorService$1 (hudson.remoting)
          run:28, ContextResettingExecutorService$1 (jenkins.util)
          run:68, ImpersonatingExecutorService$1 (jenkins.security)
          call:515, Executors$RunnableAdapter (java.util.concurrent)
          run:264, FutureTask (java.util.concurrent)
          runWorker:1128, ThreadPoolExecutor (java.util.concurrent)
          run:628, ThreadPoolExecutor$Worker (java.util.concurrent)
          run:829, Thread (java.lang)
          

          Basil Crow added a comment - The upstream bug is JDK-8231454 . As described in the upstream bug, a memory memory leak can occur when java.beans.Introspector#getBeanInfo is invoked against a class, even if java.beans.Introspector#flushFromCaches is later called on that class. In our case, java.beans.Introspector#getBeanInfo is invoked from the following stack trace: getBeanInfo:196, Introspector (java.beans) run:3328, MetaClassImpl$15 (groovy.lang) doPrivileged:-1, AccessController (java.security) addProperties:3326, MetaClassImpl (groovy.lang) initialize:3303, MetaClassImpl (groovy.lang) getMetaClassUnderLock:289, ClassInfo (org.codehaus.groovy.reflection) getMetaClass:331, ClassInfo (org.codehaus.groovy.reflection) getMetaClass:277, MetaClassRegistryImpl (org.codehaus.groovy.runtime.metaclass) getMetaClass:905, InvokerHelper (org.codehaus.groovy.runtime) createCallConstructorSite:86, CallSiteArray (org.codehaus.groovy.runtime.callsite) defaultCallConstructor:59, CallSiteArray (org.codehaus.groovy.runtime.callsite) callConstructor:238, AbstractCallSite (org.codehaus.groovy.runtime.callsite) call:208, Checker$3 (org.kohsuke.groovy.sandbox.impl) onNewInstance:42, GroovyInterceptor (org.kohsuke.groovy.sandbox) onNewInstance:173, SandboxInterceptor (org.jenkinsci.plugins.scriptsecurity.sandbox.groovy) call:205, Checker$3 (org.kohsuke.groovy.sandbox.impl) checkedConstructor:210, Checker (org.kohsuke.groovy.sandbox.impl) constructorCall:21, SandboxInvoker (com.cloudbees.groovy.cps.sandbox) dispatchOrArg:97, FunctionCallBlock$ContinuationImpl (com.cloudbees.groovy.cps.impl) fixName:78, FunctionCallBlock$ContinuationImpl (com.cloudbees.groovy.cps.impl) invoke0:-1, NativeMethodAccessorImpl (jdk.internal.reflect) invoke:62, NativeMethodAccessorImpl (jdk.internal.reflect) invoke:43, DelegatingMethodAccessorImpl (jdk.internal.reflect) invoke:566, Method (java.lang.reflect) receive:72, ContinuationPtr$ContinuationImpl (com.cloudbees.groovy.cps.impl) eval:21, ConstantBlock (com.cloudbees.groovy.cps.impl) step:83, Next (com.cloudbees.groovy.cps) call:174, Continuable$1 (com.cloudbees.groovy.cps) call:163, Continuable$1 (com.cloudbees.groovy.cps) use:136, GroovyCategorySupport$ThreadCategoryInfo (org.codehaus.groovy.runtime) use:275, GroovyCategorySupport (org.codehaus.groovy.runtime) run0:163, Continuable (com.cloudbees.groovy.cps) access$001:18, SandboxContinuable (org.jenkinsci.plugins.workflow.cps) run0:51, SandboxContinuable (org.jenkinsci.plugins.workflow.cps) runNextChunk:187, CpsThread (org.jenkinsci.plugins.workflow.cps) run:420, CpsThreadGroup (org.jenkinsci.plugins.workflow.cps) access$400:95, CpsThreadGroup (org.jenkinsci.plugins.workflow.cps) call:330, CpsThreadGroup$2 (org.jenkinsci.plugins.workflow.cps) call:294, CpsThreadGroup$2 (org.jenkinsci.plugins.workflow.cps) call:67, CpsVmExecutorService$2 (org.jenkinsci.plugins.workflow.cps) run:264, FutureTask (java.util.concurrent) run:139, SingleLaneExecutorService$1 (hudson.remoting) run:28, ContextResettingExecutorService$1 (jenkins.util) run:68, ImpersonatingExecutorService$1 (jenkins.security) call:515, Executors$RunnableAdapter (java.util.concurrent) run:264, FutureTask (java.util.concurrent) runWorker:1128, ThreadPoolExecutor (java.util.concurrent) run:628, ThreadPoolExecutor$Worker (java.util.concurrent) run:829, Thread (java.lang) java.beans.Introspector#flushFromCaches is later called on that class, but to no avail: cleanUpLoader:1300, CpsFlowExecution (org.jenkinsci.plugins.workflow.cps) cleanUpHeap:1269, CpsFlowExecution (org.jenkinsci.plugins.workflow.cps) run:464, CpsThreadGroup (org.jenkinsci.plugins.workflow.cps) access$400:95, CpsThreadGroup (org.jenkinsci.plugins.workflow.cps) call:330, CpsThreadGroup$2 (org.jenkinsci.plugins.workflow.cps) call:294, CpsThreadGroup$2 (org.jenkinsci.plugins.workflow.cps) call:67, CpsVmExecutorService$2 (org.jenkinsci.plugins.workflow.cps) run:264, FutureTask (java.util.concurrent) run:139, SingleLaneExecutorService$1 (hudson.remoting) run:28, ContextResettingExecutorService$1 (jenkins.util) run:68, ImpersonatingExecutorService$1 (jenkins.security) call:515, Executors$RunnableAdapter (java.util.concurrent) run:264, FutureTask (java.util.concurrent) runWorker:1128, ThreadPoolExecutor (java.util.concurrent) run:628, ThreadPoolExecutor$Worker (java.util.concurrent) run:829, Thread (java.lang)

          Basil Crow added a comment -

          Short-term solution

          jenkinsci/workflow-cps-plugin#543 has been released in 2705.v0449852ee36f and jenkinsci/script-security-plugin#420 has been released in 1175.v4b_d517d6db_f0, but upgrading to these releases is not sufficient: you also need jenkinsci/jenkins#6597 (planned for eventual release in weeklies and backport to LTS) or to start Jenkins with --add-opens java.desktop/com.sun.beans.introspect=ALL-UNNAMED in your Java options.

          jochenafuerbacher nsewardep phwolf shoelzle Can you please test the short-term solution and confirm that the issue has been resolved? Thanks!

          Long-term solution

          openjdk/jdk11u-dev#1106 and openjdk/jdk11u-dev#1103 have been proposed to backport the relevant upstream commits from OpenJDK 17 to OpenJDK 11.

          Basil Crow added a comment - Short-term solution jenkinsci/workflow-cps-plugin#543 has been released in 2705.v0449852ee36f and jenkinsci/script-security-plugin#420 has been released in 1175.v4b_d517d6db_f0 , but upgrading to these releases is not sufficient: you also need jenkinsci/jenkins#6597 (planned for eventual release in weeklies and backport to LTS) or to start Jenkins with --add-opens java.desktop/com.sun.beans.introspect=ALL-UNNAMED in your Java options. jochenafuerbacher nsewardep phwolf shoelzle Can you please test the short-term solution and confirm that the issue has been resolved? Thanks! Long-term solution openjdk/jdk11u-dev#1106 and openjdk/jdk11u-dev#1103 have been proposed to backport the relevant upstream commits from OpenJDK 17 to OpenJDK 11.

          Jason Gao added a comment -

          confirmed. did reproduce this issue on 2.332.3/jdk 11, but not on 2.346.1.
          however, i got another metaspace leak: JENKINS-69022

          Jason Gao added a comment - confirmed. did reproduce this issue on 2.332.3/jdk 11, but not on 2.346.1. however, i got another metaspace leak: JENKINS-69022

            basil Basil Crow
            shoelzle Stefan Hölzle
            Votes:
            5 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: