Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63766

Metaspace memory leak in Pipeline when running on Java 11 (but not 8 or 17)

XMLWordPrintable

    • 2.350, 2.346.1

      Hello,

      If the following pipeline script is run as a pipeline job in Jenkins, The instantiated class is not collected by the garbage collector:

      class asdf {
        void run () {}
      }
      
      new asdf().run()
      

      Even after a garbage collection is run using jcmd, the instantiated class is still there:

      $ jcmd <PID> GC.run
      
      # wait for garbage collection to complete
      
      $ jcmd <PID> GC.class_stats | grep asdf
      11804    13         0        640          48     3344          13       470      2472     1904      5008      6912 asdf
      

      With one class or a few classes, this issue might not lead to any problems. But our pipelines have a lot of classes and we're encountering a steady increase in the MetaSpace and OutOfMemoryExceptions (CompressedClassSpace) after every 2 to 3 weeks. Additionally the Metaspace increases by several GBs per week to about 10 GB in 2 to 3 weeks.

       

      Some more information about the environment:

      • Operating System: Jenkins-Master runs in a debian-slim docker container
      • JRE/JDK vendors and versions: OpenJDK 11.0.6
      • Jenkins and plugin versions
        • Jenkins 2.235.3
        • Plugins
          ace-editor:1.1 - https://plugins.jenkins.io/ace-editor
          apache-httpcomponents-client-4-api:4.5.10-2.0 - https://plugins.jenkins.io/apache-httpcomponents-client-4-api
          authentication-tokens:1.4 - https://plugins.jenkins.io/authentication-tokens
          authorize-project:1.3.0 - https://plugins.jenkins.io/authorize-project
          blueocean-autofavorite:1.2.4 - https://plugins.jenkins.io/blueocean-autofavorite
          blueocean-bitbucket-pipeline:1.24.0 - https://plugins.jenkins.io/blueocean-bitbucket-pipeline
          blueocean-commons:1.24.0 - https://plugins.jenkins.io/blueocean-commons
          blueocean-config:1.24.0 - https://plugins.jenkins.io/blueocean-config
          blueocean-core-js:1.24.0 - https://plugins.jenkins.io/blueocean-core-js
          blueocean-dashboard:1.24.0 - https://plugins.jenkins.io/blueocean-dashboard
          blueocean-display-url:2.4.0 - https://plugins.jenkins.io/blueocean-display-url
          blueocean-events:1.24.0 - https://plugins.jenkins.io/blueocean-events
          blueocean-git-pipeline:1.24.0 - https://plugins.jenkins.io/blueocean-git-pipeline
          blueocean-github-pipeline:1.24.0 - https://plugins.jenkins.io/blueocean-github-pipeline
          blueocean-i18n:1.24.0 - https://plugins.jenkins.io/blueocean-i18n
          blueocean-jira:1.24.0 - https://plugins.jenkins.io/blueocean-jira
          blueocean-jwt:1.24.0 - https://plugins.jenkins.io/blueocean-jwt
          blueocean-personalization:1.24.0 - https://plugins.jenkins.io/blueocean-personalization
          blueocean-pipeline-api-impl:1.24.0 - https://plugins.jenkins.io/blueocean-pipeline-api-impl
          blueocean-pipeline-editor:1.24.0 - https://plugins.jenkins.io/blueocean-pipeline-editor
          blueocean-pipeline-scm-api:1.24.0 - https://plugins.jenkins.io/blueocean-pipeline-scm-api
          blueocean-rest-impl:1.24.0 - https://plugins.jenkins.io/blueocean-rest-impl
          blueocean-rest:1.24.0 - https://plugins.jenkins.io/blueocean-rest
          blueocean-web:1.24.0 - https://plugins.jenkins.io/blueocean-web
          blueocean:1.23.2 - https://plugins.jenkins.io/blueocean
          branch-api:2.6.0 - https://plugins.jenkins.io/branch-api
          cloudbees-bitbucket-branch-source:2.9.1 - https://plugins.jenkins.io/cloudbees-bitbucket-branch-source
          cloudbees-folder:6.14 - https://plugins.jenkins.io/cloudbees-folder
          conditional-buildstep:1.3.6 - https://plugins.jenkins.io/conditional-buildstep
          configuration-as-code:1.42 - https://plugins.jenkins.io/configuration-as-code
          credentials-binding:1.23 - https://plugins.jenkins.io/credentials-binding
          credentials:2.3.13 - https://plugins.jenkins.io/credentials
          display-url-api:2.3.3 - https://plugins.jenkins.io/display-url-api
          docker-commons:1.17 - https://plugins.jenkins.io/docker-commons
          docker-workflow:1.24 - https://plugins.jenkins.io/docker-workflow
          durable-task:1.35 - https://plugins.jenkins.io/durable-task
          echarts-api:4.8.0-2 - https://plugins.jenkins.io/echarts-api
          favorite:2.3.2 - https://plugins.jenkins.io/favorite
          git-client:3.5.0 - https://plugins.jenkins.io/git-client
          git-server:1.9 - https://plugins.jenkins.io/git-server
          git:4.4.3 - https://plugins.jenkins.io/git
          github-api:1.116 - https://plugins.jenkins.io/github-api
          github-branch-source:2.9.0 - https://plugins.jenkins.io/github-branch-source
          github:1.31.0 - https://plugins.jenkins.io/github
          handlebars:1.1.1 - https://plugins.jenkins.io/handlebars
          handy-uri-templates-2-api:2.1.8-1.0 - https://plugins.jenkins.io/handy-uri-templates-2-api
          htmlpublisher:1.23 - https://plugins.jenkins.io/htmlpublisher
          http_request:1.8.26 - https://plugins.jenkins.io/http_request
          jackson2-api:2.11.2 - https://plugins.jenkins.io/jackson2-api
          javadoc:1.6 - https://plugins.jenkins.io/javadoc
          jenkins-design-language:1.24.0 - https://plugins.jenkins.io/jenkins-design-language
          jira:3.1.1 - https://plugins.jenkins.io/jira
          job-dsl:1.77 - https://plugins.jenkins.io/job-dsl
          jobConfigHistory:2.26 - https://plugins.jenkins.io/jobConfigHistory
          jquery-detached:1.2.1 - https://plugins.jenkins.io/jquery-detached
          jquery3-api:3.5.1-1 - https://plugins.jenkins.io/jquery3-api
          jsch:0.1.55.2 - https://plugins.jenkins.io/jsch
          junit:1.35 - https://plugins.jenkins.io/junit
          lockable-resources:2.8 - https://plugins.jenkins.io/lockable-resources
          mailer:1.32 - https://plugins.jenkins.io/mailer
          matrix-auth:2.6.2 - https://plugins.jenkins.io/matrix-auth
          matrix-project:1.17 - https://plugins.jenkins.io/matrix-project
          maven-plugin:3.7 - https://plugins.jenkins.io/maven-plugin
          mercurial:2.10 - https://plugins.jenkins.io/mercurial
          metrics:4.0.2.6 - https://plugins.jenkins.io/metrics
          momentjs:1.1.1 - https://plugins.jenkins.io/momentjs
          okhttp-api:3.14.9 - https://plugins.jenkins.io/okhttp-api
          parameterized-trigger:2.37 - https://plugins.jenkins.io/parameterized-trigger
          pipeline-build-step:2.13 - https://plugins.jenkins.io/pipeline-build-step
          pipeline-graph-analysis:1.10 - https://plugins.jenkins.io/pipeline-graph-analysis
          pipeline-input-step:2.12 - https://plugins.jenkins.io/pipeline-input-step
          pipeline-milestone-step:1.3.1 - https://plugins.jenkins.io/pipeline-milestone-step
          pipeline-model-api:1.7.2 - https://plugins.jenkins.io/pipeline-model-api
          pipeline-model-definition:1.7.2 - https://plugins.jenkins.io/pipeline-model-definition
          pipeline-model-extensions:1.7.2 - https://plugins.jenkins.io/pipeline-model-extensions
          pipeline-rest-api:2.15 - https://plugins.jenkins.io/pipeline-rest-api
          pipeline-stage-step:2.5 - https://plugins.jenkins.io/pipeline-stage-step
          pipeline-stage-tags-metadata:1.7.2 - https://plugins.jenkins.io/pipeline-stage-tags-metadata
          pipeline-stage-view:2.15 - https://plugins.jenkins.io/pipeline-stage-view
          pipeline-utility-steps:2.6.1 - https://plugins.jenkins.io/pipeline-utility-steps
          plain-credentials:1.7 - https://plugins.jenkins.io/plain-credentials
          plugin-util-api:1.2.5 - https://plugins.jenkins.io/plugin-util-api
          pubsub-light:1.13 - https://plugins.jenkins.io/pubsub-light
          resource-disposer:0.14 - https://plugins.jenkins.io/resource-disposer
          run-condition:1.3 - https://plugins.jenkins.io/run-condition
          scm-api:2.6.3 - https://plugins.jenkins.io/scm-api
          script-security:1.74 - https://plugins.jenkins.io/script-security
          snakeyaml-api:1.27.0 - https://plugins.jenkins.io/snakeyaml-api
          sse-gateway:1.23 - https://plugins.jenkins.io/sse-gateway
          ssh-credentials:1.18.1 - https://plugins.jenkins.io/ssh-credentials
          ssh-steps:2.0.0 - https://plugins.jenkins.io/ssh-steps
          structs:1.20 - https://plugins.jenkins.io/structs
          swarm:3.22 - https://plugins.jenkins.io/swarm
          thinBackup:1.9 - https://plugins.jenkins.io/thinBackup
          token-macro:2.12 - https://plugins.jenkins.io/token-macro
          trilead-api:1.0.10 - https://plugins.jenkins.io/trilead-api
          variant:1.3 - https://plugins.jenkins.io/variant
          webhook-step:1.4 - https://plugins.jenkins.io/webhook-step
          workflow-aggregator:2.6 - https://plugins.jenkins.io/workflow-aggregator
          workflow-api:2.40 - https://plugins.jenkins.io/workflow-api
          workflow-basic-steps:2.21 - https://plugins.jenkins.io/workflow-basic-steps
          workflow-cps-global-lib:2.17 - https://plugins.jenkins.io/workflow-cps-global-lib
          workflow-cps:2.83 - https://plugins.jenkins.io/workflow-cps
          workflow-durable-task-step:2.35 - https://plugins.jenkins.io/workflow-durable-task-step
          workflow-job:2.40 - https://plugins.jenkins.io/workflow-job
          workflow-multibranch:2.22 - https://plugins.jenkins.io/workflow-multibranch
          workflow-scm-step:2.11 - https://plugins.jenkins.io/workflow-scm-step
          workflow-step-api:2.22 - https://plugins.jenkins.io/workflow-step-api
          workflow-support:3.5 - https://plugins.jenkins.io/workflow-support
          ws-cleanup:0.38 - https://plugins.jenkins.io/ws-cleanup
          bouncycastle-api:2.16.0 - https://plugins.jenkins.io/bouncycastle-api
          command-launcher:1.2 - https://plugins.jenkins.io/command-launcher
          jdk-tool:1.0 - https://plugins.jenkins.io/jdk-tool
          jaxb:2.3.0 - https://plugins.jenkins.io/jaxb
          configuration-as-code-groovy:1.1 - https://plugins.jenkins.io/configuration-as-code-groovy
          bootstrap4-api:4.5.2-1 - https://plugins.jenkins.io/bootstrap4-api
          font-awesome-api:5.14.0-1 - https://plugins.jenkins.io/font-awesome-api
          popper-api:1.16.0-6 - https://plugins.jenkins.io/popper-api
          

            basil Basil Crow
            shoelzle Stefan Hölzle
            Votes:
            5 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: