Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63766

Jenkins Pipelines Memory Leak

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Hello,

      If the following pipeline script is run as a pipeline job in Jenkins, The instantiated class is not collected by the garbage collector:

      class asdf {
        void run () {}
      }
      
      new asdf().run()
      

      Even after a garbage collection is run using jcmd, the instantiated class is still there:

      $ jcmd <PID> GC.run
      
      # wait for garbage collection to complete
      
      $ jcmd <PID> GC.class_stats | grep asdf
      11804    13         0        640          48     3344          13       470      2472     1904      5008      6912 asdf
      

      With one class or a few classes, this issue might not lead to any problems. But our pipelines have a lot of classes and we're encountering a steady increase in the MetaSpace and OutOfMemoryExceptions (CompressedClassSpace) after every 2 to 3 weeks. Additionally the Metaspace increases by several GBs per week to about 10 GB in 2 to 3 weeks.

       

      Some more information about the environment:

      • Operating System: Jenkins-Master runs in a debian-slim docker container
      • JRE/JDK vendors and versions: OpenJDK 11.0.6
      • Jenkins and plugin versions
        • Jenkins 2.235.3
        • Plugins
          ace-editor:1.1 - https://plugins.jenkins.io/ace-editor
          apache-httpcomponents-client-4-api:4.5.10-2.0 - https://plugins.jenkins.io/apache-httpcomponents-client-4-api
          authentication-tokens:1.4 - https://plugins.jenkins.io/authentication-tokens
          authorize-project:1.3.0 - https://plugins.jenkins.io/authorize-project
          blueocean-autofavorite:1.2.4 - https://plugins.jenkins.io/blueocean-autofavorite
          blueocean-bitbucket-pipeline:1.24.0 - https://plugins.jenkins.io/blueocean-bitbucket-pipeline
          blueocean-commons:1.24.0 - https://plugins.jenkins.io/blueocean-commons
          blueocean-config:1.24.0 - https://plugins.jenkins.io/blueocean-config
          blueocean-core-js:1.24.0 - https://plugins.jenkins.io/blueocean-core-js
          blueocean-dashboard:1.24.0 - https://plugins.jenkins.io/blueocean-dashboard
          blueocean-display-url:2.4.0 - https://plugins.jenkins.io/blueocean-display-url
          blueocean-events:1.24.0 - https://plugins.jenkins.io/blueocean-events
          blueocean-git-pipeline:1.24.0 - https://plugins.jenkins.io/blueocean-git-pipeline
          blueocean-github-pipeline:1.24.0 - https://plugins.jenkins.io/blueocean-github-pipeline
          blueocean-i18n:1.24.0 - https://plugins.jenkins.io/blueocean-i18n
          blueocean-jira:1.24.0 - https://plugins.jenkins.io/blueocean-jira
          blueocean-jwt:1.24.0 - https://plugins.jenkins.io/blueocean-jwt
          blueocean-personalization:1.24.0 - https://plugins.jenkins.io/blueocean-personalization
          blueocean-pipeline-api-impl:1.24.0 - https://plugins.jenkins.io/blueocean-pipeline-api-impl
          blueocean-pipeline-editor:1.24.0 - https://plugins.jenkins.io/blueocean-pipeline-editor
          blueocean-pipeline-scm-api:1.24.0 - https://plugins.jenkins.io/blueocean-pipeline-scm-api
          blueocean-rest-impl:1.24.0 - https://plugins.jenkins.io/blueocean-rest-impl
          blueocean-rest:1.24.0 - https://plugins.jenkins.io/blueocean-rest
          blueocean-web:1.24.0 - https://plugins.jenkins.io/blueocean-web
          blueocean:1.23.2 - https://plugins.jenkins.io/blueocean
          branch-api:2.6.0 - https://plugins.jenkins.io/branch-api
          cloudbees-bitbucket-branch-source:2.9.1 - https://plugins.jenkins.io/cloudbees-bitbucket-branch-source
          cloudbees-folder:6.14 - https://plugins.jenkins.io/cloudbees-folder
          conditional-buildstep:1.3.6 - https://plugins.jenkins.io/conditional-buildstep
          configuration-as-code:1.42 - https://plugins.jenkins.io/configuration-as-code
          credentials-binding:1.23 - https://plugins.jenkins.io/credentials-binding
          credentials:2.3.13 - https://plugins.jenkins.io/credentials
          display-url-api:2.3.3 - https://plugins.jenkins.io/display-url-api
          docker-commons:1.17 - https://plugins.jenkins.io/docker-commons
          docker-workflow:1.24 - https://plugins.jenkins.io/docker-workflow
          durable-task:1.35 - https://plugins.jenkins.io/durable-task
          echarts-api:4.8.0-2 - https://plugins.jenkins.io/echarts-api
          favorite:2.3.2 - https://plugins.jenkins.io/favorite
          git-client:3.5.0 - https://plugins.jenkins.io/git-client
          git-server:1.9 - https://plugins.jenkins.io/git-server
          git:4.4.3 - https://plugins.jenkins.io/git
          github-api:1.116 - https://plugins.jenkins.io/github-api
          github-branch-source:2.9.0 - https://plugins.jenkins.io/github-branch-source
          github:1.31.0 - https://plugins.jenkins.io/github
          handlebars:1.1.1 - https://plugins.jenkins.io/handlebars
          handy-uri-templates-2-api:2.1.8-1.0 - https://plugins.jenkins.io/handy-uri-templates-2-api
          htmlpublisher:1.23 - https://plugins.jenkins.io/htmlpublisher
          http_request:1.8.26 - https://plugins.jenkins.io/http_request
          jackson2-api:2.11.2 - https://plugins.jenkins.io/jackson2-api
          javadoc:1.6 - https://plugins.jenkins.io/javadoc
          jenkins-design-language:1.24.0 - https://plugins.jenkins.io/jenkins-design-language
          jira:3.1.1 - https://plugins.jenkins.io/jira
          job-dsl:1.77 - https://plugins.jenkins.io/job-dsl
          jobConfigHistory:2.26 - https://plugins.jenkins.io/jobConfigHistory
          jquery-detached:1.2.1 - https://plugins.jenkins.io/jquery-detached
          jquery3-api:3.5.1-1 - https://plugins.jenkins.io/jquery3-api
          jsch:0.1.55.2 - https://plugins.jenkins.io/jsch
          junit:1.35 - https://plugins.jenkins.io/junit
          lockable-resources:2.8 - https://plugins.jenkins.io/lockable-resources
          mailer:1.32 - https://plugins.jenkins.io/mailer
          matrix-auth:2.6.2 - https://plugins.jenkins.io/matrix-auth
          matrix-project:1.17 - https://plugins.jenkins.io/matrix-project
          maven-plugin:3.7 - https://plugins.jenkins.io/maven-plugin
          mercurial:2.10 - https://plugins.jenkins.io/mercurial
          metrics:4.0.2.6 - https://plugins.jenkins.io/metrics
          momentjs:1.1.1 - https://plugins.jenkins.io/momentjs
          okhttp-api:3.14.9 - https://plugins.jenkins.io/okhttp-api
          parameterized-trigger:2.37 - https://plugins.jenkins.io/parameterized-trigger
          pipeline-build-step:2.13 - https://plugins.jenkins.io/pipeline-build-step
          pipeline-graph-analysis:1.10 - https://plugins.jenkins.io/pipeline-graph-analysis
          pipeline-input-step:2.12 - https://plugins.jenkins.io/pipeline-input-step
          pipeline-milestone-step:1.3.1 - https://plugins.jenkins.io/pipeline-milestone-step
          pipeline-model-api:1.7.2 - https://plugins.jenkins.io/pipeline-model-api
          pipeline-model-definition:1.7.2 - https://plugins.jenkins.io/pipeline-model-definition
          pipeline-model-extensions:1.7.2 - https://plugins.jenkins.io/pipeline-model-extensions
          pipeline-rest-api:2.15 - https://plugins.jenkins.io/pipeline-rest-api
          pipeline-stage-step:2.5 - https://plugins.jenkins.io/pipeline-stage-step
          pipeline-stage-tags-metadata:1.7.2 - https://plugins.jenkins.io/pipeline-stage-tags-metadata
          pipeline-stage-view:2.15 - https://plugins.jenkins.io/pipeline-stage-view
          pipeline-utility-steps:2.6.1 - https://plugins.jenkins.io/pipeline-utility-steps
          plain-credentials:1.7 - https://plugins.jenkins.io/plain-credentials
          plugin-util-api:1.2.5 - https://plugins.jenkins.io/plugin-util-api
          pubsub-light:1.13 - https://plugins.jenkins.io/pubsub-light
          resource-disposer:0.14 - https://plugins.jenkins.io/resource-disposer
          run-condition:1.3 - https://plugins.jenkins.io/run-condition
          scm-api:2.6.3 - https://plugins.jenkins.io/scm-api
          script-security:1.74 - https://plugins.jenkins.io/script-security
          snakeyaml-api:1.27.0 - https://plugins.jenkins.io/snakeyaml-api
          sse-gateway:1.23 - https://plugins.jenkins.io/sse-gateway
          ssh-credentials:1.18.1 - https://plugins.jenkins.io/ssh-credentials
          ssh-steps:2.0.0 - https://plugins.jenkins.io/ssh-steps
          structs:1.20 - https://plugins.jenkins.io/structs
          swarm:3.22 - https://plugins.jenkins.io/swarm
          thinBackup:1.9 - https://plugins.jenkins.io/thinBackup
          token-macro:2.12 - https://plugins.jenkins.io/token-macro
          trilead-api:1.0.10 - https://plugins.jenkins.io/trilead-api
          variant:1.3 - https://plugins.jenkins.io/variant
          webhook-step:1.4 - https://plugins.jenkins.io/webhook-step
          workflow-aggregator:2.6 - https://plugins.jenkins.io/workflow-aggregator
          workflow-api:2.40 - https://plugins.jenkins.io/workflow-api
          workflow-basic-steps:2.21 - https://plugins.jenkins.io/workflow-basic-steps
          workflow-cps-global-lib:2.17 - https://plugins.jenkins.io/workflow-cps-global-lib
          workflow-cps:2.83 - https://plugins.jenkins.io/workflow-cps
          workflow-durable-task-step:2.35 - https://plugins.jenkins.io/workflow-durable-task-step
          workflow-job:2.40 - https://plugins.jenkins.io/workflow-job
          workflow-multibranch:2.22 - https://plugins.jenkins.io/workflow-multibranch
          workflow-scm-step:2.11 - https://plugins.jenkins.io/workflow-scm-step
          workflow-step-api:2.22 - https://plugins.jenkins.io/workflow-step-api
          workflow-support:3.5 - https://plugins.jenkins.io/workflow-support
          ws-cleanup:0.38 - https://plugins.jenkins.io/ws-cleanup
          bouncycastle-api:2.16.0 - https://plugins.jenkins.io/bouncycastle-api
          command-launcher:1.2 - https://plugins.jenkins.io/command-launcher
          jdk-tool:1.0 - https://plugins.jenkins.io/jdk-tool
          jaxb:2.3.0 - https://plugins.jenkins.io/jaxb
          configuration-as-code-groovy:1.1 - https://plugins.jenkins.io/configuration-as-code-groovy
          bootstrap4-api:4.5.2-1 - https://plugins.jenkins.io/bootstrap4-api
          font-awesome-api:5.14.0-1 - https://plugins.jenkins.io/font-awesome-api
          popper-api:1.16.0-6 - https://plugins.jenkins.io/popper-api
          

        Attachments

          Activity

          Hide
          shoelzle Stefan Hölzle added a comment - - edited

          With Jenkins 2.204.6, Oracle JDK 8 and the workflow-cps-plugin version 2.83 the problem does not occur.

          Show
          shoelzle Stefan Hölzle added a comment - - edited With Jenkins 2.204.6, Oracle JDK 8 and the workflow-cps-plugin version 2.83 the problem does not occur.
          Hide
          jochenafuerbacher Jochen A. Fürbacher added a comment -

          I tested this with both official docker images: jenkins/jenkins:jdk11 and jenkins/jenkins:2.259-slim (jdk 8).

          While the problem of Stefan Hölzle also occures on jenkins:jdk11 (with nothing else installed than workflow-job and workflow-cps), it does not occure on the jdk 8 image (jenkins:2.259-slim).

          Show
          jochenafuerbacher Jochen A. Fürbacher added a comment - I tested this with both official docker images: jenkins/jenkins:jdk11 and jenkins/jenkins:2.259-slim (jdk 8). While the problem of Stefan Hölzle also occures on jenkins:jdk11 (with nothing else installed than workflow-job and workflow-cps ), it does not occure on the jdk 8 image ( jenkins:2.259-slim ).
          Hide
          jochenafuerbacher Jochen A. Fürbacher added a comment -

          Same behavior with AdoptOpenJDK13 and Oracle JDK 11. Also changing the GC via -XX:+UseG1GC or -XX:+UseParallelGC didn't solve the problem.

          Show
          jochenafuerbacher Jochen A. Fürbacher added a comment - Same behavior with AdoptOpenJDK13 and Oracle JDK 11. Also changing the GC via -XX:+UseG1GC or -XX:+UseParallelGC didn't solve the problem.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            shoelzle Stefan Hölzle
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated: