Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-72067

File system SCM uses more memory than expected with Jenkins 2.405 - 2.425

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • core, filesystem_scm-plugin
    • 2.426, 2.414.3

      See my comments below, this issue started in 2.405

      (Same behavior happens whether running on same node as the master or not)

      I have a test job that triggers itself (also I set quiet period to 0), the job uses SCM to run the example Hello World pipeline

      After a few minutes, this is the result of http://localhost:8080/monitoring?part=heaphisto

      If I downgrade to 2.404.3 and try the same thing again....

        1. 2-405-after.png
          2-405-after.png
          208 kB
        2. 2-405-before.png
          2-405-before.png
          224 kB
        3. 2-404-after.png
          2-404-after.png
          208 kB
        4. 2-404-before.png
          2-404-before.png
          228 kB
        5. a-JENKINS-72067-git.tgz
          4 kB
        6. a-run-jenkins.sh
          0.8 kB
        7. a-plugins.txt
          2 kB
        8. 02-after-800-test-runs.png
          02-after-800-test-runs.png
          64 kB
        9. 01-before-test-run.png
          01-before-test-run.png
          77 kB
        10. JENKINS-72067-git.tgz
          2 kB
        11. run-jenkins.sh
          0.8 kB
        12. plugins.txt
          2 kB
        13. image-2023-09-25-08-42-47-692.png
          image-2023-09-25-08-42-47-692.png
          73 kB
        14. image-2023-09-24-11-28-08-440.png
          image-2023-09-24-11-28-08-440.png
          105 kB
        15. image-2023-09-24-10-46-06-168.png
          image-2023-09-24-10-46-06-168.png
          99 kB

          [JENKINS-72067] File system SCM uses more memory than expected with Jenkins 2.405 - 2.425

          Alex added a comment -

          Same issue happens in Jenkins LTS 2.414.1

           

          To be clear, Jenkins LTS 2.401.3 doesn't have this issue

          Alex added a comment - Same issue happens in Jenkins LTS 2.414.1   To be clear, Jenkins LTS 2.401.3 doesn't have this issue

          Alex added a comment - - edited

          Install Jenkins 2.401.3 with plugins: Monitoring, Pipeline, and File System SCM

          Jenkins: 2.401.3
          OS: Linux - 5.15.130-20444-g44e580fbe72b
          Java: 11.0.19 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
          —
          apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
          bouncycastle-api:2.29
          branch-api:2.1128.v717130d4f816
          caffeine-api:3.1.8-133.v17b_1ff2e0599
          cloudbees-folder:6.848.ve3b_fd7839a_81
          command-launcher:107.v773860566e2e
          credentials:1271.v54b_1c2c6388a_
          credentials-binding:636.v55f1275c7b_27
          display-url-api:2.3.9
          durable-task:523.va_a_22cf15d5e0
          filesystem_scm:2.1
          git-client:4.5.0
          instance-identity:173.va_37c494ec4e5
          ionicons-api:56.v1b_1c8c49374e
          jackson2-api:2.15.2-350.v0c2f3f8fc595
          jakarta-activation-api:2.0.1-3
          jakarta-mail-api:2.0.1-3
          javax-activation-api:1.2.0-6
          javax-mail-api:1.6.2-9
          jaxb:2.3.8-1
          jdk-tool:73.vddf737284550
          mailer:463.vedf8358e006b_
          mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_
          mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_
          monitoring:1.95.0
          pipeline-build-step:505.v5f0844d8d126
          pipeline-groovy-lib:689.veec561a_dee13
          pipeline-input-step:477.v339683a_8d55e
          pipeline-milestone-step:111.v449306f708b_7
          pipeline-model-api:2.2144.v077a_d1928a_40
          pipeline-model-definition:2.2144.v077a_d1928a_40
          pipeline-model-extensions:2.2144.v077a_d1928a_40
          pipeline-stage-step:305.ve96d0205c1c6
          pipeline-stage-tags-metadata:2.2144.v077a_d1928a_40
          plain-credentials:143.v1b_df8b_d3b_e48
          scm-api:676.v886669a_199a_a_
          script-security:1275.v23895f409fb_d
          snakeyaml-api:2.2-111.vc6598e30cc65
          ssh-credentials:308.ve4497b_ccd8f4
          sshd:3.312.v1c601b_c83b_0e
          structs:325.vcb_307d2a_2782
          trilead-api:2.84.v72119de229b_7
          variant:60.v7290fc0eb_b_cd
          workflow-aggregator:596.v8c21c963d92d
          workflow-api:1281.vca_5fddb_3fceb_
          workflow-basic-steps:1042.ve7b_140c4a_e0c
          workflow-cps:3791.va_c0338ea_b_59c
          workflow-durable-task-step:1289.v4d3e7b_01546b_
          workflow-job:1326.ve643e00e9220
          workflow-multibranch:756.v891d88f2cd46
          workflow-scm-step:415.v434365564324
          workflow-step-api:639.v6eca_cd8c04a_a_
          workflow-support:865.v43e78cc44e0d

          Alex added a comment - - edited Install Jenkins 2.401.3 with plugins: Monitoring , Pipeline , and File System SCM Jenkins: 2.401.3 OS: Linux - 5.15.130-20444-g44e580fbe72b Java: 11.0.19 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) — apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 bouncycastle-api:2.29 branch-api:2.1128.v717130d4f816 caffeine-api:3.1.8-133.v17b_1ff2e0599 cloudbees-folder:6.848.ve3b_fd7839a_81 command-launcher:107.v773860566e2e credentials:1271.v54b_1c2c6388a_ credentials-binding:636.v55f1275c7b_27 display-url-api:2.3.9 durable-task:523.va_a_22cf15d5e0 filesystem_scm:2.1 git-client:4.5.0 instance-identity:173.va_37c494ec4e5 ionicons-api:56.v1b_1c8c49374e jackson2-api:2.15.2-350.v0c2f3f8fc595 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.8-1 jdk-tool:73.vddf737284550 mailer:463.vedf8358e006b_ mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_ mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_ monitoring:1.95.0 pipeline-build-step:505.v5f0844d8d126 pipeline-groovy-lib:689.veec561a_dee13 pipeline-input-step:477.v339683a_8d55e pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2144.v077a_d1928a_40 pipeline-model-definition:2.2144.v077a_d1928a_40 pipeline-model-extensions:2.2144.v077a_d1928a_40 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2144.v077a_d1928a_40 plain-credentials:143.v1b_df8b_d3b_e48 scm-api:676.v886669a_199a_a_ script-security:1275.v23895f409fb_d snakeyaml-api:2.2-111.vc6598e30cc65 ssh-credentials:308.ve4497b_ccd8f4 sshd:3.312.v1c601b_c83b_0e structs:325.vcb_307d2a_2782 trilead-api:2.84.v72119de229b_7 variant:60.v7290fc0eb_b_cd workflow-aggregator:596.v8c21c963d92d workflow-api:1281.vca_5fddb_3fceb_ workflow-basic-steps:1042.ve7b_140c4a_e0c workflow-cps:3791.va_c0338ea_b_59c workflow-durable-task-step:1289.v4d3e7b_01546b_ workflow-job:1326.ve643e00e9220 workflow-multibranch:756.v891d88f2cd46 workflow-scm-step:415.v434365564324 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:865.v43e78cc44e0d

          Alex added a comment - - edited

          Go to http://localhost:8080/manage/configure

           
          Global Pipeline Libraries
          - Click Add

          • Name: myLibrary
          • Default version: master
          • Make sure Retrieval method == Legacy SCM
          • Make sure Source Code Management == File System
          • Under Legacy SCM, under File System - Path: /var/lib/jenkins/myLibrary
          • Click Save button at bottom of page

           

           

          ----------------------------

          /var/lib/jenkins/myLibrary/vars/script.groovy file:

          def call() {
          
          pipeline {
              agent none
              stages {
                  stage('Hello') {
                      steps {
                          echo 'Hello World'
                      }
                  }
              }
          }
          
          }

           

           

           

          Alex added a comment - - edited Go to http://localhost:8080/manage/configure   Global Pipeline Libraries -  Click Add Name: myLibrary Default version: master Make sure Retrieval method == Legacy SCM Make sure Source Code Management == File System Under Legacy SCM, under File System - Path: /var/lib/jenkins/myLibrary Click Save button at bottom of page     ---------------------------- /var/lib/jenkins/myLibrary/vars/script.groovy file: def call() { pipeline {     agent none stages {         stage( 'Hello' ) {             steps {                 echo 'Hello World'             }         }     } } }      

          Alex added a comment -

          Make a new job with name test

          • Click Pipeline
          • Click OK

          Under Build Triggers

          • Click on Build after other projects are built
          • Put test
          • Click on Quiet Period
          • Replace it with 0

          For Pipeline Script:

          @Library('myLibrary') _
          
          script()

          Alex added a comment - Make a new job with name test Click Pipeline Click OK Under Build Triggers Click on Build after other projects are built Put test Click on Quiet Period Replace it with 0 For Pipeline Script: @Library( 'myLibrary' ) _ script()

          Alex added a comment -

          Click Build Now

           

          Go to http://localhost:8080/monitoring?part=heaphisto after a few minutes

           

          Notice that java.util.HashMap$Node is not growing much

          Alex added a comment - Click Build Now   Go to http://localhost:8080/monitoring?part=heaphisto after a few minutes   Notice that java.util.HashMap$Node is not growing much

          Alex added a comment -

          Update to Jenkins 2.414.2

           

          Start the test job

           

          Go to http://localhost:8080/monitoring?part=heaphisto after a few minutes

           
          Now, java.util.HashMap$Node is growing a lot!

          Alex added a comment - Update to Jenkins 2.414.2   Start the test job   Go to  http://localhost:8080/monitoring?part=heaphisto  after a few minutes   Now, java.util.HashMap$Node is growing a lot!

          Alex added a comment - - edited

          Upon further investigation, this issue started happening in Jenkins 2.405

           

          Jenkins 2.404 info

          Jenkins: 2.404
          OS: Linux - 5.15.130-20444-g44e580fbe72b
          Java: 11.0.19 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
          ---
          apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
          bouncycastle-api:2.29
          branch-api:2.1128.v717130d4f816
          caffeine-api:3.1.8-133.v17b_1ff2e0599
          cloudbees-folder:6.848.ve3b_fd7839a_81
          command-launcher:107.v773860566e2e
          credentials:1271.v54b_1c2c6388a_
          credentials-binding:636.v55f1275c7b_27
          display-url-api:2.3.9
          durable-task:523.va_a_22cf15d5e0
          filesystem_scm:2.1
          git-client:4.5.0
          instance-identity:173.va_37c494ec4e5
          ionicons-api:56.v1b_1c8c49374e
          jackson2-api:2.15.2-350.v0c2f3f8fc595
          jakarta-activation-api:2.0.1-3
          jakarta-mail-api:2.0.1-3
          javax-activation-api:1.2.0-6
          javax-mail-api:1.6.2-9
          jaxb:2.3.8-1
          jdk-tool:73.vddf737284550
          mailer:463.vedf8358e006b_
          mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_
          mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_
          monitoring:1.95.0
          pipeline-build-step:505.v5f0844d8d126
          pipeline-groovy-lib:689.veec561a_dee13
          pipeline-input-step:477.v339683a_8d55e
          pipeline-milestone-step:111.v449306f708b_7
          pipeline-model-api:2.2144.v077a_d1928a_40
          pipeline-model-definition:2.2144.v077a_d1928a_40
          pipeline-model-extensions:2.2144.v077a_d1928a_40
          pipeline-stage-step:305.ve96d0205c1c6
          pipeline-stage-tags-metadata:2.2144.v077a_d1928a_40
          plain-credentials:143.v1b_df8b_d3b_e48
          scm-api:676.v886669a_199a_a_
          script-security:1275.v23895f409fb_d
          snakeyaml-api:2.2-111.vc6598e30cc65
          ssh-credentials:308.ve4497b_ccd8f4
          sshd:3.312.v1c601b_c83b_0e
          structs:325.vcb_307d2a_2782
          trilead-api:2.84.v72119de229b_7
          variant:60.v7290fc0eb_b_cd
          workflow-aggregator:596.v8c21c963d92d
          workflow-api:1281.vca_5fddb_3fceb_
          workflow-basic-steps:1042.ve7b_140c4a_e0c
          workflow-cps:3791.va_c0338ea_b_59c
          workflow-durable-task-step:1289.v4d3e7b_01546b_
          workflow-job:1326.ve643e00e9220
          workflow-multibranch:756.v891d88f2cd46
          workflow-scm-step:415.v434365564324
          workflow-step-api:639.v6eca_cd8c04a_a_
          workflow-support:865.v43e78cc44e0d

          Jenkins 2.405 info

          Jenkins: 2.405
          OS: Linux - 5.15.130-20444-g44e580fbe72b
          Java: 11.0.19 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
          ---
          apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
          bouncycastle-api:2.29
          branch-api:2.1128.v717130d4f816
          caffeine-api:3.1.8-133.v17b_1ff2e0599
          cloudbees-folder:6.848.ve3b_fd7839a_81
          command-launcher:107.v773860566e2e
          credentials:1271.v54b_1c2c6388a_
          credentials-binding:636.v55f1275c7b_27
          display-url-api:2.3.9
          durable-task:523.va_a_22cf15d5e0
          filesystem_scm:2.1
          git-client:4.5.0
          instance-identity:173.va_37c494ec4e5
          ionicons-api:56.v1b_1c8c49374e
          jackson2-api:2.15.2-350.v0c2f3f8fc595
          jakarta-activation-api:2.0.1-3
          jakarta-mail-api:2.0.1-3
          javax-activation-api:1.2.0-6
          javax-mail-api:1.6.2-9
          jaxb:2.3.8-1
          jdk-tool:73.vddf737284550
          mailer:463.vedf8358e006b_
          mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_
          mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_
          monitoring:1.95.0
          pipeline-build-step:505.v5f0844d8d126
          pipeline-groovy-lib:689.veec561a_dee13
          pipeline-input-step:477.v339683a_8d55e
          pipeline-milestone-step:111.v449306f708b_7
          pipeline-model-api:2.2144.v077a_d1928a_40
          pipeline-model-definition:2.2144.v077a_d1928a_40
          pipeline-model-extensions:2.2144.v077a_d1928a_40
          pipeline-stage-step:305.ve96d0205c1c6
          pipeline-stage-tags-metadata:2.2144.v077a_d1928a_40
          plain-credentials:143.v1b_df8b_d3b_e48
          scm-api:676.v886669a_199a_a_
          script-security:1275.v23895f409fb_d
          snakeyaml-api:2.2-111.vc6598e30cc65
          ssh-credentials:308.ve4497b_ccd8f4
          sshd:3.312.v1c601b_c83b_0e
          structs:325.vcb_307d2a_2782
          trilead-api:2.84.v72119de229b_7
          variant:60.v7290fc0eb_b_cd
          workflow-aggregator:596.v8c21c963d92d
          workflow-api:1281.vca_5fddb_3fceb_
          workflow-basic-steps:1042.ve7b_140c4a_e0c
          workflow-cps:3791.va_c0338ea_b_59c
          workflow-durable-task-step:1289.v4d3e7b_01546b_
          workflow-job:1326.ve643e00e9220
          workflow-multibranch:756.v891d88f2cd46
          workflow-scm-step:415.v434365564324
          workflow-step-api:639.v6eca_cd8c04a_a_
          workflow-support:865.v43e78cc44e0d

           

          Alex added a comment - - edited Upon further investigation, this issue started happening in Jenkins 2.405   Jenkins 2.404 info Jenkins: 2.404 OS: Linux - 5.15.130-20444-g44e580fbe72b Java: 11.0.19 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 bouncycastle-api:2.29 branch-api:2.1128.v717130d4f816 caffeine-api:3.1.8-133.v17b_1ff2e0599 cloudbees-folder:6.848.ve3b_fd7839a_81 command-launcher:107.v773860566e2e credentials:1271.v54b_1c2c6388a_ credentials-binding:636.v55f1275c7b_27 display-url-api:2.3.9 durable-task:523.va_a_22cf15d5e0 filesystem_scm:2.1 git-client:4.5.0 instance-identity:173.va_37c494ec4e5 ionicons-api:56.v1b_1c8c49374e jackson2-api:2.15.2-350.v0c2f3f8fc595 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.8-1 jdk-tool:73.vddf737284550 mailer:463.vedf8358e006b_ mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_ mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_ monitoring:1.95.0 pipeline-build-step:505.v5f0844d8d126 pipeline-groovy-lib:689.veec561a_dee13 pipeline-input-step:477.v339683a_8d55e pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2144.v077a_d1928a_40 pipeline-model-definition:2.2144.v077a_d1928a_40 pipeline-model-extensions:2.2144.v077a_d1928a_40 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2144.v077a_d1928a_40 plain-credentials:143.v1b_df8b_d3b_e48 scm-api:676.v886669a_199a_a_ script-security:1275.v23895f409fb_d snakeyaml-api:2.2-111.vc6598e30cc65 ssh-credentials:308.ve4497b_ccd8f4 sshd:3.312.v1c601b_c83b_0e structs:325.vcb_307d2a_2782 trilead-api:2.84.v72119de229b_7 variant:60.v7290fc0eb_b_cd workflow-aggregator:596.v8c21c963d92d workflow-api:1281.vca_5fddb_3fceb_ workflow-basic-steps:1042.ve7b_140c4a_e0c workflow-cps:3791.va_c0338ea_b_59c workflow-durable-task-step:1289.v4d3e7b_01546b_ workflow-job:1326.ve643e00e9220 workflow-multibranch:756.v891d88f2cd46 workflow-scm-step:415.v434365564324 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:865.v43e78cc44e0d Jenkins 2.405 info Jenkins: 2.405 OS: Linux - 5.15.130-20444-g44e580fbe72b Java: 11.0.19 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 bouncycastle-api:2.29 branch-api:2.1128.v717130d4f816 caffeine-api:3.1.8-133.v17b_1ff2e0599 cloudbees-folder:6.848.ve3b_fd7839a_81 command-launcher:107.v773860566e2e credentials:1271.v54b_1c2c6388a_ credentials-binding:636.v55f1275c7b_27 display-url-api:2.3.9 durable-task:523.va_a_22cf15d5e0 filesystem_scm:2.1 git-client:4.5.0 instance-identity:173.va_37c494ec4e5 ionicons-api:56.v1b_1c8c49374e jackson2-api:2.15.2-350.v0c2f3f8fc595 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.8-1 jdk-tool:73.vddf737284550 mailer:463.vedf8358e006b_ mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_ mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_ monitoring:1.95.0 pipeline-build-step:505.v5f0844d8d126 pipeline-groovy-lib:689.veec561a_dee13 pipeline-input-step:477.v339683a_8d55e pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2144.v077a_d1928a_40 pipeline-model-definition:2.2144.v077a_d1928a_40 pipeline-model-extensions:2.2144.v077a_d1928a_40 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2144.v077a_d1928a_40 plain-credentials:143.v1b_df8b_d3b_e48 scm-api:676.v886669a_199a_a_ script-security:1275.v23895f409fb_d snakeyaml-api:2.2-111.vc6598e30cc65 ssh-credentials:308.ve4497b_ccd8f4 sshd:3.312.v1c601b_c83b_0e structs:325.vcb_307d2a_2782 trilead-api:2.84.v72119de229b_7 variant:60.v7290fc0eb_b_cd workflow-aggregator:596.v8c21c963d92d workflow-api:1281.vca_5fddb_3fceb_ workflow-basic-steps:1042.ve7b_140c4a_e0c workflow-cps:3791.va_c0338ea_b_59c workflow-durable-task-step:1289.v4d3e7b_01546b_ workflow-job:1326.ve643e00e9220 workflow-multibranch:756.v891d88f2cd46 workflow-scm-step:415.v434365564324 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:865.v43e78cc44e0d  

          Mark Waite added a comment -

          Thanks for the great details on the bug report. Very much appreciated.

          I'm unable to use the Pipeline shared library as described in the "Manage Jenkins" step to create the library. It complained that it was expecting an "src" directory so I moved script.groovy into a directory named "src". Unfortunately, that gave a new error message, "script step must be called with a body". Any hints on the mistake that I must be making?

          I'm running Jenkins 2.405 with the list of plugins you provided for 2.405. It starts correctly and runs with that set of plugins.

          Mark Waite added a comment - Thanks for the great details on the bug report. Very much appreciated. I'm unable to use the Pipeline shared library as described in the "Manage Jenkins" step to create the library. It complained that it was expecting an "src" directory so I moved script.groovy into a directory named "src". Unfortunately, that gave a new error message, "script step must be called with a body". Any hints on the mistake that I must be making? I'm running Jenkins 2.405 with the list of plugins you provided for 2.405. It starts correctly and runs with that set of plugins.

          Alex added a comment -

          markewaite sorry for the confusion

          I have it set up the following way....

          Alex added a comment - markewaite sorry for the confusion I have it set up the following way....

          Alex added a comment -

          markewaite Please let me know if you can reproduce the issue

          Alex added a comment - markewaite Please let me know if you can reproduce the issue

          Mark Waite added a comment -

          I'm able to reproduce the issue with Jenkins 2.405 using the following steps:

          1. Create a plugins.txt with the list of plugins you provided
          2. Create a run-jenkins.sh script that downloads Jenkins 2.405, installs the specified plugins, and starts Jenkins
          3. Create a job definition and a Pipeline library definition as described in JENKINS-72067-git.tgz
          4. Run Jenkins by executing the run-jenkins.sh script
          5. Complete the setup wizard by creating a new user and choosing to install no additional plugins (since they have already been downloaded)
          6. Run the test job once, then capture the Java Melody memory histogram (see "Before test")
          7. Modify the test job to run itself again as described in the earlier comment from alexsch01
          8. Run the test job and watch it run many times. After about 800 times, reconfigure the job to remove the "run itself again" configuration
          9. Capture the Java Melody memory histogram (see "After test")

          Before test

          After 800 test runs

          Mark Waite added a comment - I'm able to reproduce the issue with Jenkins 2.405 using the following steps: Create a plugins.txt with the list of plugins you provided Create a run-jenkins.sh script that downloads Jenkins 2.405, installs the specified plugins, and starts Jenkins Create a job definition and a Pipeline library definition as described in JENKINS-72067-git.tgz Run Jenkins by executing the run-jenkins.sh script Complete the setup wizard by creating a new user and choosing to install no additional plugins (since they have already been downloaded) Run the test job once, then capture the Java Melody memory histogram (see "Before test") Modify the test job to run itself again as described in the earlier comment from alexsch01 Run the test job and watch it run many times. After about 800 times, reconfigure the job to remove the "run itself again" configuration Capture the Java Melody memory histogram (see "After test") Before test After 800 test runs

          Alex added a comment - - edited

          Thank you for checking this out!

          I used the Eclipse Memory Analyser with a heap dump and it's not obvious where the issue is coming from

          Alex added a comment - - edited Thank you for checking this out! I used the Eclipse Memory Analyser with a heap dump and it's not obvious where the issue is coming from

          Mark Waite added a comment -

          I thought that the issue might be related to the file system SCM plugin, but I see similar results when using the git plugin.

          I thought that the issue might be related to the number of builds being retained in the history, but I see growth even when only 100 builds are retained.

          More investigation will be needed.

          Mark Waite added a comment - I thought that the issue might be related to the file system SCM plugin, but I see similar results when using the git plugin. I thought that the issue might be related to the number of builds being retained in the history, but I see growth even when only 100 builds are retained. More investigation will be needed.

          Alex added a comment -

          markewaite Can you write instructions how to reproduce this with the git plugin?

          Thanks

          Alex added a comment - markewaite  Can you write instructions how to reproduce this with the git plugin? Thanks

          Mark Waite added a comment - - edited

          I'm able to reproduce the issue with Jenkins 2.405 using the following steps:

          1. Create a a-plugins.txt with the list of plugins you provided and rename it plugins.txt
          2. Create a a-run-jenkins.sh script that downloads Jenkins 2.405, installs the specified plugins, and starts Jenkins and rename it to run-jenkins.sh
          3. Create a git based job definition and a Pipeline library definition as described in a-JENKINS-72067-git.tgz
          4. Run Jenkins by executing the run-jenkins.sh script
          5. Complete the setup wizard by creating a new user and choosing to install no additional plugins (since they have already been downloaded)
          6. Run the JENKINS-72067-memory-leak job once, then capture the Java Melody memory histogram
          7. Run the build-100-times test job to run the JENKINS-72067-memory-leak job 100 times without creating a long recursive chain of job causes
          8. Run the build-100-times several more times
          9. Capture the Java Melody memory histogram

          The increased memory use is not as severe in this case, but it is there.

          Mark Waite added a comment - - edited I'm able to reproduce the issue with Jenkins 2.405 using the following steps: Create a a-plugins.txt with the list of plugins you provided and rename it plugins.txt Create a a-run-jenkins.sh script that downloads Jenkins 2.405, installs the specified plugins, and starts Jenkins and rename it to run-jenkins.sh Create a git based job definition and a Pipeline library definition as described in a-JENKINS-72067-git.tgz Run Jenkins by executing the run-jenkins.sh script Complete the setup wizard by creating a new user and choosing to install no additional plugins (since they have already been downloaded) Run the JENKINS-72067 -memory-leak job once, then capture the Java Melody memory histogram Run the build-100-times test job to run the JENKINS-72067 -memory-leak job 100 times without creating a long recursive chain of job causes Run the build-100-times several more times Capture the Java Melody memory histogram The increased memory use is not as severe in this case, but it is there.

          Alex added a comment -

          The temporary solution to downgrade to 2.401.3 LTS worked for me

          Alex added a comment - The temporary solution to downgrade to 2.401.3 LTS worked for me

          Basil Crow added a comment -

          Bisected to jenkinsci/jenkins#7976, which significantly increases the memory usage associated with each instance of XStream2. While this is quite obviously visible with File System SCM (which creates an XStream2 instance per checkout in https://github.com/jenkinsci/filesystem_scm-plugin/blob/1764fa2f66c3aabb6d0aea0b48293aa7b6f9931b/src/main/java/hudson/plugins/filesystem_scm/ChangelogSet.java#L103) where heap usage grows by 100-200 MB or more during the test mentioned above, an increase in heap usage by about 40 MB in the test mentioned above can even be observed with the Git plugin, as mentioned by markewaite above.

          jglick FYI

          Basil Crow added a comment - Bisected to jenkinsci/jenkins#7976 , which significantly increases the memory usage associated with each instance of XStream2 . While this is quite obviously visible with File System SCM (which creates an XStream2 instance per checkout in https://github.com/jenkinsci/filesystem_scm-plugin/blob/1764fa2f66c3aabb6d0aea0b48293aa7b6f9931b/src/main/java/hudson/plugins/filesystem_scm/ChangelogSet.java#L103 ) where heap usage grows by 100-200 MB or more during the test mentioned above, an increase in heap usage by about 40 MB in the test mentioned above can even be observed with the Git plugin, as mentioned by markewaite above. jglick FYI

          Jesse Glick added a comment -

          The filesystem_scm plugin should be creating one XStream2 instance in a static field and reusing it, but at any rate markewaite did I understand that even with just the git plugin there is a constantly increasing memory usage? I could not find anything in the git plugin that would create new XStream2 instances so I was confused by this point. Or was the plateau heap simply higher than it would otherwise be after running at least one Git-based build?

          Jesse Glick added a comment - The filesystem_scm plugin should be creating one XStream2 instance in a static field and reusing it, but at any rate markewaite did I understand that even with just the git plugin there is a constantly increasing memory usage? I could not find anything in the git plugin that would create new XStream2 instances so I was confused by this point. Or was the plateau heap simply higher than it would otherwise be after running at least one Git-based build?

          Alex added a comment -

          jglick I was unable to reproduce the issue with the Git plugin

          Alex added a comment - jglick I was unable to reproduce the issue with the Git plugin

          Jesse Glick added a comment -

          Hmm, that seems to contradict information above, or perhaps it is just “less reproducible”? At any rate, https://github.com/jenkinsci/filesystem_scm-plugin/pull/16 may also help users of existing core versions (untested).

          Jesse Glick added a comment - Hmm, that seems to contradict information above, or perhaps it is just “less reproducible”? At any rate, https://github.com/jenkinsci/filesystem_scm-plugin/pull/16 may also help users of existing core versions (untested).

          Mark Waite added a comment - - edited

          I was wrong when I said that the memory leak affects the git plugin. I ran a comparison between Jenkins 2.404 and 2.405 with the same operation, 1000 runs of a Pipeline job cloning a small repository and found no relevant differences in the memory use histogram. Sorry for the mistake.

          2.404 after 1 run

          2.404 after 1001 runs

          2.405 after 1 run

          2.405 after 1001 runs

          Mark Waite added a comment - - edited I was wrong when I said that the memory leak affects the git plugin. I ran a comparison between Jenkins 2.404 and 2.405 with the same operation, 1000 runs of a Pipeline job cloning a small repository and found no relevant differences in the memory use histogram. Sorry for the mistake. 2.404 after 1 run 2.404 after 1001 runs 2.405 after 1 run 2.405 after 1001 runs

          Jesse Glick added a comment -

          OK, so to confirm: there is a significant heap usage regression affecting users of the filesystem_scm plugin, at least without my PR-16, which might affect other use cases though none are currently known. (The contributing cause in filesystem_scm was constant creation of XStream2 instances, which is specifically discouraged in XStream docs.) Thanks for rechecking!

          Jesse Glick added a comment - OK, so to confirm: there is a significant heap usage regression affecting users of the filesystem_scm plugin, at least without my PR-16, which might affect other use cases though none are currently known. (The contributing cause in filesystem_scm was constant creation of XStream2 instances, which is specifically discouraged in XStream docs.) Thanks for rechecking!

          Basil Crow added a comment -

          That is why I did not use the word "leak" in the PR description of https://github.com/jenkinsci/jenkins/pull/8529.

          Basil Crow added a comment - That is why I did not use the word "leak" in the PR description of https://github.com/jenkinsci/jenkins/pull/8529 .

          Alex added a comment -

          Can we can get this backported to 2.414 LTS series?

          Alex added a comment - Can we can get this backported to 2.414 LTS series?

          Mark Waite added a comment - - edited

          Good question alexsch01. It has been labeled as an LTS candidate. Once it is included in a weekly release, it will be eligible for backport consideration as part of LTS 2.414.3.

          krisstern is the release lead for 2.414.3. He'll review all the backport candidates as he prepares the backporting pull request for the 2.414.3 release.

          Mark Waite added a comment - - edited Good question alexsch01 . It has been labeled as an LTS candidate. Once it is included in a weekly release, it will be eligible for backport consideration as part of LTS 2.414.3. krisstern is the release lead for 2.414.3. He'll review all the backport candidates as he prepares the backporting pull request for the 2.414.3 release.

            markewaite Mark Waite
            alexsch01 Alex
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: