Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-72067

File system SCM uses more memory than expected with Jenkins 2.405 - 2.425

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • 2.426, 2.414.3

      See my comments below, this issue started in 2.405

      (Same behavior happens whether running on same node as the master or not)

      I have a test job that triggers itself (also I set quiet period to 0), the job uses SCM to run the example Hello World pipeline

      After a few minutes, this is the result of http://localhost:8080/monitoring?part=heaphisto

      If I downgrade to 2.404.3 and try the same thing again....

        1. 01-before-test-run.png
          01-before-test-run.png
          77 kB
        2. 02-after-800-test-runs.png
          02-after-800-test-runs.png
          64 kB
        3. 2-404-after.png
          2-404-after.png
          208 kB
        4. 2-404-before.png
          2-404-before.png
          228 kB
        5. 2-405-after.png
          2-405-after.png
          208 kB
        6. 2-405-before.png
          2-405-before.png
          224 kB
        7. a-JENKINS-72067-git.tgz
          4 kB
        8. a-plugins.txt
          2 kB
        9. a-run-jenkins.sh
          0.8 kB
        10. image-2023-09-24-10-46-06-168.png
          image-2023-09-24-10-46-06-168.png
          99 kB
        11. image-2023-09-24-11-28-08-440.png
          image-2023-09-24-11-28-08-440.png
          105 kB
        12. image-2023-09-25-08-42-47-692.png
          image-2023-09-25-08-42-47-692.png
          73 kB
        13. JENKINS-72067-git.tgz
          2 kB
        14. plugins.txt
          2 kB
        15. run-jenkins.sh
          0.8 kB

          [JENKINS-72067] File system SCM uses more memory than expected with Jenkins 2.405 - 2.425

          Alex added a comment -

          The temporary solution to downgrade to 2.401.3 LTS worked for me

          Alex added a comment - The temporary solution to downgrade to 2.401.3 LTS worked for me

          Basil Crow added a comment -

          Bisected to jenkinsci/jenkins#7976, which significantly increases the memory usage associated with each instance of XStream2. While this is quite obviously visible with File System SCM (which creates an XStream2 instance per checkout in https://github.com/jenkinsci/filesystem_scm-plugin/blob/1764fa2f66c3aabb6d0aea0b48293aa7b6f9931b/src/main/java/hudson/plugins/filesystem_scm/ChangelogSet.java#L103) where heap usage grows by 100-200 MB or more during the test mentioned above, an increase in heap usage by about 40 MB in the test mentioned above can even be observed with the Git plugin, as mentioned by markewaite above.

          jglick FYI

          Basil Crow added a comment - Bisected to jenkinsci/jenkins#7976 , which significantly increases the memory usage associated with each instance of XStream2 . While this is quite obviously visible with File System SCM (which creates an XStream2 instance per checkout in https://github.com/jenkinsci/filesystem_scm-plugin/blob/1764fa2f66c3aabb6d0aea0b48293aa7b6f9931b/src/main/java/hudson/plugins/filesystem_scm/ChangelogSet.java#L103 ) where heap usage grows by 100-200 MB or more during the test mentioned above, an increase in heap usage by about 40 MB in the test mentioned above can even be observed with the Git plugin, as mentioned by markewaite above. jglick FYI

          Jesse Glick added a comment -

          The filesystem_scm plugin should be creating one XStream2 instance in a static field and reusing it, but at any rate markewaite did I understand that even with just the git plugin there is a constantly increasing memory usage? I could not find anything in the git plugin that would create new XStream2 instances so I was confused by this point. Or was the plateau heap simply higher than it would otherwise be after running at least one Git-based build?

          Jesse Glick added a comment - The filesystem_scm plugin should be creating one XStream2 instance in a static field and reusing it, but at any rate markewaite did I understand that even with just the git plugin there is a constantly increasing memory usage? I could not find anything in the git plugin that would create new XStream2 instances so I was confused by this point. Or was the plateau heap simply higher than it would otherwise be after running at least one Git-based build?

          Alex added a comment -

          jglick I was unable to reproduce the issue with the Git plugin

          Alex added a comment - jglick I was unable to reproduce the issue with the Git plugin

          Jesse Glick added a comment -

          Hmm, that seems to contradict information above, or perhaps it is just “less reproducible”? At any rate, https://github.com/jenkinsci/filesystem_scm-plugin/pull/16 may also help users of existing core versions (untested).

          Jesse Glick added a comment - Hmm, that seems to contradict information above, or perhaps it is just “less reproducible”? At any rate, https://github.com/jenkinsci/filesystem_scm-plugin/pull/16 may also help users of existing core versions (untested).

          Mark Waite added a comment - - edited

          I was wrong when I said that the memory leak affects the git plugin. I ran a comparison between Jenkins 2.404 and 2.405 with the same operation, 1000 runs of a Pipeline job cloning a small repository and found no relevant differences in the memory use histogram. Sorry for the mistake.

          2.404 after 1 run

          2.404 after 1001 runs

          2.405 after 1 run

          2.405 after 1001 runs

          Mark Waite added a comment - - edited I was wrong when I said that the memory leak affects the git plugin. I ran a comparison between Jenkins 2.404 and 2.405 with the same operation, 1000 runs of a Pipeline job cloning a small repository and found no relevant differences in the memory use histogram. Sorry for the mistake. 2.404 after 1 run 2.404 after 1001 runs 2.405 after 1 run 2.405 after 1001 runs

          Jesse Glick added a comment -

          OK, so to confirm: there is a significant heap usage regression affecting users of the filesystem_scm plugin, at least without my PR-16, which might affect other use cases though none are currently known. (The contributing cause in filesystem_scm was constant creation of XStream2 instances, which is specifically discouraged in XStream docs.) Thanks for rechecking!

          Jesse Glick added a comment - OK, so to confirm: there is a significant heap usage regression affecting users of the filesystem_scm plugin, at least without my PR-16, which might affect other use cases though none are currently known. (The contributing cause in filesystem_scm was constant creation of XStream2 instances, which is specifically discouraged in XStream docs.) Thanks for rechecking!

          Basil Crow added a comment -

          That is why I did not use the word "leak" in the PR description of https://github.com/jenkinsci/jenkins/pull/8529.

          Basil Crow added a comment - That is why I did not use the word "leak" in the PR description of https://github.com/jenkinsci/jenkins/pull/8529 .

          Alex added a comment -

          Can we can get this backported to 2.414 LTS series?

          Alex added a comment - Can we can get this backported to 2.414 LTS series?

          Mark Waite added a comment - - edited

          Good question alexsch01. It has been labeled as an LTS candidate. Once it is included in a weekly release, it will be eligible for backport consideration as part of LTS 2.414.3.

          krisstern is the release lead for 2.414.3. He'll review all the backport candidates as he prepares the backporting pull request for the 2.414.3 release.

          Mark Waite added a comment - - edited Good question alexsch01 . It has been labeled as an LTS candidate. Once it is included in a weekly release, it will be eligible for backport consideration as part of LTS 2.414.3. krisstern is the release lead for 2.414.3. He'll review all the backport candidates as he prepares the backporting pull request for the 2.414.3 release.

            markewaite Mark Waite
            alexsch01 Alex
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: