Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63047

Upgrade Groovy to 2.5.6+ to address OOM / Memory leak

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Major Major
    • core, job-dsl-plugin
    • None
    • Jenkins v2.190.2
      Job DSL v1.77

      TL;DR
      We would like to know if it is possible (or already on your roadmap) to upgrade the version of Groovy in jobdsl-plugin/jenkins-core as we believe it will help with a memory leak we've been seeing.

      Background
      We have a use case in our jenkins deployment where we build pipelines via the jobdsl plugin. We have noticed that when we execute this process multiple times we eventually run out of memory.

      After lots of investigation and heap dump analysis we believe the issue is related to https://issues.jenkins-ci.org/browse/JENKINS-46687 and that the constant use of the GroovyShell.parse / parseClass eventually exhausts the memory. Part of the analysis that leads us to believe this is the attached heap analysis showing that there are instances of the MetaMethodIndex class that are never garbage collected. 

      We did try and set -Dgroovy.use.classvalue=true as per https://issues.jenkins-ci.org/browse/JENKINS-33358 , but that doesnt seem to work for us.

      Looking at this comment in the groovy issues it suggests that upgrading to 2.5.6+ will help improve the evaluation of scripts in Groovy.

      So we are wondering if it is possible to schedule (if not already scheduled) the upgrade of Groovy version? Or maybe even if you could suggest some other potential causes of the problem / solutions to try out?

      Please let us know if you need more detail around this.

       

          [JENKINS-63047] Upgrade Groovy to 2.5.6+ to address OOM / Memory leak

          Speaking for Job DSL, I would support updating to 2.5.x and I am willing to make the necessary changes in Job DSL.

          There has been a PR for Jenkins core, but it has been closed. See
          https://github.com/jenkinsci/jenkins/pull/3605. As mentioned in the closing comment by danielbeck this needs to be a joint effort as not only core and job-dsl are affected. There needs to be someone driving the process, e.g. starting a discussion on the mailing list. bez625 can you do that?

          Daniel Spilker added a comment - Speaking for Job DSL, I would support updating to 2.5.x and I am willing to make the necessary changes in Job DSL. There has been a PR for Jenkins core, but it has been closed. See https://github.com/jenkinsci/jenkins/pull/3605 . As mentioned in the closing comment by danielbeck this needs to be a joint effort as not only core and job-dsl are affected. There needs to be someone driving the process, e.g. starting a discussion on the mailing list. bez625 can you do that?

          Daniel Beck added a comment -

          Unless one of (Devin, Andrew, Jesse) is willing to do the work in Pipeline, this is DOA.

          Daniel Beck added a comment - Unless one of (Devin, Andrew, Jesse) is willing to do the work in Pipeline, this is DOA.

          Chris Berry added a comment -

          danielbeck daspilker - thank you both very much for your responses and linking to the github PR - I hadnt thought to check there! I discussed this with my colleagues and we will close the issue for the following reasons:

          1. We do not know for certain this will improve/ease the issue we are facing and it sounds like it could actually raise more issues based on how previous updates to Groovy have gone.
          2. Even if we proved out the first point I do not think we are willing to invest the time and effort it will take to develop and test that the upgrade is safe. Especially when there may be little or no feature benefit to anyone but us.

          Chris Berry added a comment - danielbeck daspilker - thank you both very much for your responses and linking to the github PR - I hadnt thought to check there! I discussed this with my colleagues and we will close the issue for the following reasons: We do not know for certain this will improve/ease the issue we are facing and it sounds like it could actually raise more issues based on how previous updates to Groovy have gone. Even if we proved out the first point I do not think we are willing to invest the time and effort it will take to develop and test that the upgrade is safe. Especially when there may be little or no feature benefit to anyone but us.

          Alexander Stohr added a comment - is it possible? yes/no i have varying indications, e.g. this one tells about 2.5.x was used: https://issues.jenkins.io/browse/JENKINS-53372?focusedCommentId=370924&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-370924

          Daniel Beck added a comment -

          Yes it's (probably) possible.

          No, it won't happen unless someone is willing to spend the extreme effort (weeks of full time work) needed to do this. So far, the reasons for an upgrade are rather weak, so this is fairly unlikely to happen soon, or be done quickly.

           

          Daniel Beck added a comment - Yes it's (probably) possible. No, it won't happen unless someone is willing to spend the extreme effort (weeks of full time work) needed to do this. So far, the reasons for an upgrade are rather weak, so this is fairly unlikely to happen soon, or be done quickly.  

          thank you.

          just for the records i am not the person that currently worries too much on OOM for a (maybe) short lived operation time span.

          instead my own motivation for asking was a problem with Groovy 2.4 missing to add a few recommended tags to generated functions so that reports from code coverage tooling is relatively useless.
          for the moment i feel like its simpler to hope at least back porting the respective fix for Groovy 2.4 works out faster.

          references:

          Alexander Stohr added a comment - thank you. just for the records i am not the person that currently worries too much on OOM for a (maybe) short lived operation time span. instead my own motivation for asking was a problem with Groovy 2.4 missing to add a few recommended tags to generated functions so that reports from code coverage tooling is relatively useless. for the moment i feel like its simpler to hope at least back porting the respective fix for Groovy 2.4 works out faster. references: a report on the JaCoCo bug reporting system on listing generated functions -  https://github.com/jacoco/jacoco/issues/1021 the bug report and fixing report on the -  https://issues.apache.org/jira/browse/GROOVY-9396 the Groovy PullRequest (and source code commit) that resolved the problem   https://github.com/apache/groovy/pull/1166     https://github.com/apache/groovy/pull/1166/commits/651a250df2ba893c17006bcda6b5522cc026032e

          Jim Klimov added a comment - - edited

          Got one more reason why to want this (popular question, balancing caution vs. needs of progress):

          I am trying to use a Jenkins plugin project I've made (with apparently a newer ecosystem) in IDEA which works there, including unit-tests with Jenkins probably inheriting newer Groovy from IDE JVM. Unfortunately it does not work in standalone Jenkins, and it involves interface static methods introduced in Java 8. This suffers the issue discussed at https://issues.apache.org/jira/browse/GROOVY-8338?focusedCommentId=16325571&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16325571 with the likes of:

          java.lang.VerifyError: (class: example/app/InterfaceWithStaticMethod$someStaticMethod, method: callStatic signature: (Ljava/lang/Class;[Ljava/lang/Object;)Ljava/lang/Object Illegal type in constant pool 

          I verified that the problem goes away, by hacking my local Jenkins instance to replace groovy-2.4.21.jar with a bunch of JARs downloaded as dependency for my plugin when I told it to require groovy-2.5.21 (as the newest 2.5.x currently). For a quick shot test with standalone Jenkins, at least my plugin code began behaving as expected. In particular, this brought in a seemingly functional implementation of Groovy Traits which are a nice language feature. I did not check if something else broke (nor know of a good way how to check that).

          But having spent a couple of weeks coding a library that we could really benefit from, just to find it is not compatible with the obsolete run-time platform, is disappointing, at least...

          Jim Klimov added a comment - - edited Got one more reason why to want this (popular question, balancing caution vs. needs of progress): I am trying to use a Jenkins plugin project I've made (with apparently a newer ecosystem) in IDEA which works there, including unit-tests with Jenkins probably inheriting newer Groovy from IDE JVM. Unfortunately it does not work in standalone Jenkins, and it involves interface static methods introduced in Java 8. This suffers the issue discussed at https://issues.apache.org/jira/browse/GROOVY-8338?focusedCommentId=16325571&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16325571 with the likes of: java.lang.VerifyError: (class: example/app/InterfaceWithStaticMethod$someStaticMethod, method: callStatic signature: (Ljava/lang/ Class ;[Ljava/lang/ Object ;)Ljava/lang/ Object Illegal type in constant pool I verified that the problem goes away, by hacking my local Jenkins instance to replace groovy-2.4.21.jar with a bunch of JARs downloaded as dependency for my plugin when I told it to require groovy-2.5.21 (as the newest 2.5.x currently). For a quick shot test with standalone Jenkins, at least my plugin code began behaving as expected. In particular, this brought in a seemingly functional implementation of Groovy Traits which are a nice language feature. I did not check if something else broke (nor know of a good way how to check that). But having spent a couple of weeks coding a library that we could really benefit from, just to find it is not compatible with the obsolete run-time platform, is disappointing, at least...

            bez625 Chris Berry
            bez625 Chris Berry
            Votes:
            1 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: