• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • core

      We have recently experienced OutOfMemory errors on some large Jenkins instances that predominantly have frequently executed jobs that contain either System Groovy scripts or Build Flow project types. In one case after disabling a groovy script that was being executed via the Global Post Script Plugin after every build, the PermGen profile improved dramatically.

      We tested further with GroovyShell to run a groovy script, not depending on Jenkins.
      Java7: java.lang.OutOfMemoryError: PermGen space
      Java8: java.lang.OutOfMemoryError: Java heap space

      The tests use 2.4.6 version of groovy which is the latest version. However, the issue is still there.
      <dependency>
      <groupId>org.codehaus.groovy</groupId>
      <artifactId>groovy-all</artifactId>
      <version>2.4.6</version>
      </dependency>

      ---------------------------------------
      Configure 10 jobs which build per minute
      Each build triggers a groovy script by Global Post Script Plugin.

      class Main {
          static void main(String... args) {
              println 'Groovy world!'
              def p = new Person()
              p.write()
          }
      }
      
      class Person {
          def write(){
              println 'i am writing ...!'
          }
      }
      

      After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.

      I checked the source code of the plugin:
      https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

      My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
      1. if this is a potential memory leak, and
      2. if anyone has met similar issue.

        1. after.png
          after.png
          43 kB
        2. before.png
          before.png
          70 kB
        3. permgenIssue.jpg
          permgenIssue.jpg
          35 kB
        4. Screenshot from 2016-03-07 09.43.29.png
          Screenshot from 2016-03-07 09.43.29.png
          113 kB
        5. Screenshot from 2016-04-01 11.04.43.png
          Screenshot from 2016-04-01 11.04.43.png
          182 kB

          [JENKINS-33358] Groovy and PermGen memory leak

          When will this fix be available in LTS?

          Christer Engde added a comment - When will this fix be available in LTS?

          Daniel Beck added a comment -

          We decided today that the next LTS will be based on 2.46 due to the risk of further regressions/incompatibilities caused by the upgrade to Groovy 2.4.8.

          So this will likely only be in LTS 16 weeks from today unless we think it's safe enough for a backport (and I would be surprised if we'd consider it safe enough).

          Daniel Beck added a comment - We decided today that the next LTS will be based on 2.46 due to the risk of further regressions/incompatibilities caused by the upgrade to Groovy 2.4.8. So this will likely only be in LTS 16 weeks from today unless we think it's safe enough for a backport (and I would be surprised if we'd consider it safe enough).

          Daniel Beck added a comment -

          Labeling just in case we do think it's safe enough (but don't get your hopes up).

          Daniel Beck added a comment - Labeling just in case we do think it's safe enough (but don't get your hopes up).

          I am rejecting this from 2.46.1 as agreed. There is first issue that is suspected to be caused by this change: JENKINS-42637.

          Oliver Gondža added a comment - I am rejecting this from 2.46.1 as agreed. There is first issue that is suspected to be caused by this change: JENKINS-42637 .

          Reporting successful fix with the new groovy version.

          We updated an affected instance on Java 8, LTS version 2.7.4 to the non-LTS 2.48 and have been running the server for a week. The memory is under control now.

          +1 in bringing this into LTS

          Samuel Beaulieu added a comment - Reporting successful fix with the new groovy version. We updated an affected instance on Java 8, LTS version 2.7.4 to the non-LTS 2.48 and have been running the server for a week. The memory is under control now. +1 in bringing this into LTS

          Daniel Beck added a comment -

          2.73.1 is scheduled for release this week and will contain Groovy 2.4.11.

          Daniel Beck added a comment - 2.73.1 is scheduled for release this week and will contain Groovy 2.4.11.

          Jesse Glick added a comment -

          In 2.61 apparently.

          Jesse Glick added a comment - In 2.61 apparently.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          Jenkinsfile
          http://jenkins-ci.org/commit/workflow-cps-plugin/9c5667d59d47d4bff4e3dfd5f68ea496448029e1
          Log:
          JENKINS-33358 Test against 2.73.x since that includes Groovy 2.4.11.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: Jenkinsfile http://jenkins-ci.org/commit/workflow-cps-plugin/9c5667d59d47d4bff4e3dfd5f68ea496448029e1 Log: JENKINS-33358 Test against 2.73.x since that includes Groovy 2.4.11.

          Code changed in jenkins
          User: Sam Van Oort
          Path:
          Jenkinsfile
          pom.xml
          http://jenkins-ci.org/commit/workflow-cps-plugin/a5dea90a589f094caf58411c271f139322f37f2f
          Log:
          Merge pull request #175 from jglick/new-Groovy-JENKINS-33358

          JENKINS-33358 Test against 2.73.x since that includes Groovy 2.4.11

          Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/3396c9715894...a5dea90a589f

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Sam Van Oort Path: Jenkinsfile pom.xml http://jenkins-ci.org/commit/workflow-cps-plugin/a5dea90a589f094caf58411c271f139322f37f2f Log: Merge pull request #175 from jglick/new-Groovy- JENKINS-33358 JENKINS-33358 Test against 2.73.x since that includes Groovy 2.4.11 Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/3396c9715894...a5dea90a589f

          Sam Van Oort added a comment -

          https://issues.jenkins-ci.org/browse/JENKINS-47758 might offer us a way to guard against some of the memory leaks. 

          Sam Van Oort added a comment - https://issues.jenkins-ci.org/browse/JENKINS-47758  might offer us a way to guard against some of the memory leaks. 

            ympaul2 paul young
            hongkailiu Hongkai Liu
            Votes:
            23 Vote for this issue
            Watchers:
            53 Start watching this issue

              Created:
              Updated:
              Resolved: