Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33358

Groovy and PermGen memory leak

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: core
    • Environment:
    • Similar Issues:

      Description

      We have recently experienced OutOfMemory errors on some large Jenkins instances that predominantly have frequently executed jobs that contain either System Groovy scripts or Build Flow project types. In one case after disabling a groovy script that was being executed via the Global Post Script Plugin after every build, the PermGen profile improved dramatically.

      We tested further with GroovyShell to run a groovy script, not depending on Jenkins.
      Java7: java.lang.OutOfMemoryError: PermGen space
      Java8: java.lang.OutOfMemoryError: Java heap space

      The tests use 2.4.6 version of groovy which is the latest version. However, the issue is still there.
      <dependency>
      <groupId>org.codehaus.groovy</groupId>
      <artifactId>groovy-all</artifactId>
      <version>2.4.6</version>
      </dependency>

      ---------------------------------------
      Configure 10 jobs which build per minute
      Each build triggers a groovy script by Global Post Script Plugin.

      class Main {
          static void main(String... args) {
              println 'Groovy world!'
              def p = new Person()
              p.write()
          }
      }
      
      class Person {
          def write(){
              println 'i am writing ...!'
          }
      }
      

      After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.

      I checked the source code of the plugin:
      https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

      My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
      1. if this is a potential memory leak, and
      2. if anyone has met similar issue.

        Attachments

        1. after.png
          after.png
          43 kB
        2. before.png
          before.png
          70 kB
        3. permgenIssue.jpg
          permgenIssue.jpg
          35 kB
        4. Screenshot from 2016-03-07 09.43.29.png
          Screenshot from 2016-03-07 09.43.29.png
          113 kB
        5. Screenshot from 2016-04-01 11.04.43.png
          Screenshot from 2016-04-01 11.04.43.png
          182 kB

          Issue Links

            Activity

            hongkailiu Hongkai Liu created issue -
            hongkailiu Hongkai Liu made changes -
            Field Original Value New Value
            Description Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            hongkailiu Hongkai Liu made changes -
            Issue Type Bug [ 1 ] Story [ 10002 ]
            hongkailiu Hongkai Liu made changes -
            Labels groovy memory-leak
            hongkailiu Hongkai Liu made changes -
            Description Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            We got the problem from the operation team who observe that the perm gen size has been increased by monitoring plugin.
            This is quite important for us because more and more groovy based steps are integrated into our CI procedure.
            After analysis, we simplify the testing environment with 10 jobs and Global Post Script Plugin.

            We test further with GroovyShell to run a groovy script, not depending on Jenkins.
            Java7: java.lang.OutOfMemoryError: PermGen space
            Java8: java.lang.OutOfMemoryError: Java heap space

            ---------------------------------------
            Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            hongkailiu Hongkai Liu made changes -
            Description We got the problem from the operation team who observe that the perm gen size has been increased by monitoring plugin.
            This is quite important for us because more and more groovy based steps are integrated into our CI procedure.
            After analysis, we simplify the testing environment with 10 jobs and Global Post Script Plugin.

            We test further with GroovyShell to run a groovy script, not depending on Jenkins.
            Java7: java.lang.OutOfMemoryError: PermGen space
            Java8: java.lang.OutOfMemoryError: Java heap space

            ---------------------------------------
            Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            We got the problem from the operation team who observe that the perm gen size has been increased by monitoring plugin.
            This is quite important for us because more and more groovy based steps are integrated into our CI procedure.
            After analysis, we simplify the testing environment with 10 jobs and Global Post Script Plugin.

            We test further with GroovyShell to run a groovy script, not depending on Jenkins.
            Java7: java.lang.OutOfMemoryError: PermGen space
            Java8: java.lang.OutOfMemoryError: Java heap space

            The tests use 2.4.6 version of groovy which is the latest version. However, the issue is still there.
            <dependency>
                        <groupId>org.codehaus.groovy</groupId>
                        <artifactId>groovy-all</artifactId>
                        <version>2.4.6</version>
            </dependency>

            ---------------------------------------
            Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            hongkailiu Hongkai Liu made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            chris31421 Christian Lague made changes -
            Issue Type Story [ 10002 ] Bug [ 1 ]
            chris31421 Christian Lague made changes -
            Component/s core [ 15593 ]
            Component/s global-post-script-plugin [ 20637 ]
            chris31421 Christian Lague made changes -
            Assignee Hao CHEN [ orctom ] Jesse Glick [ jglick ]
            chris31421 Christian Lague made changes -
            Description We got the problem from the operation team who observe that the perm gen size has been increased by monitoring plugin.
            This is quite important for us because more and more groovy based steps are integrated into our CI procedure.
            After analysis, we simplify the testing environment with 10 jobs and Global Post Script Plugin.

            We test further with GroovyShell to run a groovy script, not depending on Jenkins.
            Java7: java.lang.OutOfMemoryError: PermGen space
            Java8: java.lang.OutOfMemoryError: Java heap space

            The tests use 2.4.6 version of groovy which is the latest version. However, the issue is still there.
            <dependency>
                        <groupId>org.codehaus.groovy</groupId>
                        <artifactId>groovy-all</artifactId>
                        <version>2.4.6</version>
            </dependency>

            ---------------------------------------
            Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            We have recently experienced OutOfMemory errors on some large Jenkins instances that predominantly have frequently executed jobs that contain either System Groovy scripts or Build Flow project types. In one case after disabling a groovy script that was being executed via the Global Post Script Plugin after every build, the PermGen profile improved as shown:


            We test further with GroovyShell to run a groovy script, not depending on Jenkins.
            Java7: java.lang.OutOfMemoryError: PermGen space
            Java8: java.lang.OutOfMemoryError: Java heap space

            The tests use 2.4.6 version of groovy which is the latest version. However, the issue is still there.
            <dependency>
                        <groupId>org.codehaus.groovy</groupId>
                        <artifactId>groovy-all</artifactId>
                        <version>2.4.6</version>
            </dependency>

            ---------------------------------------
            Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            chris31421 Christian Lague made changes -
            Description We have recently experienced OutOfMemory errors on some large Jenkins instances that predominantly have frequently executed jobs that contain either System Groovy scripts or Build Flow project types. In one case after disabling a groovy script that was being executed via the Global Post Script Plugin after every build, the PermGen profile improved as shown:


            We test further with GroovyShell to run a groovy script, not depending on Jenkins.
            Java7: java.lang.OutOfMemoryError: PermGen space
            Java8: java.lang.OutOfMemoryError: Java heap space

            The tests use 2.4.6 version of groovy which is the latest version. However, the issue is still there.
            <dependency>
                        <groupId>org.codehaus.groovy</groupId>
                        <artifactId>groovy-all</artifactId>
                        <version>2.4.6</version>
            </dependency>

            ---------------------------------------
            Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            We have recently experienced OutOfMemory errors on some large Jenkins instances that predominantly have frequently executed jobs that contain either System Groovy scripts or Build Flow project types. In one case after disabling a groovy script that was being executed via the Global Post Script Plugin after every build, the PermGen profile improved dramatically.

            We tested further with GroovyShell to run a groovy script, not depending on Jenkins.
            Java7: java.lang.OutOfMemoryError: PermGen space
            Java8: java.lang.OutOfMemoryError: Java heap space

            The tests use 2.4.6 version of groovy which is the latest version. However, the issue is still there.
            <dependency>
                        <groupId>org.codehaus.groovy</groupId>
                        <artifactId>groovy-all</artifactId>
                        <version>2.4.6</version>
            </dependency>

            ---------------------------------------
            Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            chris31421 Christian Lague made changes -
            Attachment permgenIssue.jpg [ 32133 ]
            Description We have recently experienced OutOfMemory errors on some large Jenkins instances that predominantly have frequently executed jobs that contain either System Groovy scripts or Build Flow project types. In one case after disabling a groovy script that was being executed via the Global Post Script Plugin after every build, the PermGen profile improved dramatically.

            We tested further with GroovyShell to run a groovy script, not depending on Jenkins.
            Java7: java.lang.OutOfMemoryError: PermGen space
            Java8: java.lang.OutOfMemoryError: Java heap space

            The tests use 2.4.6 version of groovy which is the latest version. However, the issue is still there.
            <dependency>
                        <groupId>org.codehaus.groovy</groupId>
                        <artifactId>groovy-all</artifactId>
                        <version>2.4.6</version>
            </dependency>

            ---------------------------------------
            Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            We have recently experienced OutOfMemory errors on some large Jenkins instances that predominantly have frequently executed jobs that contain either System Groovy scripts or Build Flow project types. In one case after disabling a groovy script that was being executed via the Global Post Script Plugin after every build, the PermGen profile improved dramatically.

            !permgenIssue.jpg|thumbnail!

            We tested further with GroovyShell to run a groovy script, not depending on Jenkins.
            Java7: java.lang.OutOfMemoryError: PermGen space
            Java8: java.lang.OutOfMemoryError: Java heap space

            The tests use 2.4.6 version of groovy which is the latest version. However, the issue is still there.
            <dependency>
                        <groupId>org.codehaus.groovy</groupId>
                        <artifactId>groovy-all</artifactId>
                        <version>2.4.6</version>
            </dependency>

            ---------------------------------------
            Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            orctom Hao CHEN made changes -
            Assignee Jesse Glick [ jglick ] Hongkai Liu [ hongkailiu ]
            hongkailiu Hongkai Liu made changes -
            hongkailiu Hongkai Liu made changes -
            Assignee Hongkai Liu [ hongkailiu ] Hao CHEN [ orctom ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 169271 ] JNJira + In-Review [ 183427 ]
            daspilker Daniel Spilker made changes -
            Link This issue is related to JENKINS-37213 [ JENKINS-37213 ]
            gmc_devel GMC Software Development B&R Corporate made changes -
            Link This issue relates to JENKINS-38334 [ JENKINS-38334 ]
            danielbeck Daniel Beck made changes -
            Link This issue is duplicated by JENKINS-38503 [ JENKINS-38503 ]
            ceilfors Wisen Tanasa made changes -
            Attachment before.png [ 34830 ]
            ceilfors Wisen Tanasa made changes -
            Attachment after.png [ 34831 ]
            scm_issue_link SCM/JIRA link daemon made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            jglick Jesse Glick made changes -
            Link This issue is blocked by JENKINS-42189 [ JENKINS-42189 ]
            danielbeck Daniel Beck made changes -
            Labels groovy memory-leak groovy lts-candidate memory-leak
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is related to JENKINS-42637 [ JENKINS-42637 ]
            olivergondza Oliver Gondža made changes -
            Labels groovy lts-candidate memory-leak 2.46.1-rejected groovy lts-candidate memory-leak
            ympaul2 paul young made changes -
            Assignee Hao CHEN [ orctom ] paul young [ ympaul2 ]
            olivergondza Oliver Gondža made changes -
            Labels 2.46.1-rejected groovy lts-candidate memory-leak 2.46.1-rejected 2.46.2-rejected groovy lts-candidate memory-leak
            olivergondza Oliver Gondža made changes -
            Labels 2.46.1-rejected 2.46.2-rejected groovy lts-candidate memory-leak 2.46.1-rejected 2.46.2-rejected 2.46.3-rejected groovy memory-leak
            jamesdumay James Dumay made changes -
            Remote Link This issue links to "CloudBees Internal OSS-2127 (Web Link)" [ 18410 ]
            jamesdumay James Dumay made changes -
            Remote Link This issue links to "CloudBees Internal OSS-2079 (Web Link)" [ 18429 ]
            dehrk Derek Kinney made changes -
            Link This issue relates to JENKINS-38334 [ JENKINS-38334 ]
            dehrk Derek Kinney made changes -
            Link This issue relates to JENKINS-38334 [ JENKINS-38334 ]

              People

              Assignee:
              ympaul2 paul young
              Reporter:
              hongkailiu Hongkai Liu
              Votes:
              23 Vote for this issue
              Watchers:
              53 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: