Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33358

Groovy and PermGen memory leak

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Critical
    • Resolution: Fixed
    • core

    Description

      We have recently experienced OutOfMemory errors on some large Jenkins instances that predominantly have frequently executed jobs that contain either System Groovy scripts or Build Flow project types. In one case after disabling a groovy script that was being executed via the Global Post Script Plugin after every build, the PermGen profile improved dramatically.

      We tested further with GroovyShell to run a groovy script, not depending on Jenkins.
      Java7: java.lang.OutOfMemoryError: PermGen space
      Java8: java.lang.OutOfMemoryError: Java heap space

      The tests use 2.4.6 version of groovy which is the latest version. However, the issue is still there.
      <dependency>
      <groupId>org.codehaus.groovy</groupId>
      <artifactId>groovy-all</artifactId>
      <version>2.4.6</version>
      </dependency>

      ---------------------------------------
      Configure 10 jobs which build per minute
      Each build triggers a groovy script by Global Post Script Plugin.

      class Main {
          static void main(String... args) {
              println 'Groovy world!'
              def p = new Person()
              p.write()
          }
      }
      
      class Person {
          def write(){
              println 'i am writing ...!'
          }
      }
      

      After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.

      I checked the source code of the plugin:
      https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

      My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
      1. if this is a potential memory leak, and
      2. if anyone has met similar issue.

      Attachments

        1. after.png
          after.png
          43 kB
        2. before.png
          before.png
          70 kB
        3. permgenIssue.jpg
          permgenIssue.jpg
          35 kB
        4. Screenshot from 2016-03-07 09.43.29.png
          Screenshot from 2016-03-07 09.43.29.png
          113 kB
        5. Screenshot from 2016-04-01 11.04.43.png
          Screenshot from 2016-04-01 11.04.43.png
          182 kB

        Issue Links

          Activity

            hongkailiu Hongkai Liu created issue -
            hongkailiu Hongkai Liu made changes -
            Field Original Value New Value
            Description Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            hongkailiu Hongkai Liu made changes -
            Issue Type Bug [ 1 ] Story [ 10002 ]
            hongkailiu Hongkai Liu made changes -
            Labels groovy memory-leak
            hongkailiu Hongkai Liu made changes -
            Description Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            We got the problem from the operation team who observe that the perm gen size has been increased by monitoring plugin.
            This is quite important for us because more and more groovy based steps are integrated into our CI procedure.
            After analysis, we simplify the testing environment with 10 jobs and Global Post Script Plugin.

            We test further with GroovyShell to run a groovy script, not depending on Jenkins.
            Java7: java.lang.OutOfMemoryError: PermGen space
            Java8: java.lang.OutOfMemoryError: Java heap space

            ---------------------------------------
            Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            hongkailiu Hongkai Liu made changes -
            Description We got the problem from the operation team who observe that the perm gen size has been increased by monitoring plugin.
            This is quite important for us because more and more groovy based steps are integrated into our CI procedure.
            After analysis, we simplify the testing environment with 10 jobs and Global Post Script Plugin.

            We test further with GroovyShell to run a groovy script, not depending on Jenkins.
            Java7: java.lang.OutOfMemoryError: PermGen space
            Java8: java.lang.OutOfMemoryError: Java heap space

            ---------------------------------------
            Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            We got the problem from the operation team who observe that the perm gen size has been increased by monitoring plugin.
            This is quite important for us because more and more groovy based steps are integrated into our CI procedure.
            After analysis, we simplify the testing environment with 10 jobs and Global Post Script Plugin.

            We test further with GroovyShell to run a groovy script, not depending on Jenkins.
            Java7: java.lang.OutOfMemoryError: PermGen space
            Java8: java.lang.OutOfMemoryError: Java heap space

            The tests use 2.4.6 version of groovy which is the latest version. However, the issue is still there.
            <dependency>
                        <groupId>org.codehaus.groovy</groupId>
                        <artifactId>groovy-all</artifactId>
                        <version>2.4.6</version>
            </dependency>

            ---------------------------------------
            Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            hongkailiu Hongkai Liu made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            chris31421 Christian Lague made changes -
            Issue Type Story [ 10002 ] Bug [ 1 ]
            chris31421 Christian Lague made changes -
            Component/s core [ 15593 ]
            Component/s global-post-script-plugin [ 20637 ]
            chris31421 Christian Lague made changes -
            Assignee Hao CHEN [ orctom ] Jesse Glick [ jglick ]
            chris31421 Christian Lague made changes -
            Description We got the problem from the operation team who observe that the perm gen size has been increased by monitoring plugin.
            This is quite important for us because more and more groovy based steps are integrated into our CI procedure.
            After analysis, we simplify the testing environment with 10 jobs and Global Post Script Plugin.

            We test further with GroovyShell to run a groovy script, not depending on Jenkins.
            Java7: java.lang.OutOfMemoryError: PermGen space
            Java8: java.lang.OutOfMemoryError: Java heap space

            The tests use 2.4.6 version of groovy which is the latest version. However, the issue is still there.
            <dependency>
                        <groupId>org.codehaus.groovy</groupId>
                        <artifactId>groovy-all</artifactId>
                        <version>2.4.6</version>
            </dependency>

            ---------------------------------------
            Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            We have recently experienced OutOfMemory errors on some large Jenkins instances that predominantly have frequently executed jobs that contain either System Groovy scripts or Build Flow project types. In one case after disabling a groovy script that was being executed via the Global Post Script Plugin after every build, the PermGen profile improved as shown:


            We test further with GroovyShell to run a groovy script, not depending on Jenkins.
            Java7: java.lang.OutOfMemoryError: PermGen space
            Java8: java.lang.OutOfMemoryError: Java heap space

            The tests use 2.4.6 version of groovy which is the latest version. However, the issue is still there.
            <dependency>
                        <groupId>org.codehaus.groovy</groupId>
                        <artifactId>groovy-all</artifactId>
                        <version>2.4.6</version>
            </dependency>

            ---------------------------------------
            Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            chris31421 Christian Lague made changes -
            Description We have recently experienced OutOfMemory errors on some large Jenkins instances that predominantly have frequently executed jobs that contain either System Groovy scripts or Build Flow project types. In one case after disabling a groovy script that was being executed via the Global Post Script Plugin after every build, the PermGen profile improved as shown:


            We test further with GroovyShell to run a groovy script, not depending on Jenkins.
            Java7: java.lang.OutOfMemoryError: PermGen space
            Java8: java.lang.OutOfMemoryError: Java heap space

            The tests use 2.4.6 version of groovy which is the latest version. However, the issue is still there.
            <dependency>
                        <groupId>org.codehaus.groovy</groupId>
                        <artifactId>groovy-all</artifactId>
                        <version>2.4.6</version>
            </dependency>

            ---------------------------------------
            Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            We have recently experienced OutOfMemory errors on some large Jenkins instances that predominantly have frequently executed jobs that contain either System Groovy scripts or Build Flow project types. In one case after disabling a groovy script that was being executed via the Global Post Script Plugin after every build, the PermGen profile improved dramatically.

            We tested further with GroovyShell to run a groovy script, not depending on Jenkins.
            Java7: java.lang.OutOfMemoryError: PermGen space
            Java8: java.lang.OutOfMemoryError: Java heap space

            The tests use 2.4.6 version of groovy which is the latest version. However, the issue is still there.
            <dependency>
                        <groupId>org.codehaus.groovy</groupId>
                        <artifactId>groovy-all</artifactId>
                        <version>2.4.6</version>
            </dependency>

            ---------------------------------------
            Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            chris31421 Christian Lague made changes -
            Attachment permgenIssue.jpg [ 32133 ]
            Description We have recently experienced OutOfMemory errors on some large Jenkins instances that predominantly have frequently executed jobs that contain either System Groovy scripts or Build Flow project types. In one case after disabling a groovy script that was being executed via the Global Post Script Plugin after every build, the PermGen profile improved dramatically.

            We tested further with GroovyShell to run a groovy script, not depending on Jenkins.
            Java7: java.lang.OutOfMemoryError: PermGen space
            Java8: java.lang.OutOfMemoryError: Java heap space

            The tests use 2.4.6 version of groovy which is the latest version. However, the issue is still there.
            <dependency>
                        <groupId>org.codehaus.groovy</groupId>
                        <artifactId>groovy-all</artifactId>
                        <version>2.4.6</version>
            </dependency>

            ---------------------------------------
            Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            We have recently experienced OutOfMemory errors on some large Jenkins instances that predominantly have frequently executed jobs that contain either System Groovy scripts or Build Flow project types. In one case after disabling a groovy script that was being executed via the Global Post Script Plugin after every build, the PermGen profile improved dramatically.

            !permgenIssue.jpg|thumbnail!

            We tested further with GroovyShell to run a groovy script, not depending on Jenkins.
            Java7: java.lang.OutOfMemoryError: PermGen space
            Java8: java.lang.OutOfMemoryError: Java heap space

            The tests use 2.4.6 version of groovy which is the latest version. However, the issue is still there.
            <dependency>
                        <groupId>org.codehaus.groovy</groupId>
                        <artifactId>groovy-all</artifactId>
                        <version>2.4.6</version>
            </dependency>

            ---------------------------------------
            Configure 10 jobs which build per minute
            Each build triggers a groovy script by Global Post Script Plugin.

            {noformat}
            class Main {
                static void main(String... args) {
                    println 'Groovy world!'
                    def p = new Person()
                    p.write()
                }
            }

            class Person {
                def write(){
                    println 'i am writing ...!'
                }
            }
            {noformat}

            After sometime, the perm gen is eaten up. See the screenshot of jvisualvm.
            !Screenshot from 2016-03-07 09.43.29.png|thumbnail!

            I checked the source code of the plugin:
            https://github.com/jenkinsci/global-post-script-plugin/blob/master/src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java

            My feeling is that it is probably caused by GroovyShell. Jenkins loads and runs groovy extensively and maybe there are places in core or other plugins having similar logic, I wonder
            1. if this is a potential memory leak, and
            2. if anyone has met similar issue.





            Code changed in jenkins
            User: Hao
            Path:
            src/main/java/com/orctom/jenkins/plugin/globalpostscript/GlobalPostScript.java
            src/main/java/com/orctom/jenkins/plugin/globalpostscript/GlobalPostScriptAction.java
            src/main/java/com/orctom/jenkins/plugin/globalpostscript/GlobalPostScriptPlugin.java
            src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptContentLoader.java
            src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java
            src/main/java/com/orctom/jenkins/plugin/globalpostscript/URL.java
            src/main/java/com/orctom/jenkins/plugin/globalpostscript/model/ScriptContent.java
            src/main/java/com/orctom/jenkins/plugin/globalpostscript/model/URL.java
            src/main/java/com/orctom/jenkins/plugin/globalpostscript/runner/GroovyScriptRunner.java
            src/main/java/com/orctom/jenkins/plugin/globalpostscript/runner/ScriptRunner.java
            src/main/java/com/orctom/jenkins/plugin/globalpostscript/runner/ScriptRunners.java
            src/main/java/com/orctom/jenkins/plugin/globalpostscript/runner/ShellScriptRunner.java
            src/test/java/com/orctom/jenkins/plugin/globalpostscript/ScriptTest.java
            src/test/java/com/orctom/jenkins/plugin/globalpostscript/URLTest.java
            http://jenkins-ci.org/commit/global-post-script-plugin/d94800cfb31723ed484b4ba98339deb2544be206
            Log:
            JENKINS-33358 use 'parse()' to archive a smaller footprint

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Hao Path: src/main/java/com/orctom/jenkins/plugin/globalpostscript/GlobalPostScript.java src/main/java/com/orctom/jenkins/plugin/globalpostscript/GlobalPostScriptAction.java src/main/java/com/orctom/jenkins/plugin/globalpostscript/GlobalPostScriptPlugin.java src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptContentLoader.java src/main/java/com/orctom/jenkins/plugin/globalpostscript/ScriptExecutor.java src/main/java/com/orctom/jenkins/plugin/globalpostscript/URL.java src/main/java/com/orctom/jenkins/plugin/globalpostscript/model/ScriptContent.java src/main/java/com/orctom/jenkins/plugin/globalpostscript/model/URL.java src/main/java/com/orctom/jenkins/plugin/globalpostscript/runner/GroovyScriptRunner.java src/main/java/com/orctom/jenkins/plugin/globalpostscript/runner/ScriptRunner.java src/main/java/com/orctom/jenkins/plugin/globalpostscript/runner/ScriptRunners.java src/main/java/com/orctom/jenkins/plugin/globalpostscript/runner/ShellScriptRunner.java src/test/java/com/orctom/jenkins/plugin/globalpostscript/ScriptTest.java src/test/java/com/orctom/jenkins/plugin/globalpostscript/URLTest.java http://jenkins-ci.org/commit/global-post-script-plugin/d94800cfb31723ed484b4ba98339deb2544be206 Log: JENKINS-33358 use 'parse()' to archive a smaller footprint
            orctom Hao CHEN added a comment -

            Can you try version 1.1.1 of this plugin, I parsed and cached the groovy script, parse() do save a lot of memory, hoping it could solve your issue.

            orctom Hao CHEN added a comment - Can you try version 1.1.1 of this plugin, I parsed and cached the groovy script, parse() do save a lot of memory, hoping it could solve your issue.
            orctom Hao CHEN made changes -
            Assignee Jesse Glick [ jglick ] Hongkai Liu [ hongkailiu ]
            hongkailiu Hongkai Liu added a comment -

            Thanks for the work.
            I will update here.

            hongkailiu Hongkai Liu added a comment - Thanks for the work. I will update here.
            hongkailiu Hongkai Liu added a comment - - edited

            Testing right now.
            I will know more soon.

            At mean time, I check a bit on the commit.
            It does a more careful parse.

            if (sc.isChanged()) {
                    script = shell.parse(sc.getContent());
                  }
            

            I wonder if this would be helpful for the following issue.
            The groovy is checked out from the src and it is copied to the Jenkins_home/global-post-script folder.

            Equivalent question:
            Is the command like shell.parse(sc.getContent()) for groovy a potential source of mem leak on PERGEN in general?

            Any hint on this is appreciated.

            hongkailiu Hongkai Liu added a comment - - edited Testing right now. I will know more soon. At mean time, I check a bit on the commit. It does a more careful parse. if (sc.isChanged()) { script = shell.parse(sc.getContent()); } I wonder if this would be helpful for the following issue. The groovy is checked out from the src and it is copied to the Jenkins_home/global-post-script folder. Equivalent question: Is the command like shell.parse(sc.getContent()) for groovy a potential source of mem leak on PERGEN in general? Any hint on this is appreciated.
            orctom Hao CHEN added a comment -

            Yes to both.

            isChanged() is calculated according to file.lastModified() of the current and previous run.

            jvisualvm indicates parse() does save a lot of memory, both Heap and Metaspace/PermGen.

            orctom Hao CHEN added a comment - Yes to both. isChanged() is calculated according to file.lastModified() of the current and previous run. jvisualvm indicates parse() does save a lot of memory, both Heap and Metaspace/PermGen.
            hongkailiu Hongkai Liu added a comment -

            Thanks for the quick update.
            If the groovy file is from src, then it could be updated all the time, and so is the lastModified() amd isChanged.
            In this case, I would assume that the problem comes back.

            hongkailiu Hongkai Liu added a comment - Thanks for the quick update. If the groovy file is from src, then it could be updated all the time, and so is the lastModified() amd isChanged. In this case, I would assume that the problem comes back.
            orctom Hao CHEN added a comment -

            Hold on, did you mean you have groovy script specific to each job, and have it copied to the global-post-script folder at the beginning of each run? Then I think you might want to use Groovy Plugin instead.

            orctom Hao CHEN added a comment - Hold on, did you mean you have groovy script specific to each job, and have it copied to the global-post-script folder at the beginning of each run? Then I think you might want to use Groovy Plugin instead.
            hongkailiu Hongkai Liu added a comment -

            Yes.
            That is the reason that I feel the leak is for groovy in general (not only for global post), unless Groovy Plugin loads and runs the script differently.

            hongkailiu Hongkai Liu added a comment - Yes. That is the reason that I feel the leak is for groovy in general (not only for global post), unless Groovy Plugin loads and runs the script differently.
            hongkailiu Hongkai Liu made changes -
            hongkailiu Hongkai Liu added a comment - - edited

            I confirmed that the fix in global post plugin improve the mem issue.

            The test is done with the previous setting (the groovy script file is UNCHANGED) and run for over 24 hours. See the screenshot.

            However, i still would like to have someone to confirm the groovy mem issue in general: Loading dynamically different groovy scripts leads to PerGen OOM.

            hongkailiu Hongkai Liu added a comment - - edited I confirmed that the fix in global post plugin improve the mem issue. The test is done with the previous setting (the groovy script file is UNCHANGED) and run for over 24 hours. See the screenshot. However, i still would like to have someone to confirm the groovy mem issue in general: Loading dynamically different groovy scripts leads to PerGen OOM.
            hongkailiu Hongkai Liu made changes -
            Assignee Hongkai Liu [ hongkailiu ] Hao CHEN [ orctom ]
            orctom Hao CHEN added a comment -

            I'm been trying to figure out if there're memory leaks with groovy, but little progress.
            I see a lot of posts claiming that there's no memory leaks in Groovy.
            But I do doubt it has classloader leaks, as I connected to the program with jvisualvm, and after the groovy script executed, I forced a GC and later dumped the memory. I'm still seeing lots of objects of groovy classes.

            The compiled groovy script, which is loaded by groovy.lang.GroovyClassLoader, is not get GC:ed, even the groovy.lang.GroovyClassLoader is not get GC:ed, so that everything loaded by groovy.lang.GroovyClassLoader will not be able to get GC:ed.

            I haven't looked into the source code, but I see a lot of issues related to 'leaks'.

            https://issues.apache.org/jira/browse/GROOVY-7683?jql=project%20%3D%20GROOVY%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20text%20~%20%22leak%22

            orctom Hao CHEN added a comment - I'm been trying to figure out if there're memory leaks with groovy, but little progress. I see a lot of posts claiming that there's no memory leaks in Groovy. But I do doubt it has classloader leaks, as I connected to the program with jvisualvm, and after the groovy script executed, I forced a GC and later dumped the memory. I'm still seeing lots of objects of groovy classes. The compiled groovy script, which is loaded by groovy.lang.GroovyClassLoader, is not get GC:ed, even the groovy.lang.GroovyClassLoader is not get GC:ed, so that everything loaded by groovy.lang.GroovyClassLoader will not be able to get GC:ed. I haven't looked into the source code, but I see a lot of issues related to 'leaks'. https://issues.apache.org/jira/browse/GROOVY-7683?jql=project%20%3D%20GROOVY%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20text%20~%20%22leak%22
            hongkailiu Hongkai Liu added a comment -

            Thanks for the update and sharing the concern that I have too.

            Unless there is a better way to load Groovy script, the memory leak seems unavoidable.

            hongkailiu Hongkai Liu added a comment - Thanks for the update and sharing the concern that I have too. Unless there is a better way to load Groovy script, the memory leak seems unavoidable.
            rebelmiles Miles Wilton added a comment -

            https://issues.apache.org/jira/browse/GROOVY-7731

            Try using -Dgroovy.use.classvalue=true

            There's an interesting blogpost here about the issue:
            http://melix.github.io/blog/2015/08/permgenleak.html

            rebelmiles Miles Wilton added a comment - https://issues.apache.org/jira/browse/GROOVY-7731 Try using -Dgroovy.use.classvalue=true There's an interesting blogpost here about the issue: http://melix.github.io/blog/2015/08/permgenleak.html
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 169271 ] JNJira + In-Review [ 183427 ]
            grayaii Alex Gray added a comment -

            I am also getting bit by this after I upgraded to jenkins 2.7.2 (from 1.651.1). I posted in the jenkins google groups here, but no responses yet:
            https://groups.google.com/forum/#!topic/jenkinsci-users/q7M2GJwArJ8

            I ran the scripts here: https://wiki.jenkins-ci.org/display/JENKINS/Monitoring+Scripts and I see a bazillion things like this (Display heap histogram (object instances per class)):

            sun.reflect.GeneratedMethodAccessor
            sun.reflect.GeneratedSerializationConstructorAccessor

            I will try doing this:
            -Dgroovy.use.classvalue=true

            To see if that helps.

            grayaii Alex Gray added a comment - I am also getting bit by this after I upgraded to jenkins 2.7.2 (from 1.651.1). I posted in the jenkins google groups here, but no responses yet: https://groups.google.com/forum/#!topic/jenkinsci-users/q7M2GJwArJ8 I ran the scripts here: https://wiki.jenkins-ci.org/display/JENKINS/Monitoring+Scripts and I see a bazillion things like this (Display heap histogram (object instances per class)): sun.reflect.GeneratedMethodAccessor sun.reflect.GeneratedSerializationConstructorAccessor I will try doing this: -Dgroovy.use.classvalue=true To see if that helps.
            grayaii Alex Gray added a comment -

            -Dgroovy.use.classvalue=true did the trick for me!
            My PermGen space is not increasing and everything seems to be working just fine.
            Thanks!

            grayaii Alex Gray added a comment - -Dgroovy.use.classvalue=true did the trick for me! My PermGen space is not increasing and everything seems to be working just fine. Thanks!
            danielbeck Daniel Beck added a comment -

            daspilker Ping.

            Could you track this upstream?

            Would it be a sensible approach to make the use.classvalue option the default in Jenkins?

            CC jglick

            danielbeck Daniel Beck added a comment - daspilker Ping. Could you track this upstream? Would it be a sensible approach to make the use.classvalue option the default in Jenkins? CC jglick
            daspilker Daniel Spilker made changes -
            Link This issue is related to JENKINS-37213 [ JENKINS-37213 ]

            danielbeck what do you mean by upstream? core?

            The relevant Groovy issue is GROOVY-7591. They added the groovy.use.classvalue flag with a the default set to false to disable a feature (introduced in 2.4.0) that causes a memory leak. So setting the flag to true will replace one memory leak by another one?!? I think this needs more investigation. Maybe jglick can shine some light on us, he is mentioned in the related JDK bug (https://bugs.openjdk.java.net/browse/JDK-8136353).

            daspilker Daniel Spilker added a comment - danielbeck what do you mean by upstream? core? The relevant Groovy issue is GROOVY-7591 . They added the groovy.use.classvalue flag with a the default set to false to disable a feature (introduced in 2.4.0) that causes a memory leak. So setting the flag to true will replace one memory leak by another one?!? I think this needs more investigation. Maybe jglick can shine some light on us, he is mentioned in the related JDK bug ( https://bugs.openjdk.java.net/browse/JDK-8136353 ).
            grayaii Alex Gray added a comment -

            Just a datapoint:
            Without this value (-Dgroovy.use.classvalue=true), my jenkins master would crash after 1-2 days (and the memory graph shows steady increase in permgen memory (with a steady increase in "classes count")).
            With this value enabled I do not see an increase in memory usage both in permgen and other memory footprints.
            Nonetheless, i'll let this bake for a few weeks to see if there are memory leaks related to enabling this option.

            grayaii Alex Gray added a comment - Just a datapoint: Without this value (-Dgroovy.use.classvalue=true), my jenkins master would crash after 1-2 days (and the memory graph shows steady increase in permgen memory (with a steady increase in "classes count")). With this value enabled I do not see an increase in memory usage both in permgen and other memory footprints. Nonetheless, i'll let this bake for a few weeks to see if there are memory leaks related to enabling this option.

            I am also encountering problems with my Jenkins system where the Jenkins process consumes
            so much memory that the operating system Out of Memory (OOM) killer kills the process.

            I'm making extensive use of Pipeline.

            My system is:

            {{ OS: FreeBSD (9.3-RELEASE-p43)
            Architecture: amd64 64bit
            Java Home: /usr/local/openjdk8/jre
            Java Version: 1.8.0_102
            JVM: OpenJDK 64-Bit Server VM (25.102-b14, mixed mode)
            Java Vendor: Oracle Corporation

            Jenkins version 2.18
            }}

            I grabbed a heapdump, which I have placed at:
            https://people.freebsd.org/~rodrigc/jenkins/heapdump/

            How do I tell if groovy is the source of the memory leak?

            rodrigc Craig Rodrigues added a comment - I am also encountering problems with my Jenkins system where the Jenkins process consumes so much memory that the operating system Out of Memory (OOM) killer kills the process. I'm making extensive use of Pipeline. My system is: {{ OS: FreeBSD (9.3-RELEASE-p43) Architecture: amd64 64bit Java Home: /usr/local/openjdk8/jre Java Version: 1.8.0_102 JVM: OpenJDK 64-Bit Server VM (25.102-b14, mixed mode) Java Vendor: Oracle Corporation Jenkins version 2.18 }} I grabbed a heapdump, which I have placed at: https://people.freebsd.org/~rodrigc/jenkins/heapdump/ How do I tell if groovy is the source of the memory leak?
            gmc_devel GMC Software Development B&R Corporate made changes -
            Link This issue relates to JENKINS-38334 [ JENKINS-38334 ]
            danielbeck Daniel Beck made changes -
            Link This issue is duplicated by JENKINS-38503 [ JENKINS-38503 ]
            jglick Jesse Glick added a comment -

            Pipeline includes its own workarounds for the host of memory leaks in Groovy code. (Do not use groovy.use.classvalue.) Please do not report Pipeline-related leaks in this issue, which should be reserved for other simpler calls to Groovy in Jenkins.

            jglick Jesse Glick added a comment - Pipeline includes its own workarounds for the host of memory leaks in Groovy code. (Do not use groovy.use.classvalue .) Please do not report Pipeline-related leaks in this issue, which should be reserved for other simpler calls to Groovy in Jenkins.
            bkosty_ea Brandon Kosty added a comment -

            Hey jglick,

            I'm dealing with memory leaks in groovy code in pipelines, can you link to the workaround you are talking about?

            thanks,
            -Brandon

            bkosty_ea Brandon Kosty added a comment - Hey jglick , I'm dealing with memory leaks in groovy code in pipelines, can you link to the workaround you are talking about? thanks, -Brandon
            jglick Jesse Glick added a comment - - edited

            bkosty_ea upgrade workflow-cps to 2.23, and please do not discuss further here.

            jglick Jesse Glick added a comment - - edited bkosty_ea upgrade workflow-cps to 2.23, and please do not discuss further here.
            ceilfors Wisen Tanasa made changes -
            Attachment before.png [ 34830 ]
            ceilfors Wisen Tanasa made changes -
            Attachment after.png [ 34831 ]
            ceilfors Wisen Tanasa added a comment -

            We are faced with the same issue after upgrading Jenkins to 2.7.4. In our scenario, it is caused by job-dsl-plugin (v1.50) that we schedule to run every 15 minutes. Setting JAVA_OPTS -Dgroovy.use.classvalue=true to tomcat helps.

            Before setting -Dgroovy.use.classvalue=true

            After setting -Dgroovy.use.classvalue=true

            ceilfors Wisen Tanasa added a comment - We are faced with the same issue after upgrading Jenkins to 2.7.4. In our scenario, it is caused by job-dsl-plugin (v1.50) that we schedule to run every 15 minutes. Setting JAVA_OPTS -Dgroovy.use.classvalue=true to tomcat helps. Before setting -Dgroovy.use.classvalue=true After setting -Dgroovy.use.classvalue=true
            dorzalk Dor Zalkind added a comment -

            Seeing the same issue on Jenkins 2.19.1, SLES11SP4, Java 1.8.0.92. We use job-dsl-plugin (v1.51) and need to restart our server every couple of weeks to avoid it crashing.

            dorzalk Dor Zalkind added a comment - Seeing the same issue on Jenkins 2.19.1, SLES11SP4, Java 1.8.0.92. We use job-dsl-plugin (v1.51) and need to restart our server every couple of weeks to avoid it crashing.
            vladichko Vlad Aginsky added a comment -

            Hi all, is there some sort of "garbage collector" that can be used to clean all "leaked" memory?

            vladichko Vlad Aginsky added a comment - Hi all, is there some sort of "garbage collector" that can be used to clean all "leaked" memory?

            Jenkins 2.19.4 uses Groovy 2.4.7 that contains a memory leak: https://issues.apache.org/jira/browse/GROOVY-7683
            It will be fixed in Groovy 2.4.8 (not release yet).

            edebanne Emmanuel Debanne added a comment - Jenkins 2.19.4 uses Groovy 2.4.7 that contains a memory leak: https://issues.apache.org/jira/browse/GROOVY-7683 It will be fixed in Groovy 2.4.8 (not release yet).
            vladichko Vlad Aginsky added a comment -

            I see that new groovy was released http://groovy-lang.org/changelogs/changelog-2.4.8.html
            and indeed it comes with a fix.
            I am not sure what need to be done to update my jenkins to make use of it, any idea from respectful forum??

            vladichko Vlad Aginsky added a comment - I see that new groovy was released http://groovy-lang.org/changelogs/changelog-2.4.8.html and indeed it comes with a fix. I am not sure what need to be done to update my jenkins to make use of it, any idea from respectful forum??
            danielbeck Daniel Beck added a comment -

            what need to be done to update my jenkins to make use of it, any idea from respectful forum

            Already on the way: https://github.com/jenkinsci/jenkins/pull/2715

            (Please refrain from adding comments that basically just "vote" for its inclusion. We know this is important.)

            danielbeck Daniel Beck added a comment - what need to be done to update my jenkins to make use of it, any idea from respectful forum Already on the way: https://github.com/jenkinsci/jenkins/pull/2715 (Please refrain from adding comments that basically just "vote" for its inclusion. We know this is important.)
            vladichko Vlad Aginsky added a comment -

            I see
            "oleg-nenashev approved these changes 20 hours ago"
            Are we expected to see in in next version(2.42) before LTS?

            vladichko Vlad Aginsky added a comment - I see "oleg-nenashev approved these changes 20 hours ago" Are we expected to see in in next version(2.42) before LTS?
            vladichko Vlad Aginsky added a comment -

            Unfortunately this weekly build doesn't bring new groovy version:
            https://jenkins.io/changelog/
            What's new in 2.42 (2017/01/22)
            • IllegalStateException from Winstone when making certain requests with access logging enabled. (issue 37625)

            vladichko Vlad Aginsky added a comment - Unfortunately this weekly build doesn't bring new groovy version: https://jenkins.io/changelog/ What's new in 2.42 (2017/01/22) • IllegalStateException from Winstone when making certain requests with access logging enabled. (issue 37625)
            batmat Baptiste Mathus added a comment - - edited

            vladichko approved does not mean merged. It just means that some people are OK to merge it, maybe, some day. Once it is merged, it will take a week at worse to get included in a weekly, and ~3+ months to get into the LTS depending on many criteria (or less if judged critically important AND not too risky by the Release Officer).

            batmat Baptiste Mathus added a comment - - edited vladichko approved does not mean merged. It just means that some people are OK to merge it, maybe, some day. Once it is merged, it will take a week at worse to get included in a weekly, and ~3+ months to get into the LTS depending on many criteria (or less if judged critically important AND not too risky by the Release Officer).

            Code changed in jenkins
            User: Daniel Spilker
            Path:
            core/pom.xml
            http://jenkins-ci.org/commit/jenkins/5c8c64be06a804ef24805e4db31efbf5963fc66c
            Log:
            updated Groovy to 2.4.8

            [FIXES JENKINS-33358]

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Spilker Path: core/pom.xml http://jenkins-ci.org/commit/jenkins/5c8c64be06a804ef24805e4db31efbf5963fc66c Log: updated Groovy to 2.4.8 [FIXES JENKINS-33358]
            scm_issue_link SCM/JIRA link daemon made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            core/pom.xml
            http://jenkins-ci.org/commit/jenkins/edf7b6b977cc9186da0d3bb5fb58d068bb35c0b9
            Log:
            Merge pull request #2715 from daspilker/JENKINS-33358

            JENKINS-33358 update Groovy to 2.4.8

            Compare: https://github.com/jenkinsci/jenkins/compare/83232e6b8c8f...edf7b6b977cc

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/pom.xml http://jenkins-ci.org/commit/jenkins/edf7b6b977cc9186da0d3bb5fb58d068bb35c0b9 Log: Merge pull request #2715 from daspilker/ JENKINS-33358 JENKINS-33358 update Groovy to 2.4.8 Compare: https://github.com/jenkinsci/jenkins/compare/83232e6b8c8f...edf7b6b977cc
            oleg_nenashev Oleg Nenashev added a comment -

            The PR has been released in 2.47

            oleg_nenashev Oleg Nenashev added a comment - The PR has been released in 2.47
            jglick Jesse Glick made changes -
            Link This issue is blocked by JENKINS-42189 [ JENKINS-42189 ]

            When will this fix be available in LTS?

            christer_engde Christer Engde added a comment - When will this fix be available in LTS?
            danielbeck Daniel Beck added a comment -

            We decided today that the next LTS will be based on 2.46 due to the risk of further regressions/incompatibilities caused by the upgrade to Groovy 2.4.8.

            So this will likely only be in LTS 16 weeks from today unless we think it's safe enough for a backport (and I would be surprised if we'd consider it safe enough).

            danielbeck Daniel Beck added a comment - We decided today that the next LTS will be based on 2.46 due to the risk of further regressions/incompatibilities caused by the upgrade to Groovy 2.4.8. So this will likely only be in LTS 16 weeks from today unless we think it's safe enough for a backport (and I would be surprised if we'd consider it safe enough).
            danielbeck Daniel Beck made changes -
            Labels groovy memory-leak groovy lts-candidate memory-leak
            danielbeck Daniel Beck added a comment -

            Labeling just in case we do think it's safe enough (but don't get your hopes up).

            danielbeck Daniel Beck added a comment - Labeling just in case we do think it's safe enough (but don't get your hopes up).
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is related to JENKINS-42637 [ JENKINS-42637 ]
            olivergondza Oliver Gondža made changes -
            Labels groovy lts-candidate memory-leak 2.46.1-rejected groovy lts-candidate memory-leak

            I am rejecting this from 2.46.1 as agreed. There is first issue that is suspected to be caused by this change: JENKINS-42637.

            olivergondza Oliver Gondža added a comment - I am rejecting this from 2.46.1 as agreed. There is first issue that is suspected to be caused by this change: JENKINS-42637 .

            Reporting successful fix with the new groovy version.

            We updated an affected instance on Java 8, LTS version 2.7.4 to the non-LTS 2.48 and have been running the server for a week. The memory is under control now.

            +1 in bringing this into LTS

            sbeaulie Samuel Beaulieu added a comment - Reporting successful fix with the new groovy version. We updated an affected instance on Java 8, LTS version 2.7.4 to the non-LTS 2.48 and have been running the server for a week. The memory is under control now. +1 in bringing this into LTS
            ympaul2 paul young made changes -
            Assignee Hao CHEN [ orctom ] paul young [ ympaul2 ]
            olivergondza Oliver Gondža made changes -
            Labels 2.46.1-rejected groovy lts-candidate memory-leak 2.46.1-rejected 2.46.2-rejected groovy lts-candidate memory-leak
            olivergondza Oliver Gondža made changes -
            Labels 2.46.1-rejected 2.46.2-rejected groovy lts-candidate memory-leak 2.46.1-rejected 2.46.2-rejected 2.46.3-rejected groovy memory-leak
            danielbeck Daniel Beck added a comment -

            2.73.1 is scheduled for release this week and will contain Groovy 2.4.11.

            danielbeck Daniel Beck added a comment - 2.73.1 is scheduled for release this week and will contain Groovy 2.4.11.
            jglick Jesse Glick added a comment -

            In 2.61 apparently.

            jglick Jesse Glick added a comment - In 2.61 apparently.

            Code changed in jenkins
            User: Jesse Glick
            Path:
            Jenkinsfile
            http://jenkins-ci.org/commit/workflow-cps-plugin/9c5667d59d47d4bff4e3dfd5f68ea496448029e1
            Log:
            JENKINS-33358 Test against 2.73.x since that includes Groovy 2.4.11.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: Jenkinsfile http://jenkins-ci.org/commit/workflow-cps-plugin/9c5667d59d47d4bff4e3dfd5f68ea496448029e1 Log: JENKINS-33358 Test against 2.73.x since that includes Groovy 2.4.11.

            Code changed in jenkins
            User: Sam Van Oort
            Path:
            Jenkinsfile
            pom.xml
            http://jenkins-ci.org/commit/workflow-cps-plugin/a5dea90a589f094caf58411c271f139322f37f2f
            Log:
            Merge pull request #175 from jglick/new-Groovy-JENKINS-33358

            JENKINS-33358 Test against 2.73.x since that includes Groovy 2.4.11

            Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/3396c9715894...a5dea90a589f

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Sam Van Oort Path: Jenkinsfile pom.xml http://jenkins-ci.org/commit/workflow-cps-plugin/a5dea90a589f094caf58411c271f139322f37f2f Log: Merge pull request #175 from jglick/new-Groovy- JENKINS-33358 JENKINS-33358 Test against 2.73.x since that includes Groovy 2.4.11 Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/3396c9715894...a5dea90a589f
            svanoort Sam Van Oort added a comment -

            https://issues.jenkins-ci.org/browse/JENKINS-47758 might offer us a way to guard against some of the memory leaks. 

            svanoort Sam Van Oort added a comment - https://issues.jenkins-ci.org/browse/JENKINS-47758  might offer us a way to guard against some of the memory leaks. 
            jamesdumay James Dumay made changes -
            Remote Link This issue links to "CloudBees Internal OSS-2127 (Web Link)" [ 18410 ]
            jamesdumay James Dumay made changes -
            Remote Link This issue links to "CloudBees Internal OSS-2079 (Web Link)" [ 18429 ]
            dehrk Derek Kinney made changes -
            Link This issue relates to JENKINS-38334 [ JENKINS-38334 ]
            dehrk Derek Kinney made changes -
            Link This issue relates to JENKINS-38334 [ JENKINS-38334 ]

            People

              ympaul2 paul young
              hongkailiu Hongkai Liu
              Votes:
              23 Vote for this issue
              Watchers:
              53 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: