• Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Critical Critical
    • core

      Jenkins 2.49 Don't know what is actually the problem. Memory dump available for in-house usage via Dropbox etc. if needed. Please send me an email.

       

      Seems to be related to https://issues.apache.org/jira/browse/GROOVY-8067 Groovy bug. Don't know when Jenkins groovy will be update from 2.4.8 to 2.4.9 version .

          [JENKINS-42637] Jenkins stuck UI does not open at all

          Oleg Nenashev added a comment -

          From what I see it happens in the Groovy class management logic. So my guess is that it is just another follow-up to the Groovy update to 2.4.8 in 2.47 (for JENKINS-33358).

          The pattern is close to JENKINS-42189

          CC daspilker and jglick

          Oleg Nenashev added a comment - From what I see it happens in the Groovy class management logic. So my guess is that it is just another follow-up to the Groovy update to 2.4.8 in 2.47 (for JENKINS-33358 ). The pattern is close to JENKINS-42189 CC daspilker and jglick

          Jesse Glick added a comment -

          No idea offhand. Reproducible from scratch?

          Jesse Glick added a comment - No idea offhand. Reproducible from scratch?

          Our Lead CI developer Timo thinks that Groovy 2.49 will help because of this fix https://issues.apache.org/jira/browse/GROOVY-8067 

          Heikki Simperi added a comment - Our Lead CI developer Timo thinks that Groovy 2.49 will help because of this fix https://issues.apache.org/jira/browse/GROOVY-8067  

          This was found from thread dump:

          "Executor #37 for xxx.xxxx.xxx mini : executing xxxx Healthcheck #4921725" Id=580323 Group=main BLOCKED on org.codehaus.groovy.util.ManagedLinkedList@3fe823dd owned by "Executor #28 for xxxx.xxxx.xxx mini : executing ETJ2 Healthcheck #4921723" Id=580319
                                       at org.codehaus.groovy.reflection.ClassInfo$GlobalClassSet.add(ClassInfo.java:477)
                                       -  blocked on org.codehaus.groovy.util.ManagedLinkedList@3fe823dd
                                       at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:83)
                                       at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:79)
                                       at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue.<init>(GroovyClassValuePreJava7.java:37)
                                       at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment.createEntry(GroovyClassValuePreJava7.java:64)
                                       at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment.createEntry(GroovyClassValuePreJava7.java:55)
                                       at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.put(AbstractConcurrentMap.java:157)
                                       at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.getOrPut(AbstractConcurrentMap.java:100)
                                       at org.codehaus.groovy.util.AbstractConcurrentMap.getOrPut(AbstractConcurrentMap.java:38)
                                       at org.codehaus.groovy.reflection.GroovyClassValuePreJava7.get(GroovyClassValuePreJava7.java:94)
                                       at org.codehaus.groovy.reflection.ClassInfo.getClassInfo(ClassInfo.java:144)
                                       at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:258)
                                       at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:883)
                                       at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:34)
                                       at groovy.lang.Script.<init>(Script.java:42)
                                       at Script1.<init>(Script1.groovy)
                                       at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                                       at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
                                       at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                                       at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
                                       at org.codehaus.groovy.runtime.InvokerHelper.createScript(InvokerHelper.java:431)
                                       at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
                                       at groovy.lang.GroovyShell.evaluate(GroovyShell.java:584)
                                       at groovy.lang.GroovyShell.evaluate(GroovyShell.java:623)
                                       at groovy.lang.GroovyShell.evaluate(GroovyShell.java:594)
                                       at org.jenkinsci.plugins.envinject.service.EnvInjectEnvVars.executeAndGetMapGroovyScript(EnvInjectEnvVars.java:153)
                                       at org.jenkinsci.plugins.envinject.EnvInjectBuildWrapper.setUp(EnvInjectBuildWrapper.java:101)
                                       at hudson.model.Build$BuildExecution.doRun(Build.java:157)
                                       at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
                                       at hudson.model.Run.execute(Run.java:1728)
                                       at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                                       at hudson.model.ResourceController.execute(ResourceController.java:98)
                                       at hudson.model.Executor.run(Executor.java:405)

          Heikki Simperi added a comment - This was found from thread dump: "Executor #37 for xxx.xxxx.xxx mini : executing xxxx Healthcheck #4921725" Id=580323 Group=main BLOCKED on org.codehaus.groovy.util.ManagedLinkedList@3fe823dd owned by "Executor #28 for xxxx.xxxx.xxx mini : executing ETJ2 Healthcheck #4921723" Id=580319                              at org.codehaus.groovy.reflection.ClassInfo$GlobalClassSet.add(ClassInfo.java:477)                              -  blocked on org.codehaus.groovy.util.ManagedLinkedList@3fe823dd                              at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:83)                              at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:79)                              at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue.<init>(GroovyClassValuePreJava7.java:37)                              at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment.createEntry(GroovyClassValuePreJava7.java:64)                              at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment.createEntry(GroovyClassValuePreJava7.java:55)                              at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.put(AbstractConcurrentMap.java:157)                              at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.getOrPut(AbstractConcurrentMap.java:100)                              at org.codehaus.groovy.util.AbstractConcurrentMap.getOrPut(AbstractConcurrentMap.java:38)                              at org.codehaus.groovy.reflection.GroovyClassValuePreJava7.get(GroovyClassValuePreJava7.java:94)                              at org.codehaus.groovy.reflection.ClassInfo.getClassInfo(ClassInfo.java:144)                              at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:258)                              at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:883)                              at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:34)                              at groovy.lang.Script.<init>(Script.java:42)                              at Script1.<init>(Script1.groovy)                              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)                              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)                              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)                              at java.lang.reflect.Constructor.newInstance(Constructor.java:423)                              at org.codehaus.groovy.runtime.InvokerHelper.createScript(InvokerHelper.java:431)                              at groovy.lang.GroovyShell.parse(GroovyShell.java:700)                              at groovy.lang.GroovyShell.evaluate(GroovyShell.java:584)                              at groovy.lang.GroovyShell.evaluate(GroovyShell.java:623)                              at groovy.lang.GroovyShell.evaluate(GroovyShell.java:594)                              at org.jenkinsci.plugins.envinject.service.EnvInjectEnvVars.executeAndGetMapGroovyScript(EnvInjectEnvVars.java:153)                              at org.jenkinsci.plugins.envinject.EnvInjectBuildWrapper.setUp(EnvInjectBuildWrapper.java:101)                              at hudson.model.Build$BuildExecution.doRun(Build.java:157)                              at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)                              at hudson.model.Run.execute(Run.java:1728)                              at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)                              at hudson.model.ResourceController.execute(ResourceController.java:98)                              at hudson.model.Executor.run(Executor.java:405)

          Updated to 2.55 and still no groovy 2.4.9? At least we still face the same groovy problem 5-10 times per day and only fix is to restart jenkins.

          Heikki Simperi added a comment - Updated to 2.55 and still no groovy 2.4.9? At least we still face the same groovy problem 5-10 times per day and only fix is to restart jenkins.

          Sam Van Oort added a comment -

          heikkisi  We understand this is a critical issue for you, and in order to solve it, can you provide an isolated example that will reproduce this without depending on your infra?    

          I think the reason we're hesitant to update Groovy more is that every update seems to introduce a new issue of this sort, which generally requires a significant time investment from someone deeply specialized in this area. 

           

          Sam Van Oort added a comment - heikkisi   We understand this is a critical issue for you, and in order to solve it, can you provide an isolated example that will reproduce this without depending on your infra?     I think the reason we're hesitant to update Groovy more is that every update seems to introduce a new issue of this sort, which generally requires a significant time investment from someone deeply specialized in this area.   

          svanoort I will look into this. Groovy bug has java class to reproduce the problem in groovy but I understand that it is not quite relevant.

          Is there way to give bundled groovy command line parameter as suggested as fast fix?

           

          -Dgroovy.use.classvalue=true 

          Heikki Simperi added a comment - svanoort I will look into this. Groovy bug has java class to reproduce the problem in groovy but I understand that it is not quite relevant. Is there way to give bundled groovy command line parameter as suggested as fast fix?   -Dgroovy.use.classvalue=true 

          Alex Gray added a comment -

           

          Not sure if this will help, but in our /etc/sysconfig/jenkins you can added:

          JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Dgroovy.use.classvalue=true -Xms4096m -Xmx4096m -XX:MaxPermSize=1024m -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -Dhudson.model.ParametersAction.keepUndefinedParameters=true"
          

          We also run some crazy groovy code in some of jenkins jobs and we tend to run out of memory too, so we've installed this plugin to help us track java resources:

           

          https://wiki.jenkins-ci.org/display/JENKINS/Monitoring

          And finally, we periodically (once per hour i think) run this groovy script to clean things up:

          import net.bull.javamelody.*;
          
          before = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();
          System.gc(); 
          after = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();
          println I18N.getFormattedString("ramasse_miette_execute", Math.round((before - after) / 1024));
          
          

          And we launch it like this:

          java -jar jenkins-cli.jar -noCertificateCheck -i id_rsa -s JENKINS_URL groovy my_groovy_script.groovy'
          

          You can probably put that groovy script as a jenkins job have it run periodically.

          Ever since we we run that System.gc() command, we've never run out of memory.  We run 100s of jobs a day a AWS t2.medium without any downtime for months at a time.  Before I did these things, we were running on a huge instance and we had to restart it every week or so.

          Hope this helps!

           

          Alex Gray added a comment -   Not sure if this will help, but in our /etc/sysconfig/jenkins you can added: JENKINS_JAVA_OPTIONS= "-Djava.awt.headless= true -Dgroovy.use.classvalue= true -Xms4096m -Xmx4096m -XX:MaxPermSize=1024m -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -Dhudson.model.ParametersAction.keepUndefinedParameters= true " We also run some crazy groovy code in some of jenkins jobs and we tend to run out of memory too, so we've installed this plugin to help us track java resources:   https://wiki.jenkins-ci.org/display/JENKINS/Monitoring And finally, we periodically (once per hour i think) run this groovy script to clean things up: import net.bull.javamelody.*; before = Runtime .getRuntime().totalMemory() - Runtime .getRuntime().freeMemory(); System .gc(); after = Runtime .getRuntime().totalMemory() - Runtime .getRuntime().freeMemory(); println I18N.getFormattedString( "ramasse_miette_execute" , Math .round((before - after) / 1024)); And we launch it like this: java -jar jenkins-cli.jar -noCertificateCheck -i id_rsa -s JENKINS_URL groovy my_groovy_script.groovy' You can probably put that groovy script as a jenkins job have it run periodically. Ever since we we run that System.gc() command, we've never run out of memory.  We run 100s of jobs a day a AWS t2.medium without any downtime for months at a time.  Before I did these things, we were running on a huge instance and we had to restart it every week or so. Hope this helps!  

          Hello!

          We could not make reproducible test case with small effort.

          However, the parameter tuning from Groovy ticket seem to resolve the state. Now Jenkins has been working for one whole week without problems!

           

          Heikki Simperi added a comment - Hello! We could not make reproducible test case with small effort. However, the parameter tuning from Groovy ticket seem to resolve the state. Now Jenkins has been working for one whole week without problems!  

          Dane Kantner added a comment -

          From another bug where it was mentioned,  -Dgroovy.use.classvalue=true  seems to have fixed this for us as well. We actually don't use anything crazy at all in Groovy (I use it to read load a 3 line text file into a parameter list and there are try/catch blocks around that read operation even to fail to a default "" response), nonetheless the bugs in this version brought the entire platform to a state of frequent unavailability.  There are other bugs open related to this but it's hard to picture how this is "resolved" when it's such a major flaw that has already been fixed in groovy itself, though Jenkins still needs to upgrade to using that newer version internally.

          Dane Kantner added a comment - From another bug where it was mentioned,  -Dgroovy.use.classvalue=true  seems to have fixed this for us as well. We actually don't use anything crazy at all in Groovy (I use it to read load a 3 line text file into a parameter list and there are try/catch blocks around that read operation even to fail to a default "" response), nonetheless the bugs in this version brought the entire platform to a state of frequent unavailability.  There are other bugs open related to this but it's hard to picture how this is "resolved" when it's such a major flaw that has already been fixed in groovy itself, though Jenkins still needs to upgrade to using that newer version internally.

            Unassigned Unassigned
            heikkisi Heikki Simperi
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: