Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42637

Jenkins stuck UI does not open at all

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Jenkins 2.49 Don't know what is actually the problem. Memory dump available for in-house usage via Dropbox etc. if needed. Please send me an email.

       

      Seems to be related to https://issues.apache.org/jira/browse/GROOVY-8067 Groovy bug. Don't know when Jenkins groovy will be update from 2.4.8 to 2.4.9 version .

        Attachments

          Issue Links

            Activity

            heikkisi Heikki Simperi created issue -
            heikkisi Heikki Simperi made changes -
            Field Original Value New Value
            Attachment jenkins_stuck.log [ 36366 ]
            Description Jenkins 2.49 Don't know what is actually the problem. Jenkins 2.49 Don't know what is actually the problem


            Memory dump available for in-house usage via Dropbox etc. if needed. Please send me an email.
            Hide
            heikkisi Heikki Simperi added a comment -

            Memory dump might be partly or broken:

             

            /usr/java/latest_jdk8/bin/jmap -F -dump:format=b,file=jenkins_stuck.hprof 21247
            Attaching to process ID 21247, please wait...
            Debugger attached successfully.
            Server compiler detected.
            JVM version is 25.45-b02
            Dumping heap to jenkins_stuck.hprof ...

            Exception in thread "main" java.lang.reflect.InvocationTargetException
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:497)
            at sun.tools.jmap.JMap.runTool(JMap.java:201)
            at sun.tools.jmap.JMap.main(JMap.java:130)
            Caused by: sun.jvm.hotspot.utilities.AssertionFailure: can not get class data for java/lang/UNIXProcess$$Lambda$50x00000007c0119428
            at sun.jvm.hotspot.utilities.Assert.that(Assert.java:32)
            at sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeInstance(HeapHprofBinWriter.java:803)
            at sun.jvm.hotspot.utilities.AbstractHeapGraphWriter$1.doObj(AbstractHeapGraphWriter.java:95)
            at sun.jvm.hotspot.oops.ObjectHeap.iterateLiveRegions(ObjectHeap.java:353)
            at sun.jvm.hotspot.oops.ObjectHeap.iterate(ObjectHeap.java:171)
            at sun.jvm.hotspot.utilities.AbstractHeapGraphWriter.write(AbstractHeapGraphWriter.java:51)
            at sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:433)
            at sun.jvm.hotspot.tools.HeapDumper.run(HeapDumper.java:62)
            at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
            at sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
            at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
            at sun.jvm.hotspot.tools.HeapDumper.main(HeapDumper.java:83)
            ... 6 more

            Show
            heikkisi Heikki Simperi added a comment - Memory dump might be partly or broken:   /usr/java/latest_jdk8/bin/jmap -F -dump:format=b,file=jenkins_stuck.hprof 21247 Attaching to process ID 21247, please wait... Debugger attached successfully. Server compiler detected. JVM version is 25.45-b02 Dumping heap to jenkins_stuck.hprof ... Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at sun.tools.jmap.JMap.runTool(JMap.java:201) at sun.tools.jmap.JMap.main(JMap.java:130) Caused by: sun.jvm.hotspot.utilities.AssertionFailure: can not get class data for java/lang/UNIXProcess$$Lambda$50x00000007c0119428 at sun.jvm.hotspot.utilities.Assert.that(Assert.java:32) at sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeInstance(HeapHprofBinWriter.java:803) at sun.jvm.hotspot.utilities.AbstractHeapGraphWriter$1.doObj(AbstractHeapGraphWriter.java:95) at sun.jvm.hotspot.oops.ObjectHeap.iterateLiveRegions(ObjectHeap.java:353) at sun.jvm.hotspot.oops.ObjectHeap.iterate(ObjectHeap.java:171) at sun.jvm.hotspot.utilities.AbstractHeapGraphWriter.write(AbstractHeapGraphWriter.java:51) at sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:433) at sun.jvm.hotspot.tools.HeapDumper.run(HeapDumper.java:62) at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260) at sun.jvm.hotspot.tools.Tool.start(Tool.java:223) at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118) at sun.jvm.hotspot.tools.HeapDumper.main(HeapDumper.java:83) ... 6 more
            heikkisi Heikki Simperi made changes -
            Attachment jenkins_stuck2.log [ 36376 ]
            Hide
            heikkisi Heikki Simperi added a comment -

            Added second thread dump. The problem occures wery often, few times per day.

            Show
            heikkisi Heikki Simperi added a comment - Added second thread dump. The problem occures wery often, few times per day.
            heikkisi Heikki Simperi made changes -
            Component/s groovy-plugin [ 15549 ]
            Hide
            heikkisi Heikki Simperi added a comment -

            I'm not sure if the problem is actual Critical level issue.

            Show
            heikkisi Heikki Simperi added a comment - I'm not sure if the problem is actual Critical level issue.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            From what I see it happens in the Groovy class management logic. So my guess is that it is just another follow-up to the Groovy update to 2.4.8 in 2.47 (for JENKINS-33358).

            The pattern is close to JENKINS-42189

            CC Daniel Spilker and Jesse Glick

            Show
            oleg_nenashev Oleg Nenashev added a comment - From what I see it happens in the Groovy class management logic. So my guess is that it is just another follow-up to the Groovy update to 2.4.8 in 2.47 (for JENKINS-33358 ). The pattern is close to JENKINS-42189 CC Daniel Spilker and Jesse Glick
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is related to JENKINS-33358 [ JENKINS-33358 ]
            oleg_nenashev Oleg Nenashev made changes -
            Labels groovy regression
            oleg_nenashev Oleg Nenashev made changes -
            Component/s groovy-plugin [ 15549 ]
            Hide
            jglick Jesse Glick added a comment -

            No idea offhand. Reproducible from scratch?

            Show
            jglick Jesse Glick added a comment - No idea offhand. Reproducible from scratch?
            Hide
            heikkisi Heikki Simperi added a comment -

            Our Lead CI developer Timo thinks that Groovy 2.49 will help because of this fix https://issues.apache.org/jira/browse/GROOVY-8067 

            Show
            heikkisi Heikki Simperi added a comment - Our Lead CI developer Timo thinks that Groovy 2.49 will help because of this fix https://issues.apache.org/jira/browse/GROOVY-8067  
            Hide
            heikkisi Heikki Simperi added a comment -

            This was found from thread dump:

            "Executor #37 for xxx.xxxx.xxx mini : executing xxxx Healthcheck #4921725" Id=580323 Group=main BLOCKED on org.codehaus.groovy.util.ManagedLinkedList@3fe823dd owned by "Executor #28 for xxxx.xxxx.xxx mini : executing ETJ2 Healthcheck #4921723" Id=580319
                                         at org.codehaus.groovy.reflection.ClassInfo$GlobalClassSet.add(ClassInfo.java:477)
                                         -  blocked on org.codehaus.groovy.util.ManagedLinkedList@3fe823dd
                                         at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:83)
                                         at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:79)
                                         at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue.<init>(GroovyClassValuePreJava7.java:37)
                                         at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment.createEntry(GroovyClassValuePreJava7.java:64)
                                         at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment.createEntry(GroovyClassValuePreJava7.java:55)
                                         at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.put(AbstractConcurrentMap.java:157)
                                         at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.getOrPut(AbstractConcurrentMap.java:100)
                                         at org.codehaus.groovy.util.AbstractConcurrentMap.getOrPut(AbstractConcurrentMap.java:38)
                                         at org.codehaus.groovy.reflection.GroovyClassValuePreJava7.get(GroovyClassValuePreJava7.java:94)
                                         at org.codehaus.groovy.reflection.ClassInfo.getClassInfo(ClassInfo.java:144)
                                         at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:258)
                                         at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:883)
                                         at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:34)
                                         at groovy.lang.Script.<init>(Script.java:42)
                                         at Script1.<init>(Script1.groovy)
                                         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                                         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
                                         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                                         at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
                                         at org.codehaus.groovy.runtime.InvokerHelper.createScript(InvokerHelper.java:431)
                                         at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
                                         at groovy.lang.GroovyShell.evaluate(GroovyShell.java:584)
                                         at groovy.lang.GroovyShell.evaluate(GroovyShell.java:623)
                                         at groovy.lang.GroovyShell.evaluate(GroovyShell.java:594)
                                         at org.jenkinsci.plugins.envinject.service.EnvInjectEnvVars.executeAndGetMapGroovyScript(EnvInjectEnvVars.java:153)
                                         at org.jenkinsci.plugins.envinject.EnvInjectBuildWrapper.setUp(EnvInjectBuildWrapper.java:101)
                                         at hudson.model.Build$BuildExecution.doRun(Build.java:157)
                                         at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
                                         at hudson.model.Run.execute(Run.java:1728)
                                         at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                                         at hudson.model.ResourceController.execute(ResourceController.java:98)
                                         at hudson.model.Executor.run(Executor.java:405)
            Show
            heikkisi Heikki Simperi added a comment - This was found from thread dump: "Executor #37 for xxx.xxxx.xxx mini : executing xxxx Healthcheck #4921725" Id=580323 Group=main BLOCKED on org.codehaus.groovy.util.ManagedLinkedList@3fe823dd owned by "Executor #28 for xxxx.xxxx.xxx mini : executing ETJ2 Healthcheck #4921723" Id=580319                              at org.codehaus.groovy.reflection.ClassInfo$GlobalClassSet.add(ClassInfo.java:477)                              -  blocked on org.codehaus.groovy.util.ManagedLinkedList@3fe823dd                              at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:83)                              at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:79)                              at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue.<init>(GroovyClassValuePreJava7.java:37)                              at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment.createEntry(GroovyClassValuePreJava7.java:64)                              at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment.createEntry(GroovyClassValuePreJava7.java:55)                              at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.put(AbstractConcurrentMap.java:157)                              at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.getOrPut(AbstractConcurrentMap.java:100)                              at org.codehaus.groovy.util.AbstractConcurrentMap.getOrPut(AbstractConcurrentMap.java:38)                              at org.codehaus.groovy.reflection.GroovyClassValuePreJava7.get(GroovyClassValuePreJava7.java:94)                              at org.codehaus.groovy.reflection.ClassInfo.getClassInfo(ClassInfo.java:144)                              at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:258)                              at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:883)                              at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:34)                              at groovy.lang.Script.<init>(Script.java:42)                              at Script1.<init>(Script1.groovy)                              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)                              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)                              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)                              at java.lang.reflect.Constructor.newInstance(Constructor.java:423)                              at org.codehaus.groovy.runtime.InvokerHelper.createScript(InvokerHelper.java:431)                              at groovy.lang.GroovyShell.parse(GroovyShell.java:700)                              at groovy.lang.GroovyShell.evaluate(GroovyShell.java:584)                              at groovy.lang.GroovyShell.evaluate(GroovyShell.java:623)                              at groovy.lang.GroovyShell.evaluate(GroovyShell.java:594)                              at org.jenkinsci.plugins.envinject.service.EnvInjectEnvVars.executeAndGetMapGroovyScript(EnvInjectEnvVars.java:153)                              at org.jenkinsci.plugins.envinject.EnvInjectBuildWrapper.setUp(EnvInjectBuildWrapper.java:101)                              at hudson.model.Build$BuildExecution.doRun(Build.java:157)                              at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)                              at hudson.model.Run.execute(Run.java:1728)                              at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)                              at hudson.model.ResourceController.execute(ResourceController.java:98)                              at hudson.model.Executor.run(Executor.java:405)
            heikkisi Heikki Simperi made changes -
            Description Jenkins 2.49 Don't know what is actually the problem


            Memory dump available for in-house usage via Dropbox etc. if needed. Please send me an email.
            Jenkins 2.49 Don't know what is actually the problem. Memory dump available for in-house usage via Dropbox etc. if needed. Please send me an email.

             

            Seems to be related to [https://issues.apache.org/jira/browse/GROOVY-8067] Groovy bug. Don't know when Jenkins groovy plugin will be update from 2.4.8 to 2.4.9 version .
            heikkisi Heikki Simperi made changes -
            Description Jenkins 2.49 Don't know what is actually the problem. Memory dump available for in-house usage via Dropbox etc. if needed. Please send me an email.

             

            Seems to be related to [https://issues.apache.org/jira/browse/GROOVY-8067] Groovy bug. Don't know when Jenkins groovy plugin will be update from 2.4.8 to 2.4.9 version .
            Jenkins 2.49 Don't know what is actually the problem. Memory dump available for in-house usage via Dropbox etc. if needed. Please send me an email.

             

            Seems to be related to [https://issues.apache.org/jira/browse/GROOVY-8067] Groovy bug. Don't know when Jenkins groovy will be update from 2.4.8 to 2.4.9 version .
            Hide
            heikkisi Heikki Simperi added a comment -

            Updated to 2.55 and still no groovy 2.4.9? At least we still face the same groovy problem 5-10 times per day and only fix is to restart jenkins.

            Show
            heikkisi Heikki Simperi added a comment - Updated to 2.55 and still no groovy 2.4.9? At least we still face the same groovy problem 5-10 times per day and only fix is to restart jenkins.
            heikkisi Heikki Simperi made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            Hide
            svanoort Sam Van Oort added a comment -

            Heikki Simperi  We understand this is a critical issue for you, and in order to solve it, can you provide an isolated example that will reproduce this without depending on your infra?    

            I think the reason we're hesitant to update Groovy more is that every update seems to introduce a new issue of this sort, which generally requires a significant time investment from someone deeply specialized in this area. 

             

            Show
            svanoort Sam Van Oort added a comment - Heikki Simperi   We understand this is a critical issue for you, and in order to solve it, can you provide an isolated example that will reproduce this without depending on your infra?     I think the reason we're hesitant to update Groovy more is that every update seems to introduce a new issue of this sort, which generally requires a significant time investment from someone deeply specialized in this area.   
            Hide
            heikkisi Heikki Simperi added a comment -

            Sam Van Oort I will look into this. Groovy bug has java class to reproduce the problem in groovy but I understand that it is not quite relevant.

            Is there way to give bundled groovy command line parameter as suggested as fast fix?

             

            -Dgroovy.use.classvalue=true 

            Show
            heikkisi Heikki Simperi added a comment - Sam Van Oort I will look into this. Groovy bug has java class to reproduce the problem in groovy but I understand that it is not quite relevant. Is there way to give bundled groovy command line parameter as suggested as fast fix?   -Dgroovy.use.classvalue=true 
            Hide
            grayaii Alex Gray added a comment -

             

            Not sure if this will help, but in our /etc/sysconfig/jenkins you can added:

            JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Dgroovy.use.classvalue=true -Xms4096m -Xmx4096m -XX:MaxPermSize=1024m -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -Dhudson.model.ParametersAction.keepUndefinedParameters=true"
            

            We also run some crazy groovy code in some of jenkins jobs and we tend to run out of memory too, so we've installed this plugin to help us track java resources:

             

            https://wiki.jenkins-ci.org/display/JENKINS/Monitoring

            And finally, we periodically (once per hour i think) run this groovy script to clean things up:

            import net.bull.javamelody.*;
            
            before = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();
            System.gc(); 
            after = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();
            println I18N.getFormattedString("ramasse_miette_execute", Math.round((before - after) / 1024));
            
            

            And we launch it like this:

            java -jar jenkins-cli.jar -noCertificateCheck -i id_rsa -s JENKINS_URL groovy my_groovy_script.groovy'
            

            You can probably put that groovy script as a jenkins job have it run periodically.

            Ever since we we run that System.gc() command, we've never run out of memory.  We run 100s of jobs a day a AWS t2.medium without any downtime for months at a time.  Before I did these things, we were running on a huge instance and we had to restart it every week or so.

            Hope this helps!

             

            Show
            grayaii Alex Gray added a comment -   Not sure if this will help, but in our /etc/sysconfig/jenkins you can added: JENKINS_JAVA_OPTIONS= "-Djava.awt.headless= true -Dgroovy.use.classvalue= true -Xms4096m -Xmx4096m -XX:MaxPermSize=1024m -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -Dhudson.model.ParametersAction.keepUndefinedParameters= true " We also run some crazy groovy code in some of jenkins jobs and we tend to run out of memory too, so we've installed this plugin to help us track java resources:   https://wiki.jenkins-ci.org/display/JENKINS/Monitoring And finally, we periodically (once per hour i think) run this groovy script to clean things up: import net.bull.javamelody.*; before = Runtime .getRuntime().totalMemory() - Runtime .getRuntime().freeMemory(); System .gc(); after = Runtime .getRuntime().totalMemory() - Runtime .getRuntime().freeMemory(); println I18N.getFormattedString( "ramasse_miette_execute" , Math .round((before - after) / 1024)); And we launch it like this: java -jar jenkins-cli.jar -noCertificateCheck -i id_rsa -s JENKINS_URL groovy my_groovy_script.groovy' You can probably put that groovy script as a jenkins job have it run periodically. Ever since we we run that System.gc() command, we've never run out of memory.  We run 100s of jobs a day a AWS t2.medium without any downtime for months at a time.  Before I did these things, we were running on a huge instance and we had to restart it every week or so. Hope this helps!  
            Hide
            heikkisi Heikki Simperi added a comment -

            Hello!

            We could not make reproducible test case with small effort.

            However, the parameter tuning from Groovy ticket seem to resolve the state. Now Jenkins has been working for one whole week without problems!

             

            Show
            heikkisi Heikki Simperi added a comment - Hello! We could not make reproducible test case with small effort. However, the parameter tuning from Groovy ticket seem to resolve the state. Now Jenkins has been working for one whole week without problems!  
            heikkisi Heikki Simperi made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            jglick Jesse Glick made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            jglick Jesse Glick made changes -
            Resolution Cannot Reproduce [ 5 ]
            Status Reopened [ 4 ] Resolved [ 5 ]
            io39d io39d made changes -
            Link This issue relates to JENKINS-43197 [ JENKINS-43197 ]
            Hide
            danekan Dane Kantner added a comment -

            From another bug where it was mentioned,  -Dgroovy.use.classvalue=true  seems to have fixed this for us as well. We actually don't use anything crazy at all in Groovy (I use it to read load a 3 line text file into a parameter list and there are try/catch blocks around that read operation even to fail to a default "" response), nonetheless the bugs in this version brought the entire platform to a state of frequent unavailability.  There are other bugs open related to this but it's hard to picture how this is "resolved" when it's such a major flaw that has already been fixed in groovy itself, though Jenkins still needs to upgrade to using that newer version internally.

            Show
            danekan Dane Kantner added a comment - From another bug where it was mentioned,  -Dgroovy.use.classvalue=true  seems to have fixed this for us as well. We actually don't use anything crazy at all in Groovy (I use it to read load a 3 line text file into a parameter list and there are try/catch blocks around that read operation even to fail to a default "" response), nonetheless the bugs in this version brought the entire platform to a state of frequent unavailability.  There are other bugs open related to this but it's hard to picture how this is "resolved" when it's such a major flaw that has already been fixed in groovy itself, though Jenkins still needs to upgrade to using that newer version internally.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              heikkisi Heikki Simperi
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: