Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17571

The main Jenkins dashboard page is slow to reload, seems to cause unusually high CPU load in the process

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Critical Critical
    • core
    • None
    • Jenkins 1.510 + gerrit 2.5.2 on Ubuntu 12.04. See attached systeminfo file for details as reported by Jenkins.

      I just updated from Jenkins 1.505 today as part of resolving another outstanding issue. I notice that the main dashboard page is much slower to load than it was before, and also notice that the server gets high CPU load in the process (it gets super-bad if you try and reload it a few times, as one might do when a page seems frozen). This is not the case on the individual job pages, the "manage Jenkins" page or other such ancillary pages.

      My main page has quite a few jobs in the main view, but that didn't have an adverse effect before.

      It did have some trouble connecting to the slave, right after the update, but that appears to be working properly now.

      At the moment, several minutes after it seemed to have normalized a bit, I see one CPU is now completely spiked by Jenkins.

      Downgraded to 1.505 + git-client 1.03 until this is resolved or explained.

      Note 1: After the downgrade things are back to being speedy. The dashboard loads quickly on opening it or on switching to it, and any CPU load is very brief (versus waiting several seconds for the page to display, or much longer if it was refreshed a few times). I hit reload several times in succession... it was reloaded in a second with no major load on the CPU (versus waiting around 30 seconds for the page to finally appear on the newer version).

      Note 2: Even in 1.505 you end up with one Jenkins-bearing CPU that stubbornly spikes at 100% after a few quick reloads... and appears to stay that way (if you reload a few more times you still end up with this condition, though it might be on another CPU in the system). So the persistent CPU spike may or may not be related to this issue.

        1. systeminfo.txt
          4 kB

          [JENKINS-17571] The main Jenkins dashboard page is slow to reload, seems to cause unusually high CPU load in the process

          Martin Falatic added a comment - - edited

          On that persistent CPU thing: several full refreshes of the Jenkins dashboard page (shift-ctrl-R in Windows) is quite effective at spiking not just one but several CPUs at once. I'm curious if this is a known issue or how to track down where the problem is.

          As relates to this present 1.510 issue, even with almost all the cores maxed out because of this weird persistent CPU thing, 1.505 still loads and presents the dashboard quickly. 1.510 would just get worse and worse the more you hammered it.

          Now, on the prudence of actually refreshing this much: At this point I'm testing the boundaries of this issue. Obviously this isn't a great thing to do, but the tool shouldn't have persistent adverse side-effects because of it, and performance completely tanks in 1.510 if you do this even a little (it was the lag on switching back to the dashboard view that caused me to refresh and trigger this in the first place, and to notice how much extra time even a couple of loads/refreshes added to an already extended load time).

          A few users accessing the dashboard at once would have a similar effect in terms of simultaneous load, with no doubt similar unpleasant results.

          Martin Falatic added a comment - - edited On that persistent CPU thing: several full refreshes of the Jenkins dashboard page (shift-ctrl-R in Windows) is quite effective at spiking not just one but several CPUs at once. I'm curious if this is a known issue or how to track down where the problem is. As relates to this present 1.510 issue, even with almost all the cores maxed out because of this weird persistent CPU thing, 1.505 still loads and presents the dashboard quickly. 1.510 would just get worse and worse the more you hammered it. Now, on the prudence of actually refreshing this much: At this point I'm testing the boundaries of this issue. Obviously this isn't a great thing to do, but the tool shouldn't have persistent adverse side-effects because of it, and performance completely tanks in 1.510 if you do this even a little (it was the lag on switching back to the dashboard view that caused me to refresh and trigger this in the first place, and to notice how much extra time even a couple of loads/refreshes added to an already extended load time). A few users accessing the dashboard at once would have a similar effect in terms of simultaneous load, with no doubt similar unpleasant results.

          dave smolin added a comment -

          I just updated jenkins from 1.505 to 1.511 and immediately encountered this issue. Jenkins in this state is unusable, and we will be downgrading back to 1.505. We're using jenkins as a native package on ubuntu 10.10 on a beefy box.

          dave smolin added a comment - I just updated jenkins from 1.505 to 1.511 and immediately encountered this issue. Jenkins in this state is unusable, and we will be downgrading back to 1.505. We're using jenkins as a native package on ubuntu 10.10 on a beefy box.

          Justin Slattery added a comment -

          Another confirmation here. We upgraded to 1.511 and immediately encountered the issue. I lost more time than I care to admit troubleshooting. Downgrading to 1.505 fixed it right away.

          Justin Slattery added a comment - Another confirmation here. We upgraded to 1.511 and immediately encountered the issue. I lost more time than I care to admit troubleshooting. Downgrading to 1.505 fixed it right away.

          Jesse Glick added a comment -

          To anyone suffering from this: it is impossible to even begin diagnosis without a thread dump (e.g. /threadDump in the web UI, or jstack from a shell on the master).

          Jesse Glick added a comment - To anyone suffering from this: it is impossible to even begin diagnosis without a thread dump (e.g. /threadDump in the web UI, or jstack from a shell on the master).

          Arnaud Héritier added a comment -

          Forme this issue duplicates this one JENKINS-15858 which is related to JENKINS-15929

          Arnaud Héritier added a comment - Forme this issue duplicates this one JENKINS-15858 which is related to JENKINS-15929

          Jesse Glick added a comment -

          Possible duplicate (impossible to say without a thread dump).

          Jesse Glick added a comment - Possible duplicate (impossible to say without a thread dump).

          Niels Beekman added a comment - - edited

          We are also experiencing slowness on the main dashboard. In our case it appears to be unrelated to JENKINS-15858, but rather related to symlink resolving. Relevant stacktrace:

          "Handling GET / : RequestHandlerThread1" Id=695 Group=main WAITING on java.lang.UNIXProcess$Gate@43c4147a
          at java.lang.Object.wait(Native Method)

          • waiting on java.lang.UNIXProcess$Gate@43c4147a
            at java.lang.Object.wait(Object.java:502)
            at java.lang.UNIXProcess$Gate.waitForExit(UNIXProcess.java:80)
            at java.lang.UNIXProcess.<init>(UNIXProcess.java:161)
            at java.lang.ProcessImpl.start(ProcessImpl.java:81)
            at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
            at java.lang.Runtime.exec(Runtime.java:610)
            at org.jruby.ext.posix.util.ExecIt.run(ExecIt.java:61)
            at org.jruby.ext.posix.util.ExecIt.runAndWait(ExecIt.java:51)
            at org.jruby.ext.posix.JavaLibCHelper.readlink(JavaLibCHelper.java:196)
            at org.jruby.ext.posix.JavaPOSIX.readlink(JavaPOSIX.java:160)
            at hudson.Util.resolveSymlink(Util.java:1240)
            at jenkins.model.PeepholePermalink.readSymlink(PeepholePermalink.java:154)
            at jenkins.model.PeepholePermalink.resolve(PeepholePermalink.java:85)
            at hudson.model.Job.getLastSuccessfulBuild(Job.java:802)
            at sun.reflect.GeneratedMethodAccessor70.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:616)
            at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
            at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
            at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
            at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
            at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
            at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
            at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
            at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
            at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
            at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
            at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146)
            at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
            at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
            at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
            at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
            at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
            at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
            at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
            at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
            at org.kohsuke.stapler.jelly.groovy.JellyBuilder.doInvokeMethod(JellyBuilder.java:276)
            at org.kohsuke.stapler.jelly.groovy.Namespace$ProxyImpl.invoke(Namespace.java:92)
            at sun.proxy.$Proxy33.projectView(Unknown Source)
            at lib.JenkinsTagLib$projectView.call(Unknown Source)
            at hudson.model.View.main.run(main.groovy:14)
            at org.kohsuke.stapler.jelly.groovy.GroovierJellyScript.run(GroovierJellyScript.java:69)
            at org.kohsuke.stapler.jelly.groovy.GroovierJellyScript.run(GroovierJellyScript.java:62)
            at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146)
            at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98)
            at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
            at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98)
            at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
            at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
            at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
            at org.kohsuke.stapler.jelly.CompressTag.doTag(CompressTag.java:44)
            at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
            at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
            at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
            at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
            at org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112)
            at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:127)
            at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:666)
            at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)
            at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241)
            at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
            at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677)
            at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)
            at org.kohsuke.stapler.Stapler.invoke(Stapler.java:583)
            at org.kohsuke.stapler.Stapler.service(Stapler.java:214)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
            at winstone.ServletConfiguration.execute(ServletConfiguration.java:248)
            at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)
            at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
            at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
            at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
            at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
            at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
            at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
            at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
            at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
            at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
            at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
            at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
            at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
            at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
            at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227)
            at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
            at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
            at java.util.concurrent.FutureTask.run(FutureTask.java:166)
            at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:679)

          Number of locked synchronizers = 1

          • java.util.concurrent.ThreadPoolExecutor$Worker@637a91a

          Environment:
          Jenkins ver. 1.515
          Ubuntu 12.0.4

          Niels Beekman added a comment - - edited We are also experiencing slowness on the main dashboard. In our case it appears to be unrelated to JENKINS-15858 , but rather related to symlink resolving. Relevant stacktrace: "Handling GET / : RequestHandlerThread 1 " Id=695 Group=main WAITING on java.lang.UNIXProcess$Gate@43c4147a at java.lang.Object.wait(Native Method) waiting on java.lang.UNIXProcess$Gate@43c4147a at java.lang.Object.wait(Object.java:502) at java.lang.UNIXProcess$Gate.waitForExit(UNIXProcess.java:80) at java.lang.UNIXProcess.<init>(UNIXProcess.java:161) at java.lang.ProcessImpl.start(ProcessImpl.java:81) at java.lang.ProcessBuilder.start(ProcessBuilder.java:468) at java.lang.Runtime.exec(Runtime.java:610) at org.jruby.ext.posix.util.ExecIt.run(ExecIt.java:61) at org.jruby.ext.posix.util.ExecIt.runAndWait(ExecIt.java:51) at org.jruby.ext.posix.JavaLibCHelper.readlink(JavaLibCHelper.java:196) at org.jruby.ext.posix.JavaPOSIX.readlink(JavaPOSIX.java:160) at hudson.Util.resolveSymlink(Util.java:1240) at jenkins.model.PeepholePermalink.readSymlink(PeepholePermalink.java:154) at jenkins.model.PeepholePermalink.resolve(PeepholePermalink.java:85) at hudson.model.Job.getLastSuccessfulBuild(Job.java:802) at sun.reflect.GeneratedMethodAccessor70.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125) at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314) at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185) at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75) at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83) at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57) at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51) at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80) at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74) at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119) at org.kohsuke.stapler.jelly.groovy.JellyBuilder.doInvokeMethod(JellyBuilder.java:276) at org.kohsuke.stapler.jelly.groovy.Namespace$ProxyImpl.invoke(Namespace.java:92) at sun.proxy.$Proxy33.projectView(Unknown Source) at lib.JenkinsTagLib$projectView.call(Unknown Source) at hudson.model.View.main.run(main.groovy:14) at org.kohsuke.stapler.jelly.groovy.GroovierJellyScript.run(GroovierJellyScript.java:69) at org.kohsuke.stapler.jelly.groovy.GroovierJellyScript.run(GroovierJellyScript.java:62) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119) at org.kohsuke.stapler.jelly.CompressTag.doTag(CompressTag.java:44) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53) at org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112) at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:127) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:666) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:583) at org.kohsuke.stapler.Stapler.service(Stapler.java:214) at javax.servlet.http.HttpServlet.service(HttpServlet.java:45) at winstone.ServletConfiguration.execute(ServletConfiguration.java:248) at winstone.RequestDispatcher.forward(RequestDispatcher.java:333) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@637a91a Environment: Jenkins ver. 1.515 Ubuntu 12.0.4

          Niels Beekman added a comment -

          Issue seems to be related to the fallback POSIX implementation being used instead of a native implementation. Similar to JENKINS-13341. Installing the Oracle JVM fixed the issue.

          Niels Beekman added a comment - Issue seems to be related to the fallback POSIX implementation being used instead of a native implementation. Similar to JENKINS-13341 . Installing the Oracle JVM fixed the issue.

          Jesse Glick added a comment -

          @nbeekman: your issue is unrelated I think. Not sure if it is specifically filed, but on more exotic platforms if you are using Java 6 (or 5) the symlink handling code is slow. Better to run Java 7, but there is also a pull request open to evaluate jnr-posix which may help.

          Jesse Glick added a comment - @nbeekman: your issue is unrelated I think. Not sure if it is specifically filed, but on more exotic platforms if you are using Java 6 (or 5) the symlink handling code is slow. Better to run Java 7, but there is also a pull request open to evaluate jnr-posix which may help.

          Niels Beekman added a comment -

          Fair enough, impossible to say without a stack dump from the issue reported I guess.
          A small note regarding "exotic" though, this problem occurred on a stock Ubuntu 12.0.4 with OpenJDK 7 installed using apt-get. This might impact more users than one might think.
          A related question: is there a specific reason that a process is launched to resolve a symlink? For example, could http://docs.oracle.com/javase/6/docs/api/java/io/File.html#getCanonicalPath%28%29 be used?

          Niels Beekman added a comment - Fair enough, impossible to say without a stack dump from the issue reported I guess. A small note regarding "exotic" though, this problem occurred on a stock Ubuntu 12.0.4 with OpenJDK 7 installed using apt-get. This might impact more users than one might think. A related question: is there a specific reason that a process is launched to resolve a symlink? For example, could http://docs.oracle.com/javase/6/docs/api/java/io/File.html#getCanonicalPath%28%29 be used?

          Jesse Glick added a comment -

          @nbeekman: usually it is AIX/HP-UX users with issues. If you are on Linux using Java 5/6 then the JNA library should be used to load readlink from libc, and on Java 7 for any platform java.nio.file should be used. From your stack trace it seems like every attempt to read symlinks normally failed: java.nio.file; JNA; and the LinuxPOSIX part of jna-posix (which should have printed something like Failed to load native POSIX impl to console); finally fall back to the crappiest method: forking /usr/bin/readlink. So something is seriously wrong with either your Java installation or the Jenkins method Util.resolveSymlink; check e.g. UtilTest.testSymlink in a debugger.

          Jesse Glick added a comment - @nbeekman: usually it is AIX/HP-UX users with issues. If you are on Linux using Java 5/6 then the JNA library should be used to load readlink from libc , and on Java 7 for any platform java.nio.file should be used. From your stack trace it seems like every attempt to read symlinks normally failed: java.nio.file ; JNA; and the LinuxPOSIX part of jna-posix (which should have printed something like Failed to load native POSIX impl to console); finally fall back to the crappiest method: forking /usr/bin/readlink . So something is seriously wrong with either your Java installation or the Jenkins method Util.resolveSymlink ; check e.g. UtilTest.testSymlink in a debugger.

          Niels Beekman added a comment -

          Using OpenJDK 6 I see the following exception in the log:
          Failed to load native POSIX impl; falling back on Java impl. Stacktrace follows.
          java.lang.UnsatisfiedLinkError: Unable to load library 'libc.so.6': com.sun.jna.Native.open(Ljava/lang/String;)J
          at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:166)
          at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:239)
          at com.sun.jna.Library$Handler.<init>(Library.java:140)
          at com.sun.jna.Native.loadLibrary(Native.java:366)
          at org.jruby.ext.posix.POSIXFactory.loadLibC(POSIXFactory.java:96)
          at org.jruby.ext.posix.POSIXFactory.loadLinuxPOSIX(POSIXFactory.java:65)
          at org.jruby.ext.posix.POSIXFactory.getPOSIX(POSIXFactory.java:24)
          at hudson.os.PosixAPI.<clinit>(PosixAPI.java:41)
          at hudson.Util.resolveSymlink(Util.java:1240)
          at hudson.Util.resolveSymlinkToFile(Util.java:1166)
          at hudson.model.Run.parseTimestampFromBuildDir(Run.java:346)
          at hudson.model.Run.<init>(Run.java:294)
          at hudson.model.AbstractBuild.<init>(AbstractBuild.java:184)
          at hudson.maven.AbstractMavenBuild.<init>(AbstractMavenBuild.java:54)
          at hudson.maven.MavenModuleSetBuild.<init>(MavenModuleSetBuild.java:136)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
          at hudson.model.AbstractProject.loadBuild(AbstractProject.java:1122)
          at hudson.model.AbstractProject$1.create(AbstractProject.java:322)
          at hudson.model.AbstractProject$1.create(AbstractProject.java:320)
          at hudson.model.RunMap.retrieve(RunMap.java:225)
          at hudson.model.RunMap.retrieve(RunMap.java:59)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:667)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:629)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:368)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:526)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:379)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:321)
          at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:1059)
          at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184)
          at hudson.maven.MavenModuleSet.createTransientActions(MavenModuleSet.java:455)
          at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:717)
          at hudson.maven.MavenModuleSet.updateTransientActions(MavenModuleSet.java:451)
          at hudson.model.AbstractProject.onLoad(AbstractProject.java:316)
          at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:753)
          at hudson.model.Items.load(Items.java:221)
          at jenkins.model.Jenkins$17.run(Jenkins.java:2542)
          at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146)
          at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
          at jenkins.model.Jenkins$7.runTask(Jenkins.java:893)
          at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
          at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:679)

          Niels Beekman added a comment - Using OpenJDK 6 I see the following exception in the log: Failed to load native POSIX impl; falling back on Java impl. Stacktrace follows. java.lang.UnsatisfiedLinkError: Unable to load library 'libc.so.6': com.sun.jna.Native.open(Ljava/lang/String;)J at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:166) at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:239) at com.sun.jna.Library$Handler.<init>(Library.java:140) at com.sun.jna.Native.loadLibrary(Native.java:366) at org.jruby.ext.posix.POSIXFactory.loadLibC(POSIXFactory.java:96) at org.jruby.ext.posix.POSIXFactory.loadLinuxPOSIX(POSIXFactory.java:65) at org.jruby.ext.posix.POSIXFactory.getPOSIX(POSIXFactory.java:24) at hudson.os.PosixAPI.<clinit>(PosixAPI.java:41) at hudson.Util.resolveSymlink(Util.java:1240) at hudson.Util.resolveSymlinkToFile(Util.java:1166) at hudson.model.Run.parseTimestampFromBuildDir(Run.java:346) at hudson.model.Run.<init>(Run.java:294) at hudson.model.AbstractBuild.<init>(AbstractBuild.java:184) at hudson.maven.AbstractMavenBuild.<init>(AbstractMavenBuild.java:54) at hudson.maven.MavenModuleSetBuild.<init>(MavenModuleSetBuild.java:136) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:532) at hudson.model.AbstractProject.loadBuild(AbstractProject.java:1122) at hudson.model.AbstractProject$1.create(AbstractProject.java:322) at hudson.model.AbstractProject$1.create(AbstractProject.java:320) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:667) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:629) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:368) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:526) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:379) at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:321) at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:1059) at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184) at hudson.maven.MavenModuleSet.createTransientActions(MavenModuleSet.java:455) at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:717) at hudson.maven.MavenModuleSet.updateTransientActions(MavenModuleSet.java:451) at hudson.model.AbstractProject.onLoad(AbstractProject.java:316) at hudson.maven.MavenModuleSet.onLoad(MavenModuleSet.java:753) at hudson.model.Items.load(Items.java:221) at jenkins.model.Jenkins$17.run(Jenkins.java:2542) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259) at jenkins.model.Jenkins$7.runTask(Jenkins.java:893) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679)

          Jesse Glick added a comment -

          Maybe we are bundling a version of JNA which is too old? Is this 64-bit?

          Jesse Glick added a comment - Maybe we are bundling a version of JNA which is too old? Is this 64-bit?

          Niels Beekman added a comment -

          java version "1.6.0_27"
          OpenJDK Runtime Environment (IcedTea6 1.12.3) (6b27-1.12.3-0ubuntu1~12.04.1)
          OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

          Niels Beekman added a comment - java version "1.6.0_27" OpenJDK Runtime Environment (IcedTea6 1.12.3) (6b27-1.12.3-0ubuntu1~12.04.1) OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

          dave added a comment -

          Linux tgts-jenkins 2.6.35-22-server #35-Ubuntu SMP Sat Oct 16 22:02:33 UTC 2010 x86_64 GNU/Linux
          Ubuntu 10.10

          and
          java version "1.6.0_21"
          Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
          Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)

          dave added a comment - Linux tgts-jenkins 2.6.35-22-server #35-Ubuntu SMP Sat Oct 16 22:02:33 UTC 2010 x86_64 GNU/Linux Ubuntu 10.10 and java version "1.6.0_21" Java(TM) SE Runtime Environment (build 1.6.0_21-b06) Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)

          Marco Ambu added a comment -

          I can see that this is assigned to component dashboard-view which is a plugin. Are you using the dashboard-view plugin to display your default job page in Jenkins? If not, you need to choose another component to get this bug looked at.

          Marco Ambu added a comment - I can see that this is assigned to component dashboard-view which is a plugin. Are you using the dashboard-view plugin to display your default job page in Jenkins? If not, you need to choose another component to get this bug looked at.

          Martin Falatic added a comment - - edited

          If "dashboard-view" is the default Jenkins view (appears so, looking at a snapshot of the "Standard Jenkins jobs list") then this is correct, assuming that's where the problem is (versus some other part of Jenkins that transacts with the dashboard-view plugin).

          Martin Falatic added a comment - - edited If "dashboard-view" is the default Jenkins view (appears so, looking at a snapshot of the "Standard Jenkins jobs list") then this is correct, assuming that's where the problem is (versus some other part of Jenkins that transacts with the dashboard-view plugin).

          Jesse Glick added a comment -

          Note that dashboard-view 2.8 includes a fix for JENKINS-15858; it is unclear whether or not this bug is a duplicate.

          Jesse Glick added a comment - Note that dashboard-view 2.8 includes a fix for JENKINS-15858 ; it is unclear whether or not this bug is a duplicate.

          Martin Falatic added a comment -

          15858 seems to involve much earlier versions of Jenkins as well, where I've never encountered the problem. Jenkins 1.505 is the last stable release from my perspective... and that's assuming this was the only new showstopper introduced since then that remains outstanding (thankfully jgit as a default was removed quickly).

          Martin Falatic added a comment - 15858 seems to involve much earlier versions of Jenkins as well, where I've never encountered the problem. Jenkins 1.505 is the last stable release from my perspective... and that's assuming this was the only new showstopper introduced since then that remains outstanding (thankfully jgit as a default was removed quickly).

          Marco Ambu added a comment -

          The default job page in Jenkins is not the dashboard view plugin page.
          The dashboard-view is a plugin and must be installed and configured.
          Removed dashboard-view from the components.

          Marco Ambu added a comment - The default job page in Jenkins is not the dashboard view plugin page. The dashboard-view is a plugin and must be installed and configured. Removed dashboard-view from the components.

          Jesse Glick added a comment -

          JNR in 1.518+ ought to help, given the reported thread dumps.

          Jesse Glick added a comment - JNR in 1.518+ ought to help, given the reported thread dumps.

          Patrick Hemmer added a comment -

          Experienced this same issue with the LTS 1.509.1 release on Ubuntu LTS 12.04.
          Upgrading to openjdk-7 resolved this issue for me. Was experiencing the exact same `JavaPOSIX.readlink` thing.
          I wanted to stay on the LTS release, and not have to upgrade to 1.518+. So just providing this info for anyone else who is looking for the same.

          Patrick Hemmer added a comment - Experienced this same issue with the LTS 1.509.1 release on Ubuntu LTS 12.04. Upgrading to openjdk-7 resolved this issue for me. Was experiencing the exact same `JavaPOSIX.readlink` thing. I wanted to stay on the LTS release, and not have to upgrade to 1.518+. So just providing this info for anyone else who is looking for the same.

            Unassigned Unassigned
            martymacgyver Martin Falatic
            Votes:
            5 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated:
              Resolved: