Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-6473

Deadlock when rendering test result graphs

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Won't Fix
    • core
    • None
    • JDK 1.6.0.17
      Red Hat Enterprise Linux Server release 5.4 (Tikanga)
      2.6.18-164.9.1.el5 #1 SMP Wed Dec 9 03:29:54 EST 2009 i686 i686 i386 GNU/Linux
      Hudson 1.356

    Description

      The attached threaddump shows a deadlock which was initiated when attempting to view a job in my Hudson instance.

      The deadlock is at the following two threads (these are snippets from the attachment):

      "Handling GET /view/Quicksilver/job/users-service/test/trend : http-8080-9":
      at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1394)

      • waiting to lock <0xadf7e550> (a java.lang.String)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1361)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
      • locked <0x700966f0> (a org.apache.catalina.loader.WebappClassLoader)
        at hudson.tasks.test.AbstractTestResultAction.createChart(AbstractTestResultAction.java:260)
        at hudson.tasks.test.AbstractTestResultAction.doGraph(AbstractTestResultAction.java:208)

      "Handling GET /view/Quicksilver/job/users-service/tasks/trendGraph/png : http-8080-10":
      at java.lang.ClassLoader.checkCerts(ClassLoader.java:745)

      • waiting to lock <0x700966f0> (a org.apache.catalina.loader.WebappClassLoader)
        at java.lang.ClassLoader.preDefineClass(ClassLoader.java:484)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:610)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
        at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2527)
      • locked <0xadf7e550> (a java.lang.String)
        at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1010)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1483)
      • locked <0xadf7e550> (a java.lang.String)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:296)
      • locked <0x6ff84938> (a hudson.ClassicPluginStrategy$DependencyClassLoader)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:296)
      • locked <0x6ff83078> (a java.net.URLClassLoader)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
      • locked <0x6ff83078> (a java.net.URLClassLoader)
        at hudson.plugins.analysis.graph.CategoryBuildResultGraph.createAreaChart(CategoryBuildResultGraph.java:391)
        at hudson.plugins.analysis.graph.PriorityGraph.createChart(PriorityGraph.java:55)

      I first started noticing this behavior when updating the Static Analysis Collector Plug-in, Checkstyle Plug-in, PMD Plug-in, and FindBugs Plug-in to the latest versions.

      I tried to copy and paste the list of my installed plugins and version numbers.

      Attachments

        Issue Links

          Activity

            mindless Alan Harder added a comment -

            how repeatable is this? if you restart hudson and view the page again, does this happen every time?

            Threads from hudson core and an analysis plugin are both calling jfreechart at the same time, which both go to load some classes, apparently via classloaders coming in a different order. Looks like this is running in Tomcat? (what version?) Would be nice if tomcat was resilient to locking issues from loading via different classloaders.. maybe a different tomcat version or no tomcat would work better. Not sure what the best workaround in hudson code would be..

            mindless Alan Harder added a comment - how repeatable is this? if you restart hudson and view the page again, does this happen every time? Threads from hudson core and an analysis plugin are both calling jfreechart at the same time, which both go to load some classes, apparently via classloaders coming in a different order. Looks like this is running in Tomcat? (what version?) Would be nice if tomcat was resilient to locking issues from loading via different classloaders.. maybe a different tomcat version or no tomcat would work better. Not sure what the best workaround in hudson code would be..
            mindless Alan Harder added a comment -

            I see synchronized (name.intern()) in Tomcat 6.0.26 code.. I don't see this in 6.0.20 (it's there from 21 on) or 5.5.29 or 7.0.0-RC1, so maybe you'll have better luck with those?

            mindless Alan Harder added a comment - I see synchronized (name.intern()) in Tomcat 6.0.26 code.. I don't see this in 6.0.20 (it's there from 21 on) or 5.5.29 or 7.0.0-RC1, so maybe you'll have better luck with those?
            msbcode msbcode added a comment -

            > how repeatable is this? if you restart hudson and view the page again, does this happen every time?

            I'm not certain about "every time" but it has definitely happened more than once.

            > Looks like this is running in Tomcat? (what version?)

            I believe 6.0.26 (don't have access to the machine at the current moment). Will try to downgrade on Monday and get back to you.

            msbcode msbcode added a comment - > how repeatable is this? if you restart hudson and view the page again, does this happen every time? I'm not certain about "every time" but it has definitely happened more than once. > Looks like this is running in Tomcat? (what version?) I believe 6.0.26 (don't have access to the machine at the current moment). Will try to downgrade on Monday and get back to you.
            msbcode msbcode added a comment -

            I switched from using Tomcat to host Hudson to simply running it within Winstone - no further deadlock problems.

            msbcode msbcode added a comment - I switched from using Tomcat to host Hudson to simply running it within Winstone - no further deadlock problems.
            mindless Alan Harder added a comment -

            See comments 12-17 in apache bug 44041 .. sounds like Tomcat 6.0.27 might work better.

            mindless Alan Harder added a comment - See comments 12-17 in apache bug 44041 .. sounds like Tomcat 6.0.27 might work better.

            WebAppClassLoader source code in question for a reference.

            The dead lock is the inconsistent lock order between the interned class name and the WebappClassLoader itself. The loadClass method itself isn't synchronized, which enables String->WebAppClassLoader order, but when called from methods like loadClassInternal, the order becomes the other way around.

            A possible work around is to externally synchronize WebappClassLoader before calling loadClass. This can be done from our classloader, and it might reduce the risk, if not eliminate the problem.

            kohsuke Kohsuke Kawaguchi added a comment - WebAppClassLoader source code in question for a reference. The dead lock is the inconsistent lock order between the interned class name and the WebappClassLoader itself. The loadClass method itself isn't synchronized, which enables String->WebAppClassLoader order, but when called from methods like loadClassInternal, the order becomes the other way around. A possible work around is to externally synchronize WebappClassLoader before calling loadClass. This can be done from our classloader, and it might reduce the risk, if not eliminate the problem.

            People

              Unassigned Unassigned
              msbcode msbcode
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: