The following thread dump (from one stuck slave) shows a deadlock between

      pool-635-thread-1 and pool-635-thread-2

      Looks like it is due to commons-logging used by maven wagon.

          [JENKINS-15846] Deadlock during Maven builds Parsing POM step

          Vincent Latombe created issue -

          Vincent Latombe added a comment - Looks like it is due to https://issues.apache.org/jira/browse/LOGGING-119
          Vincent Latombe made changes -
          Environment Original: Jenkins 1.473 New: Jenkins 1.473
          JDK 1.6.0_26

          Jesse Glick added a comment -
          "pool-635-thread-1" … BLOCKED on org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable@13cbe4 owned by "pool-635-thread-2" …
          	at java.util.Hashtable.remove(Hashtable.java:435)
          	-  blocked on org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable@13cbe4
          	at org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable.purgeOne(WeakHashtable.java:338)
          	-  locked java.lang.ref.ReferenceQueue@1b8447f
          	at org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable.put(WeakHashtable.java:238)
          	at org.apache.maven.wagon.providers.http.commons.logging.LogFactory.cacheFactory(LogFactory.java:1004)
          
          "pool-635-thread-2" … BLOCKED on java.lang.ref.ReferenceQueue@1b8447f owned by "pool-635-thread-1" …
          	at org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable.purge(WeakHashtable.java:323)
          	-  blocked on java.lang.ref.ReferenceQueue@1b8447f
          	at org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable.rehash(WeakHashtable.java:312)
          	at java.util.Hashtable.put(Hashtable.java:412)
          	-  locked org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable@13cbe4
          	at org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable.put(WeakHashtable.java:242)
          	at org.apache.maven.wagon.providers.http.commons.logging.LogFactory.cacheFactory(LogFactory.java:1004)
          

          Jesse Glick added a comment - "pool-635-thread-1" … BLOCKED on org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable@13cbe4 owned by "pool-635-thread-2" … at java.util.Hashtable.remove(Hashtable.java:435) - blocked on org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable@13cbe4 at org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable.purgeOne(WeakHashtable.java:338) - locked java.lang.ref.ReferenceQueue@1b8447f at org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable.put(WeakHashtable.java:238) at org.apache.maven.wagon.providers.http.commons.logging.LogFactory.cacheFactory(LogFactory.java:1004) "pool-635-thread-2" … BLOCKED on java.lang.ref.ReferenceQueue@1b8447f owned by "pool-635-thread-1" … at org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable.purge(WeakHashtable.java:323) - blocked on java.lang.ref.ReferenceQueue@1b8447f at org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable.rehash(WeakHashtable.java:312) at java.util.Hashtable.put(Hashtable.java:412) - locked org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable@13cbe4 at org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable.put(WeakHashtable.java:242) at org.apache.maven.wagon.providers.http.commons.logging.LogFactory.cacheFactory(LogFactory.java:1004)

          Jesse Glick added a comment -

          Tricky because there is not yet a Commons Logging release including the fix, and the current version (1.1.1) is packed into wagon-http-2.3-shaded.jar. To fix, would need at least a beta of 1.1.2 to be released, and then for Wagon 2.5 or similar to be released, at which point Jenkins could pick it up in the Maven plugin. Or Jenkins would have to fork both Commons Logging and Wagon, which is not an appealing prospect.

          Jesse Glick added a comment - Tricky because there is not yet a Commons Logging release including the fix, and the current version (1.1.1) is packed into wagon-http-2.3-shaded.jar . To fix, would need at least a beta of 1.1.2 to be released, and then for Wagon 2.5 or similar to be released, at which point Jenkins could pick it up in the Maven plugin. Or Jenkins would have to fork both Commons Logging and Wagon, which is not an appealing prospect.
          Jesse Glick made changes -
          Assignee New: Jesse Glick [ jglick ]

          Jesse Glick added a comment -

          Possible workaround being tested: add to slave JVM args: -Dorg.apache.maven.wagon.providers.http.commons.logging.LogFactory.HashtableImpl=java.util.Hashtable

          Jesse Glick added a comment - Possible workaround being tested: add to slave JVM args: -Dorg.apache.maven.wagon.providers.http.commons.logging.LogFactory.HashtableImpl=java.util.Hashtable

          Vincent Latombe added a comment - - edited

          New samples of the deadlock (with thread dumps from master and slave). In the example, only one job is stuck : app-sell2.0_v16.2_cr5743145_ci

          Vincent Latombe added a comment - - edited New samples of the deadlock (with thread dumps from master and slave). In the example, only one job is stuck : app-sell2.0_v16.2_cr5743145_ci
          Vincent Latombe made changes -
          Attachment New: masterThreadDump.txt [ 23218 ]
          Attachment New: slaveThreadDump.txt [ 23219 ]

          Jesse Glick added a comment -

          Is this with or without the aforementioned system property set?

          Jesse Glick added a comment - Is this with or without the aforementioned system property set?

            olamy Olivier Lamy
            vlatombe Vincent Latombe
            Votes:
            3 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: