The following thread dump (from one stuck slave) shows a deadlock between

      pool-635-thread-1 and pool-635-thread-2

      Looks like it is due to commons-logging used by maven wagon.

          [JENKINS-15846] Deadlock during Maven builds Parsing POM step

          Vincent Latombe added a comment - Looks like it is due to https://issues.apache.org/jira/browse/LOGGING-119

          Jesse Glick added a comment -
          "pool-635-thread-1" … BLOCKED on org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable@13cbe4 owned by "pool-635-thread-2" …
          	at java.util.Hashtable.remove(Hashtable.java:435)
          	-  blocked on org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable@13cbe4
          	at org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable.purgeOne(WeakHashtable.java:338)
          	-  locked java.lang.ref.ReferenceQueue@1b8447f
          	at org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable.put(WeakHashtable.java:238)
          	at org.apache.maven.wagon.providers.http.commons.logging.LogFactory.cacheFactory(LogFactory.java:1004)
          
          "pool-635-thread-2" … BLOCKED on java.lang.ref.ReferenceQueue@1b8447f owned by "pool-635-thread-1" …
          	at org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable.purge(WeakHashtable.java:323)
          	-  blocked on java.lang.ref.ReferenceQueue@1b8447f
          	at org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable.rehash(WeakHashtable.java:312)
          	at java.util.Hashtable.put(Hashtable.java:412)
          	-  locked org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable@13cbe4
          	at org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable.put(WeakHashtable.java:242)
          	at org.apache.maven.wagon.providers.http.commons.logging.LogFactory.cacheFactory(LogFactory.java:1004)
          

          Jesse Glick added a comment - "pool-635-thread-1" … BLOCKED on org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable@13cbe4 owned by "pool-635-thread-2" … at java.util.Hashtable.remove(Hashtable.java:435) - blocked on org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable@13cbe4 at org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable.purgeOne(WeakHashtable.java:338) - locked java.lang.ref.ReferenceQueue@1b8447f at org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable.put(WeakHashtable.java:238) at org.apache.maven.wagon.providers.http.commons.logging.LogFactory.cacheFactory(LogFactory.java:1004) "pool-635-thread-2" … BLOCKED on java.lang.ref.ReferenceQueue@1b8447f owned by "pool-635-thread-1" … at org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable.purge(WeakHashtable.java:323) - blocked on java.lang.ref.ReferenceQueue@1b8447f at org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable.rehash(WeakHashtable.java:312) at java.util.Hashtable.put(Hashtable.java:412) - locked org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable@13cbe4 at org.apache.maven.wagon.providers.http.commons.logging.impl.WeakHashtable.put(WeakHashtable.java:242) at org.apache.maven.wagon.providers.http.commons.logging.LogFactory.cacheFactory(LogFactory.java:1004)

          Jesse Glick added a comment -

          Tricky because there is not yet a Commons Logging release including the fix, and the current version (1.1.1) is packed into wagon-http-2.3-shaded.jar. To fix, would need at least a beta of 1.1.2 to be released, and then for Wagon 2.5 or similar to be released, at which point Jenkins could pick it up in the Maven plugin. Or Jenkins would have to fork both Commons Logging and Wagon, which is not an appealing prospect.

          Jesse Glick added a comment - Tricky because there is not yet a Commons Logging release including the fix, and the current version (1.1.1) is packed into wagon-http-2.3-shaded.jar . To fix, would need at least a beta of 1.1.2 to be released, and then for Wagon 2.5 or similar to be released, at which point Jenkins could pick it up in the Maven plugin. Or Jenkins would have to fork both Commons Logging and Wagon, which is not an appealing prospect.

          Jesse Glick added a comment -

          Possible workaround being tested: add to slave JVM args: -Dorg.apache.maven.wagon.providers.http.commons.logging.LogFactory.HashtableImpl=java.util.Hashtable

          Jesse Glick added a comment - Possible workaround being tested: add to slave JVM args: -Dorg.apache.maven.wagon.providers.http.commons.logging.LogFactory.HashtableImpl=java.util.Hashtable

          Vincent Latombe added a comment - - edited

          New samples of the deadlock (with thread dumps from master and slave). In the example, only one job is stuck : app-sell2.0_v16.2_cr5743145_ci

          Vincent Latombe added a comment - - edited New samples of the deadlock (with thread dumps from master and slave). In the example, only one job is stuck : app-sell2.0_v16.2_cr5743145_ci

          Jesse Glick added a comment -

          Is this with or without the aforementioned system property set?

          Jesse Glick added a comment - Is this with or without the aforementioned system property set?

          without

          Vincent Latombe added a comment - without

          the linked commons-logging issues has been resolved - https://issues.apache.org/jira/browse/LOGGING-119
          also, apache have released version 1.2.1 of commons logging that includes this fix - http://commons.apache.org/proper/commons-logging/download_logging.cgi

          so, in theory, nothing now stands in the way of upgrading jenkins' version of commons logging and resolving this issue?

          radai rosenblatt added a comment - the linked commons-logging issues has been resolved - https://issues.apache.org/jira/browse/LOGGING-119 also, apache have released version 1.2.1 of commons logging that includes this fix - http://commons.apache.org/proper/commons-logging/download_logging.cgi so, in theory, nothing now stands in the way of upgrading jenkins' version of commons logging and resolving this issue?

          Jesse Glick added a comment -

          Nope, now we need Wagon 2.5 with a fix of: https://jira.codehaus.org/browse/WAGON-394

          Jesse Glick added a comment - Nope, now we need Wagon 2.5 with a fix of: https://jira.codehaus.org/browse/WAGON-394

          Tony Bridges added a comment -

          Are all the pieces in place ? Is this expected in a release any time soon ? I'm hitting this issue a lot more often than now and then - about twice a day.

          Tony Bridges added a comment - Are all the pieces in place ? Is this expected in a release any time soon ? I'm hitting this issue a lot more often than now and then - about twice a day.

          Tony Bridges added a comment -

          For the record, the HashTable workaround does not solve the problem.

          Tony Bridges added a comment - For the record, the HashTable workaround does not solve the problem.

          Code changed in jenkins
          User: Olivier Lamy
          Path:
          core/pom.xml
          maven-plugin/pom.xml
          pom.xml
          http://jenkins-ci.org/commit/jenkins/13ae5c793663976587a06d3dca585f7559173efb
          Log:
          JENKINS-15846 Deadlock during Maven builds Parsing POM step
          Do not use anymore shaded version of wagon-http as it we can control commons-logging version

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Olivier Lamy Path: core/pom.xml maven-plugin/pom.xml pom.xml http://jenkins-ci.org/commit/jenkins/13ae5c793663976587a06d3dca585f7559173efb Log: JENKINS-15846 Deadlock during Maven builds Parsing POM step Do not use anymore shaded version of wagon-http as it we can control commons-logging version

          Code changed in jenkins
          User: Olivier Lamy
          Path:
          changelog.html
          http://jenkins-ci.org/commit/jenkins/2995c2869d5bdde84dd83976e5df1b63091acbf0
          Log:
          JENKINS-15846 add changelog entry

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Olivier Lamy Path: changelog.html http://jenkins-ci.org/commit/jenkins/2995c2869d5bdde84dd83976e5df1b63091acbf0 Log: JENKINS-15846 add changelog entry

          dogfood added a comment -

          Integrated in jenkins_main_trunk #2675
          JENKINS-15846 Deadlock during Maven builds Parsing POM step (Revision 13ae5c793663976587a06d3dca585f7559173efb)
          JENKINS-15846 add changelog entry (Revision 2995c2869d5bdde84dd83976e5df1b63091acbf0)

          Result = SUCCESS
          Olivier Lamy : 13ae5c793663976587a06d3dca585f7559173efb
          Files :

          • pom.xml
          • core/pom.xml
          • maven-plugin/pom.xml

          Olivier Lamy : 2995c2869d5bdde84dd83976e5df1b63091acbf0
          Files :

          • changelog.html

          dogfood added a comment - Integrated in jenkins_main_trunk #2675 JENKINS-15846 Deadlock during Maven builds Parsing POM step (Revision 13ae5c793663976587a06d3dca585f7559173efb) JENKINS-15846 add changelog entry (Revision 2995c2869d5bdde84dd83976e5df1b63091acbf0) Result = SUCCESS Olivier Lamy : 13ae5c793663976587a06d3dca585f7559173efb Files : pom.xml core/pom.xml maven-plugin/pom.xml Olivier Lamy : 2995c2869d5bdde84dd83976e5df1b63091acbf0 Files : changelog.html

          Jesse Glick added a comment -

          I guess that should fix the issue, right?

          Jesse Glick added a comment - I guess that should fix the issue, right?

          I have to double-check on my side, but if commons-logging is really fixed, that should be good

          Vincent Latombe added a comment - I have to double-check on my side, but if commons-logging is really fixed, that should be good

          Code changed in jenkins
          User: Olivier Lamy
          Path:
          pom.xml
          http://jenkins-ci.org/commit/maven-plugin/3d7f55641ab5563b83ca7da439da3e76ba356ecb
          Log:
          JENKINS-15846 Deadlock during Maven builds Parsing POM step
          Do not use anymore shaded version of wagon-http as it we can control commons-logging version

          Originally-Committed-As: 13ae5c793663976587a06d3dca585f7559173efb

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Olivier Lamy Path: pom.xml http://jenkins-ci.org/commit/maven-plugin/3d7f55641ab5563b83ca7da439da3e76ba356ecb Log: JENKINS-15846 Deadlock during Maven builds Parsing POM step Do not use anymore shaded version of wagon-http as it we can control commons-logging version Originally-Committed-As: 13ae5c793663976587a06d3dca585f7559173efb

            olamy Olivier Lamy
            vlatombe Vincent Latombe
            Votes:
            3 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: