• 2.266

      Related to JENKINS-13154, we are still seeing lots of congestion with Jenkins 1.517 and our builds may still stay tens of minutes in "Waiting for Jenkins to finish collecting data" -phase.

      The place has changed though, now thread dumps show lots of these:

      java.lang.Thread.State: BLOCKED (on object monitor)
      at java.util.Collections$SynchronizedMap.get(Collections.java:2031)
      waiting to lock <0x000000070c6dcfa8> (a java.util.Collections$SynchronizedMap)
      at com.thoughtworks.xstream.core.DefaultConverterLookup.lookupConverterForType(DefaultConverterLookup.java:49)

          [JENKINS-19561] Unsafe & inefficient concurrency in XStream

          Jesse Glick added a comment -

          AFAICT these are our patches. Note in particular the addition of synchronized to DefaultConverterLookup.lookupConverterForType, which can cause deadlocks and monitor contention.

          Jesse Glick added a comment - AFAICT these are our patches . Note in particular the addition of synchronized to DefaultConverterLookup.lookupConverterForType , which can cause deadlocks and monitor contention.

          Jesse Glick added a comment -

          The above ConcurrentModificationException may actually be some unrelated problem in a non-thread-safe Map-valued field somewhere. Currently RobustMapConverter does not override marshal to catch that and report it in a helpful manner. Looks like the same could be true of RobustCollectionConverter.

          My current thinking on the thread-safe variant would be to subclass XStream2 and override various marshal, unmarshal, toXML, and fromXML overloads to delegate to a pooled instance, initialized with a copy of all various fields like ConverterLookup. Not perfect since calls to alias and the like which happened concurrently with reading or writing could still cause problems, but this is much less likely than race conditions initializing annotation-based metadata. Not sure how to reproduce the original problems in a test, though. Maybe it would suffice to just try replacing the version of XStream in Jenkins with a stock one and seeing what tests fail or flake.

          Jesse Glick added a comment - The above ConcurrentModificationException may actually be some unrelated problem in a non-thread-safe Map -valued field somewhere. Currently RobustMapConverter does not override marshal to catch that and report it in a helpful manner. Looks like the same could be true of RobustCollectionConverter . My current thinking on the thread-safe variant would be to subclass XStream2 and override various marshal , unmarshal , toXML , and fromXML overloads to delegate to a pooled instance, initialized with a copy of all various fields like ConverterLookup . Not perfect since calls to alias and the like which happened concurrently with reading or writing could still cause problems, but this is much less likely than race conditions initializing annotation-based metadata. Not sure how to reproduce the original problems in a test, though. Maybe it would suffice to just try replacing the version of XStream in Jenkins with a stock one and seeing what tests fail or flake.

          Gareth Western added a comment - - edited

          I believe that this is a known issue in the X-Stream library. Hopefully it will be addressed soon.

          Gareth Western added a comment - - edited I believe that this is a known issue in the X-Stream library . Hopefully it will be addressed soon.

          Jesse Glick added a comment -

          gareth_western while there may be bugs in XStream which affect Jenkins, the basic issue here is a bug in Jenkins: it is using XStream in a way that is explicitly listed as unsupported.

          Jesse Glick added a comment - gareth_western while there may be bugs in XStream which affect Jenkins, the basic issue here is a bug in Jenkins: it is using XStream in a way that is explicitly listed as unsupported.

          Jesse Glick added a comment -

          svanoort I consider this a defect, not an “improvement”: Jenkins is violating the XStream usage guidelines at its peril.

          Jesse Glick added a comment - svanoort I consider this a defect, not an “improvement”: Jenkins is violating the XStream usage guidelines at its peril.

          Alexey Shein added a comment -

          Any news on this one? Is there anything that can be done from my (user's) side to help? It causes serious Jenkins degradation issues for us

          Alexey Shein added a comment - Any news on this one? Is there anything that can be done from my (user's) side to help? It causes serious Jenkins degradation issues for us

          Jesse Glick added a comment -

          Is there anything that can be done from my (user's) side to help?

          Not that I can think of.

          Jesse Glick added a comment - Is there anything that can be done from my (user's) side to help? Not that I can think of.

          Jesse Glick added a comment -

          Jesse Glick added a comment - https://x-stream.github.io/annotations-tutorial.html#AutoDetect

          Jesse Glick added a comment -

          jenkins #4944 should address this.

          Jesse Glick added a comment - jenkins #4944 should address this.

          Jesse Glick added a comment -

          The CME in InputStepExecution.postSettlement may have been fixed by https://github.com/jenkinsci/pipeline-plugin/pull/309/commits/52ddff0780f577b911b4780b6f9c1e09e752337c.

          The WorkflowRun#execution looks similar to the error purportedly fixed by https://github.com/jenkinsci/pipeline-plugin/commit/1d5797c49a8f0edb6064b55143614a2da4c207d4; I do not recall seeing anything like it recently.

          Jesse Glick added a comment - The CME in InputStepExecution.postSettlement may have been fixed by https://github.com/jenkinsci/pipeline-plugin/pull/309/commits/52ddff0780f577b911b4780b6f9c1e09e752337c . The WorkflowRun#execution looks similar to the error purportedly fixed by https://github.com/jenkinsci/pipeline-plugin/commit/1d5797c49a8f0edb6064b55143614a2da4c207d4 ; I do not recall seeing anything like it recently.

            jglick Jesse Glick
            mp3 Mikko Peltonen
            Votes:
            7 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved: