Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46460

Performance issue in AsyncResourceDisposer.persist

    XMLWordPrintable

Details

    Description

      A thread dump of a Jenkins system suffering very poor performance showed many threads blocked in AsyncResourceDisposer.persist, here:

      "Computer.threadPoolForRemoting [#...]" ... waiting for monitor entry [...]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at com.thoughtworks.xstream.core.DefaultConverterLookup.lookupConverterForType(DefaultConverterLookup.java:49)
      	- waiting to lock <...> (a com.thoughtworks.xstream.core.DefaultConverterLookup)
      	at com.thoughtworks.xstream.XStream$1.lookupConverterForType(XStream.java:498)
      	at ...
      	at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
      	at hudson.XmlFile.write(XmlFile.java:170)
      	at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.persist(AsyncResourceDisposer.java:175)
      	at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.access$300(AsyncResourceDisposer.java:65)
      	at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer$WorkItem.run(AsyncResourceDisposer.java:262)
      	at ...
      

      This is written poorly and causing massive contention on a single lock in XStream; it should be coalescing calls to persist, and preferably even delaying them by a few seconds' grace period to try to batch up as much as possible.

      The workaround is to remove this plugin. It is typically used by ws-cleanup; better to, say, use the CleanBeforeCheckout option in Git.

      Attachments

        Issue Links

          Activity

            Released on 0.8.

            olivergondza Oliver Gondža added a comment - Released on 0.8.

            After second though, I do not think the the benefit of having the lastState persisted reliably is worth the price. It was excluded from the persistence altogether (comment indicated I wanted to do that a long time ago) and every run of WorkItem does not initiate save any longer.

            olivergondza Oliver Gondža added a comment - After second though, I do not think the the benefit of having the lastState persisted reliably is worth the price. It was excluded from the persistence altogether (comment indicated I wanted to do that a long time ago) and every run of WorkItem does not initiate save any longer.

            Another approach would be to avoid persisting when the lastState have not changed after dispose was called.

            olivergondza Oliver Gondža added a comment - Another approach would be to avoid persisting when the lastState have not changed after dispose was called.
            jglick Jesse Glick added a comment -

            After some digging, found another case of a system with >4k threads blocked in this way.

            jglick Jesse Glick added a comment - After some digging, found another case of a system with >4k threads blocked in this way.
            jglick Jesse Glick added a comment -

            Problem likely aggravated by locks from JENKINS-19561.

            jglick Jesse Glick added a comment - Problem likely aggravated by locks from  JENKINS-19561 .

            People

              olivergondza Oliver Gondža
              jglick Jesse Glick
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: