Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-13154

Heavy thread congestion with FingerPrint.save

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Major
    • Resolution: Fixed
    • core

    Description

      We are seeing repeatable very heavy lock congestion with FingerPrint.save.

      Jenkins apparently slides down to a state where a lot of threads have locked / are competing for a lock on FingerPrint instance and are competing for a lock on AnnotationMapper (a singleton). Everything grings to a halt.

      Attachments

        Issue Links

          Activity

            jglick Jesse Glick added a comment -

            @edenman: separate issue please. (Can use JIRA’s “related” link if appropriate.) Your thread dump anyway does not show Jenkins being blocked, but rather excessively busy doing a couple different things: loading fingerprints; and loading historical build records (perhaps after having discarded them under memory pressure).

            jglick Jesse Glick added a comment - @edenman: separate issue please. (Can use JIRA’s “related” link if appropriate.) Your thread dump anyway does not show Jenkins being blocked, but rather excessively busy doing a couple different things: loading fingerprints; and loading historical build records (perhaps after having discarded them under memory pressure).
            edenman Eric Denman added a comment -

            @jglick thanks! Filed JENKINS-17412

            edenman Eric Denman added a comment - @jglick thanks! Filed JENKINS-17412

            We are still seeing lots of congestion with Jenkins 1.517 and our builds may still stay tens of minutes in "Waiting for Jenkins to finish collecting data" -phase.

            The place has changed though, now thread dumps show lots of these:

            java.lang.Thread.State: BLOCKED (on object monitor)
            at java.util.Collections$SynchronizedMap.get(Collections.java:2031)

            • waiting to lock <0x000000070c6dcfa8> (a java.util.Collections$SynchronizedMap)
              at com.thoughtworks.xstream.core.DefaultConverterLookup.lookupConverterForType(DefaultConverterLookup.java:49)
            mp3 Mikko Peltonen added a comment - We are still seeing lots of congestion with Jenkins 1.517 and our builds may still stay tens of minutes in "Waiting for Jenkins to finish collecting data" -phase. The place has changed though, now thread dumps show lots of these: java.lang.Thread.State: BLOCKED (on object monitor) at java.util.Collections$SynchronizedMap.get(Collections.java:2031) waiting to lock <0x000000070c6dcfa8> (a java.util.Collections$SynchronizedMap) at com.thoughtworks.xstream.core.DefaultConverterLookup.lookupConverterForType(DefaultConverterLookup.java:49)

            Attached relevant parts of the whole thread dump.

            mp3 Mikko Peltonen added a comment - Attached relevant parts of the whole thread dump.
            jglick Jesse Glick added a comment -

            @mp3 whatever you are seeing is a distinct bug. Please file separately and use JIRA’s “is related to” link as needed.

            (I would have suspected a regression from JENKINS-18775, but that is much newer than 1.517.)

            jglick Jesse Glick added a comment - @mp3 whatever you are seeing is a distinct bug. Please file separately and use JIRA’s “is related to” link as needed. (I would have suspected a regression from JENKINS-18775 , but that is much newer than 1.517.)

            People

              jglick Jesse Glick
              t_kurki Teppo Kurki
              Votes:
              4 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: