Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52636

Gerrit triggered jobs getting delayed

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Critical
    • Resolution: Unresolved
    • gerrit-trigger-plugin

    Description

      We are experiencing a delay in Gerrit triggered Jobs in our Jenkins Jobs.

      attached is the stack trace of the blocking threads.

      Jul 18, 2018 5:07:57 PM com.sonymobile.tools.gerrit.gerritevents.GerritHandler checkQueueSize
      WARNING: The Gerrit incoming events queue contains 28095 items! Something might be stuck, or your system can't process the commands fast enough. Try to increase the number of receiving worker threads. Current thread-pool size: 30

       

      Jul 18, 2018 6:54:37 PM com.sonymobile.tools.gerrit.gerritevents.GerritJsonEventFactory getEvent
      FINE: Constructor with JSONObject as parameter missing, trying default constructor.
      java.lang.NoSuchMethodException: com.sonymobile.tools.gerrit.gerritevents.dto.events.RefUpdated.<init>(net.sf.json.JSONObject)
      at java.lang.Class.getConstructor0(Class.java:3082)
      at java.lang.Class.getConstructor(Class.java:1825)
      at com.sonymobile.tools.gerrit.gerritevents.GerritJsonEventFactory.getEvent(GerritJsonEventFactory.java:69)
      at com.sonymobile.tools.gerrit.gerritevents.workers.AbstractJsonObjectWork.perform(AbstractJsonObjectWork.java:69)
      at com.sonymobile.tools.gerrit.gerritevents.workers.StreamEventsStringWork.perform(StreamEventsStringWork.java:67)
      at com.sonymobile.tools.gerrit.gerritevents.workers.EventThread.run(EventThread.java:66)
      at com.sonyericsson.hudson.plugins.gerrit.trigger.SystemEventThread.run(SystemEventThread.java:66)

       

       

       

      Attachments

        Activity

          dsoprea Dustin Oprea added a comment -

          Any status on this? There's a chance that my team just encountered this.

           

          dsoprea Dustin Oprea added a comment - Any status on this? There's a chance that my team just encountered this.  

          It doesn't look like there's been any action on the PR yet although it mostly looks good to go from here.

          In the meantime, I have a patched version we're using at the company I'm at. You're free to use while we wait: https://github.com/DarrienG/gerrit-trigger-plugin/releases/tag/2.31.0-uninterested

          It's basically HEAD from the official repo + ignores irrelevant events. We have probably 3000+ builds a day and haven't seen any issues with it while we wait. Jenkins was unusable for us otherwise.

          darrien Darrien Glasser added a comment - It doesn't look like there's been any action on the PR yet although it mostly looks good to go from here. In the meantime, I have a patched version we're using at the company I'm at. You're free to use while we wait: https://github.com/DarrienG/gerrit-trigger-plugin/releases/tag/2.31.0-uninterested It's basically HEAD from the official repo + ignores irrelevant events. We have probably 3000+ builds a day and haven't seen any issues with it while we wait. Jenkins was unusable for us otherwise.
          dsoprea Dustin Oprea added a comment - - edited

          We tried bumping to 2.29.0 during a system upgrade, and the queue started wildly accumulating without actually starting any builds. We reverted the plugin to 2.27.5 and things appear to be rolling again.

          dsoprea Dustin Oprea added a comment - - edited We tried bumping to 2.29.0 during a system upgrade, and the queue started wildly accumulating without actually starting any builds. We reverted the plugin to 2.27.5 and things appear to be rolling again.
          lemonjia Jia Jia added a comment -

          Any status on this? There's also a chance that my team just encountered this recently.

          lemonjia Jia Jia added a comment - Any status on this? There's also a chance that my team just encountered this recently.

          Recently 2.30.0 was released that include two changes meant to reduce queue load. One reduces the disk writing when playback is enabled and the other gives you the ability to filter out unnecessary gerrit messages from the main settings panel under advanced. Some may still be experiencing delays and queue build ups though.

          ezcorch Christoffer Cortes Sjöwall added a comment - Recently 2.30.0 was released that include two changes meant to reduce queue load. One reduces the disk writing when playback is enabled and the other gives you the ability to filter out unnecessary gerrit messages from the main settings panel under advanced. Some may still be experiencing delays and queue build ups though.

          People

            scoheb Scott Hebert
            eattsma Amit Sharma
            Votes:
            12 Vote for this issue
            Watchers:
            25 Start watching this issue

            Dates

              Created:
              Updated: