Audrey Azra added a comment - 2018-07-18 17:31 - edited To add more info to this, issue started happening after several plugins were upgraded (Pipeline, script security, junit...) Enclosed list of installed plugins Issue also encountered on other Jenkins instances running different Jenkins core/gerrit-trigger plugin: Core version 1.651.3 & gerrit trigger version - 2.18.3

Hugo Ares added a comment - 2018-08-07 15:28 I investigated the issue as I am experiencing it in my own jenkins server and here are my findings. The issue only happens when Gerrit-Trigger is connected to a Gerrit server with a lot of activity(events).  Even if Gerrit-Trigger uses a thread pool(with 3 threads by default) to process events, it cannot keep up, the queue of events to process grows until the build up causes delays in triggering the builds(can be few hours). Cranking up the number of threads does not really help as there is a major bottleneck in the Gerrit-Trigger code. The bottleneck is in GerritMissedEventsPlaybackManager.persist method. This method serialize to disk the last event received in order for the playback manager to know what is the last event processed and retrigger missed event while Gerrit-Trigger was down in case it goes down or the connection towards Gerrit is lost. The problem is that method is synchronized and called by the threads processing the events. So even if you have 100 receiving threads, they will all bottleneck in that method which is very slow as it does IO, this is even worst if Jenkins home is on NFS or any kind of slow volume. The stack trace in the description is not right, here is the stack traces of a blocked thread followed by the stack trace of the thread holding the lock:   "Gerrit Worker EventThread_0" Id=62 Group=main BLOCKED on com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager@24996aa8 owned by "Gerrit Worker EventThread_1" Id=63 at com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager.persist(GerritMissedEventsPlaybackManager.java:455) - blocked on com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager@24996aa8 at com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager.gerritEvent(GerritMissedEventsPlaybackManager.java:300) at com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListener(GerritHandler.java:350) at com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListeners(GerritHandler.java:317) at com.sonyericsson.hudson.plugins.gerrit.trigger.JenkinsAwareGerritHandler.notifyListeners(JenkinsAwareGerritHandler.java:77) at com.sonymobile.tools.gerrit.gerritevents.workers.AbstractGerritEventWork.perform(AbstractGerritEventWork.java:46) at com.sonymobile.tools.gerrit.gerritevents.workers.AbstractJsonObjectWork.perform(AbstractJsonObjectWork.java:77) at com.sonymobile.tools.gerrit.gerritevents.workers.StreamEventsStringWork.perform(StreamEventsStringWork.java:67) at com.sonymobile.tools.gerrit.gerritevents.workers.EventThread.run(EventThread.java:66) at com.sonyericsson.hudson.plugins.gerrit.trigger.SystemEventThread.run(SystemEventThread.java:66)     "Gerrit Worker EventThread_1" Id=63 Group=main RUNNABLE at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromItemType(MapperWrapper.java:88) at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromItemType(MapperWrapper.java:88) at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromItemType(MapperWrapper.java:88) at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromItemType(MapperWrapper.java:88) at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromItemType(MapperWrapper.java:88) at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromItemType(MapperWrapper.java:88) at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromItemType(MapperWrapper.java:88) at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromItemType(MapperWrapper.java:88) at hudson.util.xstream.MapperDelegate.getConverterFromItemType(MapperDelegate.java:103) at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromItemType(MapperWrapper.java:88) at hudson.util.RobustReflectionConverter$1.visit(RobustReflectionConverter.java:195) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:191) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88) at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64) at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88) at com.thoughtworks.xstream.converters.reflection.SerializableConverter$1.defaultWriteObject(SerializableConverter.java:214) at com.thoughtworks.xstream.core.util.CustomObjectOutputStream.defaultWriteObject(CustomObjectOutputStream.java:80) at java.util.Collections$SynchronizedCollection.writeObject(Collections.java:2081) - locked java.util.Collections$SynchronizedList@424b9a87 at sun.reflect.GeneratedMethodAccessor151.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callWriteObject(SerializationMethodInvoker.java:135) at com.thoughtworks.xstream.converters.reflection.SerializableConverter.doMarshal(SerializableConverter.java:259) at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshal(AbstractReflectionConverter.java:83) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88) at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshal(AbstractReflectionConverter.java:81) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015) at com.thoughtworks.xstream.XStream.toXML(XStream.java:988) at hudson.XmlFile.write(XmlFile.java:193) at com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager.persist(GerritMissedEventsPlaybackManager.java:492) - locked com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager@24996aa8 at com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager.gerritEvent(GerritMissedEventsPlaybackManager.java:300) at com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListener(GerritHandler.java:350) at com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListeners(GerritHandler.java:317) at com.sonyericsson.hudson.plugins.gerrit.trigger.JenkinsAwareGerritHandler.notifyListeners(JenkinsAwareGerritHandler.java:77) at com.sonymobile.tools.gerrit.gerritevents.workers.AbstractGerritEventWork.perform(AbstractGerritEventWork.java:46) at com.sonymobile.tools.gerrit.gerritevents.workers.AbstractJsonObjectWork.perform(AbstractJsonObjectWork.java:77) at com.sonymobile.tools.gerrit.gerritevents.workers.StreamEventsStringWork.perform(StreamEventsStringWork.java:67) at com.sonymobile.tools.gerrit.gerritevents.workers.EventThread.run(EventThread.java:66) at com.sonyericsson.hudson.plugins.gerrit.trigger.SystemEventThread.run(SystemEventThread.java:66)

Hugo Ares added a comment - 2018-08-07 19:15 scoheb Any chance you have time to fix this

Fred May added a comment - 2018-08-24 16:24 In the situation above, we have found the vast majority of the Gerrit events relate to "git notes" in the git repos. They are not required for triggering Jenkins jobs. A plugin config option that would allow certain events to be ignored by the plugin would be useful.  

Ganesh Saraf added a comment - 2018-09-03 08:55 scoheb , Can you please let us know by any change you might have looked into this?

Scott Hebert added a comment - 2018-09-03 17:23 hugares fred_may ganeshsrf I will try to look into this shortly. 

Marco Miller added a comment - 2018-09-11 18:38 - edited Thx scoheb ! Btw, a colleague of ours noticed [1] recently, if need be, or there might be other /related PRs too: [1] https://github.com/jenkinsci/gerrit-trigger-plugin/pull/363 Note: not sure how close [1] is to hugares ' well-detailed findings above, though (did not check myself).

Darrien Glasser added a comment - 2019-03-13 18:05 - edited Update, we've been debugging this issue, and it seems like the Replication Plugin may be (somehow) interfering with the missed events playback plugin/event trigger. We're not entirely certain, but it almost looks like the event trigger (that sends builds to Jenkins) is receiving replication events and gets stuck trying to understand what to do with them. We will continue investigating. Disabling the replication plugin reduces our backlog queue to 0 and events are perfectly streamed to Jenkins.

Darrien Glasser added a comment - 2019-03-18 20:44 - edited Our team has confirmed that Gerrit trigger listens to all gerrit events, including ones it cannot act upon. It takes an inordinate time to determine whether or not it is able to act on them, finally only throwing a `NoSuchMethodException` trying to reflect into an object and see if the event has what it is looking for. This wasn't really an issue before, as all data was stored in SQL somewhere. NoteDB stores all data (events included) in refs/changes/* which all generate events that Jenkins listens for. We started using the replication plugin, and added reviewers automatically when a review was posted, and our Jenkins server was inundated almost immediately. As it stands, this makes the gerrit-trigger plugin unusable for anybody using the replication plugin, and as gerrit generates more metadata over time, will make the plugin unusable in all general cases unless it starts only listening to relevant events. We've found this gerrit trigger implemented with pipelines does not have the issue https://github.com/jenkinsci/gerrit-code-review-plugin And will likely be switching to it in the future.

Christoffer Cortes Sjöwall added a comment - 2019-05-06 07:57 - edited I have two PRs related to this issue. https://github.com/jenkinsci/gerrit-trigger-plugin/pull/397 https://github.com/jenkinsci/gerrit-trigger-plugin/pull/398 The first one is to add an event filter to the gerrit event stream and the second one is delegate disk writing to a thread instead of letting the workers have a hold up.

Ganesh Saraf added a comment - 2019-05-15 08:45 Hi Christoffer, That is good news. When we can expect it to get released?  

Dustin Oprea added a comment - 2019-07-27 18:47 Any status on this? There's a chance that my team just encountered this.  

Darrien Glasser added a comment - 2019-07-27 19:02 It doesn't look like there's been any action on the PR yet although it mostly looks good to go from here. In the meantime, I have a patched version we're using at the company I'm at. You're free to use while we wait: https://github.com/DarrienG/gerrit-trigger-plugin/releases/tag/2.31.0-uninterested It's basically HEAD from the official repo + ignores irrelevant events. We have probably 3000+ builds a day and haven't seen any issues with it while we wait. Jenkins was unusable for us otherwise.

Dustin Oprea added a comment - 2019-07-27 19:23 - edited We tried bumping to 2.29.0 during a system upgrade, and the queue started wildly accumulating without actually starting any builds. We reverted the plugin to 2.27.5 and things appear to be rolling again.

Jia Jia added a comment - 2019-08-06 03:18 Any status on this? There's also a chance that my team just encountered this recently.

Christoffer Cortes Sjöwall added a comment - 2019-08-06 08:52 Recently 2.30.0 was released that include two changes meant to reduce queue load. One reduces the disk writing when playback is enabled and the other gives you the ability to filter out unnecessary gerrit messages from the main settings panel under advanced. Some may still be experiencing delays and queue build ups though.