-
Bug
-
Resolution: Unresolved
-
Critical
-
Operating System :
LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 6.8 (Santiago)
Release: 6.8
Codename: Santiago
JRE/JDK vendors and versions : 1.8.0_45
Jenkins versions : 2.46.2
Gerrit grigger plugin versions : 2.27.5
pipeline plugin version : 2.5
script-security plugin version : 1.44
running Jenkins directly or in a container : Jenkins is running on a Virtual machine on tomcat.
How you installed Jenkins : jenkins.war file to install Jenkins.
how you're launching any involved slave nodes : yes, we have slaves using ssh.
Your web browser : Crome Version 67.0.3396.99 (Official Build) (64-bit)Operating System : LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch Distributor ID: RedHatEnterpriseServer Description: Red Hat Enterprise Linux Server release 6.8 (Santiago) Release: 6.8 Codename: Santiago JRE/JDK vendors and versions : 1.8.0_45 Jenkins versions : 2.46.2 Gerrit grigger plugin versions : 2.27.5 pipeline plugin version : 2.5 script-security plugin version : 1.44 running Jenkins directly or in a container : Jenkins is running on a Virtual machine on tomcat. How you installed Jenkins : jenkins.war file to install Jenkins. how you're launching any involved slave nodes : yes, we have slaves using ssh. Your web browser : Crome Version 67.0.3396.99 (Official Build) (64-bit)
We are experiencing a delay in Gerrit triggered Jobs in our Jenkins Jobs.
attached is the stack trace of the blocking threads.
Jul 18, 2018 5:07:57 PM com.sonymobile.tools.gerrit.gerritevents.GerritHandler checkQueueSize
WARNING: The Gerrit incoming events queue contains 28095 items! Something might be stuck, or your system can't process the commands fast enough. Try to increase the number of receiving worker threads. Current thread-pool size: 30
Jul 18, 2018 6:54:37 PM com.sonymobile.tools.gerrit.gerritevents.GerritJsonEventFactory getEvent
FINE: Constructor with JSONObject as parameter missing, trying default constructor.
java.lang.NoSuchMethodException: com.sonymobile.tools.gerrit.gerritevents.dto.events.RefUpdated.<init>(net.sf.json.JSONObject)
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.getConstructor(Class.java:1825)
at com.sonymobile.tools.gerrit.gerritevents.GerritJsonEventFactory.getEvent(GerritJsonEventFactory.java:69)
at com.sonymobile.tools.gerrit.gerritevents.workers.AbstractJsonObjectWork.perform(AbstractJsonObjectWork.java:69)
at com.sonymobile.tools.gerrit.gerritevents.workers.StreamEventsStringWork.perform(StreamEventsStringWork.java:67)
at com.sonymobile.tools.gerrit.gerritevents.workers.EventThread.run(EventThread.java:66)
at com.sonyericsson.hudson.plugins.gerrit.trigger.SystemEventThread.run(SystemEventThread.java:66)
[JENKINS-52636] Gerrit triggered jobs getting delayed
Amit Sharma
created issue -
Audrey Azra
made changes -
| Attachment | New: plugins.txt [ 43428 ] |
Amit Sharma
made changes -
| Attachment | Original: fem019-eiffel003_td_1.txt [ 43427 ] |
Amit Sharma
made changes -
| Attachment | Original: fem019-eiffel003_td_3.txt [ 43426 ] |
Amit Sharma
made changes -
| Attachment | New: fem019-eiffel003_td__1.txt [ 43507 ] |
Amit Sharma
made changes -
| Attachment | New: fem019-eiffel003_td__3.txt [ 43508 ] |
I investigated the issue as I am experiencing it in my own jenkins server and here are my findings.
The issue only happens when Gerrit-Trigger is connected to a Gerrit server with a lot of activity(events). Even if Gerrit-Trigger uses a thread pool(with 3 threads by default) to process events, it cannot keep up, the queue of events to process grows until the build up causes delays in triggering the builds(can be few hours).
Cranking up the number of threads does not really help as there is a major bottleneck in the Gerrit-Trigger code. The bottleneck is in GerritMissedEventsPlaybackManager.persist method. This method serialize to disk the last event received in order for the playback manager to know what is the last event processed and retrigger missed event while Gerrit-Trigger was down in case it goes down or the connection towards Gerrit is lost. The problem is that method is synchronized and called by the threads processing the events. So even if you have 100 receiving threads, they will all bottleneck in that method which is very slow as it does IO, this is even worst if Jenkins home is on NFS or any kind of slow volume.
The stack trace in the description is not right, here is the stack traces of a blocked thread followed by the stack trace of the thread holding the lock:
"Gerrit Worker EventThread_0" Id=62 Group=main BLOCKED on com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager@24996aa8 owned by "Gerrit Worker EventThread_1" Id=63 at com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager.persist(GerritMissedEventsPlaybackManager.java:455) - blocked on com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager@24996aa8 at com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager.gerritEvent(GerritMissedEventsPlaybackManager.java:300) at com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListener(GerritHandler.java:350) at com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListeners(GerritHandler.java:317) at com.sonyericsson.hudson.plugins.gerrit.trigger.JenkinsAwareGerritHandler.notifyListeners(JenkinsAwareGerritHandler.java:77) at com.sonymobile.tools.gerrit.gerritevents.workers.AbstractGerritEventWork.perform(AbstractGerritEventWork.java:46) at com.sonymobile.tools.gerrit.gerritevents.workers.AbstractJsonObjectWork.perform(AbstractJsonObjectWork.java:77) at com.sonymobile.tools.gerrit.gerritevents.workers.StreamEventsStringWork.perform(StreamEventsStringWork.java:67) at com.sonymobile.tools.gerrit.gerritevents.workers.EventThread.run(EventThread.java:66) at com.sonyericsson.hudson.plugins.gerrit.trigger.SystemEventThread.run(SystemEventThread.java:66)
"Gerrit Worker EventThread_1" Id=63 Group=main RUNNABLE at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromItemType(MapperWrapper.java:88) at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromItemType(MapperWrapper.java:88) at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromItemType(MapperWrapper.java:88) at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromItemType(MapperWrapper.java:88) at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromItemType(MapperWrapper.java:88) at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromItemType(MapperWrapper.java:88) at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromItemType(MapperWrapper.java:88) at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromItemType(MapperWrapper.java:88) at hudson.util.xstream.MapperDelegate.getConverterFromItemType(MapperDelegate.java:103) at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromItemType(MapperWrapper.java:88) at hudson.util.RobustReflectionConverter$1.visit(RobustReflectionConverter.java:195) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:191) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88) at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64) at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88) at com.thoughtworks.xstream.converters.reflection.SerializableConverter$1.defaultWriteObject(SerializableConverter.java:214) at com.thoughtworks.xstream.core.util.CustomObjectOutputStream.defaultWriteObject(CustomObjectOutputStream.java:80) at java.util.Collections$SynchronizedCollection.writeObject(Collections.java:2081) - locked java.util.Collections$SynchronizedList@424b9a87 at sun.reflect.GeneratedMethodAccessor151.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callWriteObject(SerializationMethodInvoker.java:135) at com.thoughtworks.xstream.converters.reflection.SerializableConverter.doMarshal(SerializableConverter.java:259) at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshal(AbstractReflectionConverter.java:83) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88) at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshal(AbstractReflectionConverter.java:81) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015) at com.thoughtworks.xstream.XStream.toXML(XStream.java:988) at hudson.XmlFile.write(XmlFile.java:193) at com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager.persist(GerritMissedEventsPlaybackManager.java:492) - locked com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager@24996aa8 at com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager.gerritEvent(GerritMissedEventsPlaybackManager.java:300) at com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListener(GerritHandler.java:350) at com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListeners(GerritHandler.java:317) at com.sonyericsson.hudson.plugins.gerrit.trigger.JenkinsAwareGerritHandler.notifyListeners(JenkinsAwareGerritHandler.java:77) at com.sonymobile.tools.gerrit.gerritevents.workers.AbstractGerritEventWork.perform(AbstractGerritEventWork.java:46) at com.sonymobile.tools.gerrit.gerritevents.workers.AbstractJsonObjectWork.perform(AbstractJsonObjectWork.java:77) at com.sonymobile.tools.gerrit.gerritevents.workers.StreamEventsStringWork.perform(StreamEventsStringWork.java:67) at com.sonymobile.tools.gerrit.gerritevents.workers.EventThread.run(EventThread.java:66) at com.sonyericsson.hudson.plugins.gerrit.trigger.SystemEventThread.run(SystemEventThread.java:66)
| Component/s | Original: pipeline [ 21692 ] | |
| Component/s | Original: script-security-plugin [ 18520 ] |
To add more info to this, issue started happening after several plugins were upgraded (Pipeline, script security, junit...)
Enclosed list of installed plugins
Issue also encountered on other Jenkins instances running different Jenkins core/gerrit-trigger plugin: Core version 1.651.3 & gerrit trigger version - 2.18.3