Details
-
Bug
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
- OS: RHEL 7.2
- Jenkins Version: 2.46.3
- Gerrit Trigger Plugin Version: 2.23.2
Description
Hello,
Last week we upgraded our Jenkins instance to version 2.46.3 (on/around 05/25/2017). Upon successfully completing/testing that upgrade, we upgraded our Gerrit trigger plug-in version from 2.23.0 -> 2.23.2.
Since performing that Gerrit trigger plug-in upgrade, our Jenkins server has completely stopped processing ANY of the stream events generated by code pushes by our users, period. I've pushed code a number of times myself and the Gerrit trigger plug-in simply ignores all of the events, and this is in spite of the fact that I am monitoring the events coming across myself (in a separate window).
Our configuration is simple; we are not filtering-out any specific event types. This software simply isn't working.
The only way to get things working again was to back-out the 2.23.2 version of the Gerrit trigger plug-in and go back to the 2.23.0 version. Once we bounced the Jenkins server and re-pushed some of the previously-failed changes, everything started working again like clockwork.
Please let me know if there is any additional information I can provide you to help troubleshoot this. We don't have access to the log files on our Gerrit servers (it's centralized in a datacenter in France), but I do have access to our Jenkins server's logs.
I will assist you in any way I can.
I created this tracker as 'Major' because I believe it deserves investigation and prior to the creation of this tracker – literally only an hour ago – it was blocking our ability to deliver our product. I consider the fallback to 2.23.0 to be a fortunate work-around, but it's not optimal. We like to keep our plug-ins updated with the latest versions whenever possible.
Any help you can offer is greatly appreciated!
Kind Regards,
Marcus E. Harris
Attachments
Issue Links
- duplicates
-
-
- Resolved
-
Activity
I was able to obtain GerritHub messages of 78KB in less than a day of saving the event stream so we know at least one of the bugs with the current code. I propose configuring a buffer of at least 256KB.
This should be implemented and released ASAP and after this we should have enough time to rewrite the implementation without having daily outages.
PR at https://github.com/sonyxperiadev/gerrit-events/pull/67 – seems like a good enough reason for a dot release.
Locally my fixed version is running stable since Friday while the buggy version stopped twice since then.
Markus Winter
added a comment - - edited Locally my fixed version is running stable since Friday while the buggy version stopped twice since then.
PR: https://github.com/sonyxperiadev/gerrit-events/pull/68 
Martin Wiklundh
added a comment - Any progress on this? Any way to test some of the suggested solutions?
Martin Wiklundh
added a comment - Any progress on this? Any way to test some of the suggested solutions? Markus Winter
added a comment - The fix is in since 2.25.0 
Actually with the current Unit tests I can easily reproduce the current problem.
In GerritConnectionTest just add the line
connection.setSshRxBufferSize(20);
The test will then never finish, because the position is set after the clear to the capacity of the buffer and we have one event that is bigger than the buffer.
If I remove the change of position after the clear, the test fails because data is lost.