-
Bug
-
Resolution: Fixed
-
Major
-
jenkins: 2.176.3
gerrit trigger: 2.30
After upgrade gerrit trigger to 2.30, sometimes gerrit trigger can not trigger jenkins jobs, but the jenkins account can receive the gerrit stream events at the same time. We find a lot of WARNING logs in jenkins log,
The Gerrit incoming events queue contains 6389 items! Something might be stuck, or your system can't process the commands fast enough. Try to increase the number of receiving worker threads. Current thead-pool size: 10.
By the way, we set gerrit-trigger only listening to change-merged and comment-added event types. Even we restart jenkins server, gerrit-trigger seems still stuck.
- duplicates
-
-
- In Review
-
[JENKINS-59215] gerrit trigger stuck and can not trigger jobs
I found the same problem and found a way to reproduce. For me it appears when I have a
- disabled job at jenkins start time
- configured with dynamic trigger
If the job is enabled and jenkins start time, I don't see the problem, even if I disable it later. If the job is disabled at start time, the problem occurs and still stays even if I enable the job later. If I remove the dynamic trigger configuration, the problem disappears.
I can reproduce the problem with 2.29.0 and 2.30.0, didn't see it with 2.27.X
Thorsten Möllers
added a comment - - edited I found the same problem and found a way to reproduce. For me it appears when I have a
disabled job at jenkins start time
configured with dynamic trigger
If the job is enabled and jenkins start time, I don't see the problem, even if I disable it later. If the job is disabled at start time, the problem occurs and still stays even if I enable the job later. If I remove the dynamic trigger configuration, the problem disappears.
I can reproduce the problem with 2.29.0 and 2.30.0, didn't see it with 2.27.X 
Charles Tudor
added a comment - Ran into the same problem on 2.29 and 2.30. Went back to 2.28 and things are working as expected.
Charles Tudor
added a comment - Ran into the same problem on 2.29 and 2.30. Went back to 2.28 and things are working as expected. At least the issue described by having a dynamic trigger job disabled on startup is caused by https://github.com/jenkinsci/gerrit-trigger-plugin/pull/350
The executors are waiting for the project to be ready, i.e. the dynamic trigger to be loaded. However, the the GerritTriggerTimerTask does a couple of checks (including job.isBuildable()) before updateTriggerConfigURL. This means the latch introduced in that pull request is never released and the executors are blocked forever.
Possible fix: https://github.com/jenkinsci/gerrit-trigger-plugin/pull/403
Georg Bremer
added a comment - - edited At least the issue described by having a dynamic trigger job disabled on startup is caused by https://github.com/jenkinsci/gerrit-trigger-plugin/pull/350
The executors are waiting for the project to be ready, i.e. the dynamic trigger to be loaded. However, the the GerritTriggerTimerTask does a couple of checks (including job.isBuildable()) before updateTriggerConfigURL. This means the latch introduced in that pull request is never released and the executors are blocked forever.
Possible fix: https://github.com/jenkinsci/gerrit-trigger-plugin/pull/403 
Max P
added a comment - This is the same as JENKINS-56528 ? Georg Bremer
added a comment - From the description (despite being a bit vague about the actual error message) it looks exactly like this issue, it also matches what we have seen.
Georg Bremer
added a comment - From the description (despite being a bit vague about the actual error message) it looks exactly like this issue, it also matches what we have seen. Georg Bremer
added a comment - Symptoms match JENKINS-56528, closing this as duplicate since the other one is older.
Georg Bremer
added a comment - Symptoms match JENKINS-56528 , closing this as duplicate since the other one is older.
Exactly the same problem has found on Jenkins 2.176.2 and gerrit trigger plugin 2.30. Downgraded to 2.27.7 helped.