-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins 2.479.3
p4-plugin 1.17.0
We observed that newly created jobs with the p4 trigger aren't started when new changes are submitted, while existing jobs are triggered normally. In the logs there are no "probing" and "poking" messages for the new jobs at all.
After some investigation I believe the root cause is the implementation of doChange in P4Hook.java: whenever the hook is triggered, a polling task is enqueued for the executorService, and the list of jobs to poll is stored at that time, so supposing the polling task take a long time, it's possible that there's a long queue of polling tasks which have an outdated list of jobs, and newly created jobs aren't processed until all polling tasks that were stored before their creation are processed.
We have about 200 jobs with the p4 trigger enabled, and based on "poking" messages in the log, it can take 30 minutes or longer on average to make a round trip and poll the same job twice (which is a problem in itself). If it takes 30 minutes to poll jobs after a single commit (and every commit incurs that work), then processing 100 commits takes more than 2 days, so with a conservative estimate of 100 commits per day, the plugin will always have a growing backlog of polling tasks to work through, before it first "sees" a newly created job.
After restarting Jenkins, the new jobs get properly triggered (presumably because the queue of polling tasks is cleared and the first poll after the restart uses an up to date list of jobs), but jobs created after a few days of uptime will once again have the same issue.
Apparently simply moving the step to get the list of jobs inside the polling task is not a solution, as that exact change is cited as the cause of JENKINS-61156.