[JENKINS-61745] Scan multibranch with date range or dop '@now' - Jenkins Jira

XML

Word

Printable

Details

Type: Improvement
Status: Closed (View Workflow)
Priority: Blocker
Resolution: Fixed
Component/s: p4-plugin
Labels:
- P4_VERIFY
Environment:
p4-plugin 10.1.9

Similar Issues:

Show

Description

When scanning for changes in multibranch pipelines we use p4 changes -m1 //Path/...@now. This is triggering a similar resource behavior to:

~~JENKINS-57870~~

For example with a big dataset in the multibranch scan log we see:

p4 changes -m1 //Path/branch/...@now
Request too large (over 500000); see 'p4 help maxresults'.
ERROR: [Mon Mar 30 13:07:05 CEST 2020] Could not fetch branches from source d5fe37f7-6c40-4111-af23-d4f539fec120
com.perforce.p4java.exception.RequestException: Request too large (over 500000); see 'p4 help maxresults'.

When checking this at the command line we see that limiting the query or dropping '@now' works:

$ p4 changes -m1 //path/branch/...@now
Request too large (over 500000); see 'p4 help maxresults'.$ p4 changes -m1 //path/branch/...@2020/03/30
Request too large (over 500000); see 'p4 help maxresults'.$ p4 changes -m1 //path/branch/...
Change 62552 on 2020/03/30 by user@PC1 'CL Description'$ p4 changes -m1 //path/branch/...@52552,62552
Change 62552 on 2020/03/30 by user@PC1 'CL Description'

Would be great if we could either drop the usage of '@now' or restrict to the last n changes.

Attachments

Issue Links

is related to

JENKINS-57870 Request too large for server memory since 10.1.0

Closed

Activity

Ascending order - Click to sort in descending order

Paul Allen added a comment - 2020-03-31 08:09

I doubt the p4d is scanning the wrong way (the -m1 flag should be minimal load). The Build user should not have MaxResults set; perhaps MaxLockTime would be more appropriate or run off a replica to minimise load.

Notes: Possibly apply ConnectionHelper::getHeadLimit() to AbstractP4ScmSource::findLatestChange(...)

Paul Allen added a comment - 2020-03-31 08:09 I doubt the p4d is scanning the wrong way (the -m1 flag should be minimal load). The Build user should not have MaxResults set; perhaps MaxLockTime would be more appropriate or run off a replica to minimise load. Notes: Possibly apply ConnectionHelper::getHeadLimit() to AbstractP4ScmSource::findLatestChange(...)

Paul Allen added a comment - 2020-04-01 10:19

Added fix: https://github.com/jenkinsci/p4-plugin/commit/62c83c4368a10a2538a04f1988ae59c9c00d11cc

Paul Allen added a comment - 2020-04-01 10:19 Added fix: https://github.com/jenkinsci/p4-plugin/commit/62c83c4368a10a2538a04f1988ae59c9c00d11cc

Paul Allen added a comment - 2020-04-01 10:31

Latest build: https://ci.jenkins.io/job/Plugins/job/p4-plugin/job/master/446/

Paul Allen added a comment - 2020-04-01 10:31 Latest build: https://ci.jenkins.io/job/Plugins/job/p4-plugin/job/master/446/

Paul Allen added a comment - 2020-06-24 15:57

Release in 1.10.13

Paul Allen added a comment - 2020-06-24 15:57 Release in 1.10.13

Ilkka Halila added a comment - 2020-11-10 09:19

Hi,

This change has caused a problem for us. We have a multibranch pipeline set up for a project and as soon as a stream's last revision is 1000 revisions older than the server's latest (this corresponds to DEFAULT_HEAD_LIMIT in PerforceSCM.java) the multibranch scanning will not find any changes for that stream, default to using the "latest" revision ie. the whole server's latest, and triggers a build. You can see this in a snippet from our multibranch "Scan Multibranch Pipeline Log":

... p4 streams //Project/... +
... p4 login -s +
... p4 client -o jenkinsTemp-5189b562-c22b-4b8c-a28d-bd862e2c51d6 +
... p4 client -i +
... p4 client -o jenkinsTemp-5189b562-c22b-4b8c-a28d-bd862e2c51d6 +
... p4 info +
... p4 info +
... p4 client -o jenkinsTemp-5189b562-c22b-4b8c-a28d-bd862e2c51d6 +
... p4 client -i +
... View: +
... p4 counter change +
... p4 counter change +
... p4 changes -m1 -ssubmitted //jenkinsTemp-5189b562-c22b-4b8c-a28d-bd862e2c51d6/...@1___ +
P4: no revisions under //jenkinsTemp-5189b562-c22b-4b8c-a28d-bd862e2c51d6/...@149283,150283 using change: 150283
Scanning for //Project/some_stream/Jenkinsfile
... p4 files -e //jenkinsTemp-5189b562-c22b-4b8c-a28d-bd862e2c51d6/Jenkinsfile +
 ‘Jenkinsfile’ found
Changes detected: some_stream (149129 → 150283)

What this means in practice is that a lot of streams with no new changes are being triggered for every single change to our Perforce server and clogging up our build pipeline with builds that should not have been triggered in the first place. We have had to disable our multibranch pipeline to prevent this.

It seems to me that the plugin should default to not building if there are no revisions in the range instead of defaulting to triggering a build every time?

Ilkka Halila added a comment - 2020-11-10 09:19 Hi, This change has caused a problem for us. We have a multibranch pipeline set up for a project and as soon as a stream's last revision is 1000 revisions older than the server's latest (this corresponds to DEFAULT_HEAD_LIMIT in PerforceSCM.java) the multibranch scanning will not find any changes for that stream, default to using the "latest" revision ie. the whole server's latest, and triggers a build. You can see this in a snippet from our multibranch "Scan Multibranch Pipeline Log": ... p4 streams //Project/... + ... p4 login -s + ... p4 client -o jenkinsTemp-5189b562-c22b-4b8c-a28d-bd862e2c51d6 + ... p4 client -i + ... p4 client -o jenkinsTemp-5189b562-c22b-4b8c-a28d-bd862e2c51d6 + ... p4 info + ... p4 info + ... p4 client -o jenkinsTemp-5189b562-c22b-4b8c-a28d-bd862e2c51d6 + ... p4 client -i + ... View: + ... p4 counter change + ... p4 counter change + ... p4 changes -m1 -ssubmitted //jenkinsTemp-5189b562-c22b-4b8c-a28d-bd862e2c51d6/...@1___ + P4: no revisions under //jenkinsTemp-5189b562-c22b-4b8c-a28d-bd862e2c51d6/...@149283,150283 using change: 150283 Scanning for //Project/some_stream/Jenkinsfile ... p4 files -e //jenkinsTemp-5189b562-c22b-4b8c-a28d-bd862e2c51d6/Jenkinsfile + ‘Jenkinsfile’ found Changes detected: some_stream (149129 → 150283) What this means in practice is that a lot of streams with no new changes are being triggered for every single change to our Perforce server and clogging up our build pipeline with builds that should not have been triggered in the first place. We have had to disable our multibranch pipeline to prevent this. It seems to me that the plugin should default to not building if there are no revisions in the range instead of defaulting to triggering a build every time?

Stuart Rowe added a comment - 2020-11-10 18:04

ihalila_ser we have the same issue. Could you create a new issue?

Stuart Rowe added a comment - 2020-11-10 18:04 ihalila_ser we have the same issue. Could you create a new issue?

Ilkka Halila added a comment - 2020-11-11 08:35

Created a bug to track the issue: https://issues.jenkins.io/browse/JENKINS-64193

Ilkka Halila added a comment - 2020-11-11 08:35 Created a bug to track the issue: https://issues.jenkins.io/browse/JENKINS-64193

Karl Wirth added a comment - 2020-11-11 09:24

Hi stuartrowe - Does increasing 'Head change query limit' woraround the behavior?

Jenkins > Manage Jenkins > Configure System > Head change query limit

Regards,

Karl

Karl Wirth added a comment - 2020-11-11 09:24 Hi stuartrowe - Does increasing 'Head change query limit' woraround the behavior? Jenkins > Manage Jenkins > Configure System > Head change query limit Regards, Karl

People

Assignee:: Paul Allen

Reporter:: Karl Wirth

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 2020-03-30 15:53

Updated:: 2020-11-11 09:24

Resolved:: 2020-06-24 15:57