Hi atwatsoniii

Please can you show my how you are specifying the changelist in the sync. Is this using pipeline/multibranch pipeline.

Thanks in advance.

Using declarative pipeline:

checkout perforce(

populate:

syncOnly(
force: false,
have: true,
modtime: false,
parallel: [enable: true, minbytes: '1024', minfiles: '1', threads: '4'],
pin: "${env.CHANGELIST}",
quiet: true,
revert: true)

Hi atwatsoniii,

Thanks. Why are you trying to pin a polled job? Are you trying to implement 'pin at polled' - JENKINS-63879?

Usually you want polling to find the highest changeslist and then sync to that highest changelist which is the way the code expects. The main reason you dont want to is when the job takes a long time to run on a busy system and contains multiple sync steps that need to be consistent which is what the above enhancement is trying to implement.

Does your job have one or more than one sync/checkout step? If more than one sync step do you need polling for all the code paths including the pinned one?

Regards,

Karl

I only have one sync step that I want to do polling on, and that's the example above.   However, as an example, I may initiate a manual run at a specific changelist.  In the current paradigm, doing that will disable polling until a run is made without "pin".   What is the reason behind the current methodology?  It seems this behavior is always undesirable, it effectively disables polling.  

Any updates or questions?

I'm also thinking this isn't much of a 'minor' issue. In my case I have one primary job that triggers a bunch of sub-jobs. The primary job polls and gets the changelist, then passes that to the sub-jobs (which handle their own email/notifications to committers) so they need to use that changelist to get who is responsible. And all the jobs are capable to have changelist passed in as a parameter as well. And they are setup to also be able to run independently.

So I really do need to be able to pass along a changelist and have it not break polling as a result.

I'm going to move this to Major as it means you can result with changes in the 'blame' list that aren't part of a build, as you have to always poll latest even if you are syncing to a specific build.

meaning you have to do a sync with no changelist to get polling working, but if you initiate a build to a specific changelist that will be a seperate sync so polling would keep working

Hi kyates

The way polling works is that it looks in the last succesful build build.xml file and then should look for changes since then. For example I used the Perforce built in label 'now' during build 13. The  contents of JENKINS_HOME/jobs/JOB_NAME/builds/13/build.xml is:

  <refChanges>
        <org.jenkinsci.plugins.p4.changes.P4LabelRef>
          <label>now</label>
        </org.jenkinsci.plugins.p4.changes.P4LabelRef>
 </refChanges>

when I manually run the polling I see:

Found last change now on syncID jenkins-NODE_NAME-PinPolledFreestyle-EXECUTOR_NUMBER

If change 13 had built an actual change instead of a label the build.xml contains that highest change. For example:

      <refChanges>
        <org.jenkinsci.plugins.p4.changes.P4ChangeRef>
          <change>80</change>
        </org.jenkinsci.plugins.p4.changes.P4ChangeRef>
      </refChanges>

So next time it gets stuck can you please get me the 'build.xml' for the last succesful job and the polling log after that which does not find anything.

Thanks in advance,

Karl

p4karl, I provided 288148 and that's what the build.xml shows.

    <org.jenkinsci.plugins.p4.tagging.TagAction plugin="p4@1.11.5">
      <tags/>
      <refChanges>
        <org.jenkinsci.plugins.p4.changes.P4ChangeRef>
          <change>288148</change>
        </org.jenkinsci.plugins.p4.changes.P4ChangeRef>
      </refChanges>

Leaving it as an empty string results in the same changes to build.xml. I use the empty string for the latest, I've never used 'now'.

Here's the polling log even though 288159 has changes, the polling isn't polling anything past the provided changelist.

P4: Polling with range: 288148,288148
...tick...
... p4 changes -m10 //*****/...@288148,288148
 +
... p4 repos -C
 +
P4: Polling no changes found.
P4: Polling no changes found.
no polling baseline in *****

p4karl,

I think the issue is in PollTask.java, this usage of the 'pin' which is how you currently sync to a change in the p4 plugin is by using the pin. I think it shouldn't be using the pin at all and should just as the original reporter suggested, always poll latest.

Because pin is how you sync to a specific change, once you've done it polling will forever try to check for changes with that being the reference and the thing to compare against.

	@Override
	public Object task(ClientHelper p4) throws Exception {
		List<P4Ref> changes = new ArrayList<P4Ref>();

		// find changes...
		if (pin != null && !pin.isEmpty()) {
			changes = p4.listHaveChanges(lastRefs, new P4LabelRef(pin));
		} else {
			changes = p4.listHaveChanges(lastRefs);
		}

It's getting that from PerforceScm.java:

	private List<P4Ref> lookForChanges(FilePath buildWorkspace, Workspace ws, Run<?, ?> lastRun, TaskListener listener)
			throws IOException, InterruptedException {

		// Set EXPANDED client
		String syncID = ws.getSyncID();

		// Set EXPANDED pinned label/change
		String pin = populate.getPin();
		if (pin != null && !pin.isEmpty()) {
			pin = ws.getExpand().format(pin, false);
			ws.getExpand().set(ReviewProp.P4_LABEL.toString(), pin);
		}

		// Calculate last change, build if null (JENKINS-40356)
		List<P4Ref> lastRefs = TagAction.getLastChange(lastRun, listener, syncID);
		if (lastRefs == null || lastRefs.isEmpty()) {
			// no previous build, return null.
			listener.getLogger().println("P4: Polling: No changes in previous build.");
			return null;
		}

		// Create task
		PollTask task = new PollTask(credential, lastRun, listener, filter, lastRefs);
		task.setWorkspace(ws);
		task.setLimit(pin);

		// Execute remote task
		List<P4Ref> changes = buildWorkspace.act(task);
		return changes;
	}

Hi kyates,

Thanks. I was having problems reproducing it, but just managed it. Assigning to the devs. They will review the defect during their next sprint review. I do not know when this will be but I will also highlight this to the product manager.

Reproduction steps for Dev:

(1) Create the following directory structure in Perforce:

//depot/PinPolled2/Jenkinsfile/Jenkinsfile
//depot/PinPolled2/src/f1
//depot/PinPolled2/src/f2

and use the Jenkinsfile:

pipeline {
  agent { label 'master' }
  stages {
    stage("Repro") {
      steps {
        script {
	checkout perforce(credential: 'JenkinsMaster', populate: forceClean(have: false, parallel: [enable: false, minbytes: '1024', minfiles: '1', threads: '4'], pin: params.PINCL, quiet: false), workspace: manualSpec(charset: 'none', cleanup: false, name: 'jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}-SRC', pinHost: false, spec: clientSpec(allwrite: false, backup: true, changeView: '', clobber: true, compress: false, line: 'LOCAL', locked: false, modtime: false, rmdir: false, serverID: '', streamName: '', type: 'WRITABLE', view: '//depot/PinPolled2/src/... //${P4_CLIENT}/...')))
        }
      } 
    }
  }
}

(2) Create a new pipeline job with the string paramater "PINCL" with no default value and Jenkinsfile from SCM source:

//depot/PinPolled2/Jenkinsfile/... //${P4_CLIENT}/...

(3) Manually build the job with no paramater to set up polling.

(4) Submit a new file to 'src' subtree.

(5) Run 'Poll Now' (assumes poll now plugin installed).

(6) Correct changelist found.

(7) Submit two new files to 'src' subtree in seperate changelists.

(8) Manually build the job with paramater set to first changelist from step (7). Therefore we have not built head.

(9) Run 'Poll now' again. The changelist from step (8) is used as the lower and upper limit. For example it was CL 123 (pure coincidence ) in my testing and head revision for 'src' was CL 124:

Perforce Software Polling Log

Started on Jun 9, 2021 11:36:03 AM
Executor number at runtime: 0
P4: Polling on: master with:jenkins-master-PinPolled2JenkinsfileNew-0
Found last change 121 on syncID jenkins-NODE_NAME-PinPolled2JenkinsfileNew-EXECUTOR_NUMBER
... p4 login -s +
... p4 client -o jenkins-master-PinPolled2JenkinsfileNew-0 +
... p4 info +
... p4 info +... p4 client -o jenkins-master-PinPolled2JenkinsfileNew-0 +
P4 Task: establishing connection.
... server: 192.168.1.160:1666
... node: vm-kwirth-swarm202-xenial
P4: Polling with range: 121,now
... p4 changes -m20 //jenkins-master-PinPolled2JenkinsfileNew-0/...@121,now +
... p4 repos -C +
P4: Polling no changes found.

Executor number at runtime: 0
P4: Polling on: master with:jenkins-master-PinPolled2JenkinsfileNew-0-SRC
Found last change 123 on syncID jenkins-NODE_NAME-PinPolled2JenkinsfileNew-EXECUTOR_NUMBER-SRC
... p4 login -s +
... p4 client -o jenkins-master-PinPolled2JenkinsfileNew-0-SRC +
... p4 info +
... p4 info +
... p4 client -o jenkins-master-PinPolled2JenkinsfileNew-0-SRC +
P4 Task: establishing connection.
... server: 192.168.1.160:1666
... node: vm-kwirth-swarm202-xenial
P4: Polling with range: 123,123
... p4 changes -m20 //jenkins-master-PinPolled2JenkinsfileNew-0-SRC/...@123,123 +
... p4 repos -C +P4: Polling no changes found.
Done. Took 55 ms
No changes

Note the "p4 changes -m20 //jenkins-master-PinPolled2JenkinsfileNew-0-SRC/...@123,123".

Any updates?

Ping, Any Updates for this issue?

Have been asked to highlight to developers that this one is still causing problems.

It would be really helpful to have this fixed in the near future.

Hi Karl, could you check with product owner and/or developers whether there is a plan for fixing this?

Hi all here, we are looking at looking at implementing this. The idea behind the methodology adopted here was if you are only ever going to sync to a particular CL in the past, why do want to poll to the latest and keep building the same thing over and over.

Can all the watchers here document a few cases from their own usages, it would help us prioritize this. 
Many thanks in advance!

Please have a look at the summay at JENKINS-63879. The point is that you want to ensure that multiple build paths - as part of the pipeline - will synch to the same source code state.

For my part I have a lot of Linux build nodes and only a few Windows build nodes. So if the pipeline requires to build for both platforms then the Linux part might start early and the Windows part might start an hour later.

To my knowledge there is no way beside using a pinned build to ensure that both parts use the same source code state. And beyond that: if I have later steps depending on the initial build result - e.g. starting a Coverity build or  starting UI tests on a different system only if building itself has passed then I also need to make sure that the same source code state is used.

Okay, one could use stashing ... but that is far too slow with Jenkins. Especially when having a build node inside Azure with a local P4 proxy and the CI server locally.

So ... currently I cannot do incremental builds due to this problem.

Please also see for

• the communication as part of case #00863974 as of March
• JENKINS-71364
• JENKINS-71143

Hi heiko_nardmann we are currently actively working on this and another issue and should be out with a release within the next couple of weeks. Will update this if anything changes. Thanks for the use case and the assist on this.

Hi Saurabh,

after more than a month now since your last statement I wonder whether there are any news wrt. a new plugin release?

Hi heiko_nardmann the developers were stuck with a difficult problem in another product. We are now back to P4 plugin on Jenkins and actively working this. The plan as mentioned earlier is to allow for an option in the SCM checkout step (along with the existing polling related options like polling latest change). While the development effort on this is relatively lesser, we will be spending time testing out the very many scenarios possible. We would like to share a preview build for your feedback as well, if you can share any issues you see, it would be helpful, expect an update in the next week. Thanks!

Feel free to provide me some beta version: I can test this then on the weekend and revert back to the official release in case of trouble.

Luckily many scenarios are already part of the automated test cases.

Hi heiko_nardmann Karl has very kindly created a case for us to share the preview hpi file. Hope you have received it and are able to install it. While we do not expect this fix to have a drastic impact on any environment but of course it would be prudent to try this out in a non-risky scenario. Hoping to hear from you. Many thanks again!

Hi, in P4 plugin 1.14.3 release, we have added a new polling build filter - 'Polling latest change with pin' which when enabled, overrides the pin in the checkout step and polls to 'now'
Closing this

Please don't forget to update the corr. documentation: my search for "latestWithPin" did not show up any hit inside the *.md files.