That sounds more like a race condition to me. The second change would trigger job a again and update the counter before job b can grab it. If you increased the quiet period on job A, or set up a lock so A and B can't run at the same time, then I think that should fix the problem.

Alternatively, you can pass the changeset down as a job parameter instead of a counter.

– Posted from Bugbox for Android

Hi Rob,

I initially thought it was a race condition as well but with further tests I think I've got to the bottom of what is happening.

For testing purposes my Jenkins setup has only 1 build executor thread.

Consider this scenario:

1) FileA is checked in.
2) JobA is triggered, this queues up 4 child jobs because it's a matrix project. P4 counter is correct assigned latest changelist value.
3) My 1 build executor is assigned a job and starts building (sub-job 1 of 4).
4) The job from 3) completes so my build executor is assigned another job and starts building (sub-job 2 of 4).
5) P4 counter is still correct.
6) FileB is checked in which will trigger JobA again.
7) I now see JobA has been added to the "Build Queue" but it can't start building because the build executor is still busy with sub-job 2 still.
8) At this point the counter is incorrect. The value is the same as the CL containing FileB.

It appears that the P4 counter is updated when JobA is added to the build queue, not when it is dispatched to actually build. This seems like a bug to me.

Regards,
Tom.

No perforce operations are executed when a build is queued.

From your description, the new file is checked in when between sub jobs. Wouldn't that explain why the counter is getting updated? Once sub job 3 starts, it will check out the latest and update the counter. A sub job is just a regular job from an SCM point of view, so unless you tell it otherwise, it will always check out the latest.

Hi Rob,

So sub job from a matrix project will also update the counter? That seems odd to me. I'd expect the parent of the matrix job to set the counter then the sub jobs to actually use it, not update it again.

How do I tell the sub jobs not to update the counter?

Tom.

Yes, every sub job bahaves like a regular job. The perforce plugin doesn't explicitly support matrix builds yet, so not all the functionality will work as you might expect.

You can tell the sub jobs to not update the counter by disabling counter updates entirely. If you choose this route, you'll need to find some other way to pass the changeset to the downstream builds.

Hi Rob,

I've been using the Perforce plugin with matrix jobs for a while now. Is this something that can be fixed in the Perforce plugin anytime soon?

Thanks,
Tom.

Define soon? I'm currently indisposed for a week or two at least, so I won't have much time to work on it. If you need it sooner than that, I can try to grind out a solution, but you'll have to test it since I don't currently have access to the facilities to do so myself.

As soon as possible really I don't mind testing it at all. Thanks for this, I appreciate it.

Tom.

Code changed in jenkins
User: Rob Petti
Path:
src/main/java/hudson/plugins/perforce/PerforceSCM.java
http://jenkins-ci.org/commit/perforce-plugin/acf2eba79edfccec2a67b05bb658a82478dd311e
Log:
JENKINS-10592 if we're a matrixrun, use the same changeset from the parent when syncing

Integrated in plugins_perforce #125
JENKINS-10592 if we're a matrixrun, use the same changeset from the parent when syncing

Rob Petti :
Files :

• src/main/java/hudson/plugins/perforce/PerforceSCM.java

I've added code that should make all matrix sub jobs use the same changeset when syncing. That should fix the issue you are having.

Can you test the latest snapshot, please? You can find it here: http://files.robpetti.com/perforce-plugin/target/perforce.hpi

Hi Rob,

I don't believe the fix is correct. Looking at PerforceSCM.java I'm seeing the possibility of:

log.println("Updating counter " + p4Counter + " to " + newestChange);

being called for each sub job. The parent job has the "Update counter to most recent changeset" option ticked but I think that needs disabling for sub jobs. Something like this might do it:

if (p4Counter != null && updateCounterValue && notASubJob)

What do you think? Have I got that completely wrong?

Regards,
Tom.

Attach possible fix.

I don't see the reason for disabling the counter for sub jobs so long as each sub job is syncing to the same changeset. What issues did you run into when you tested my fix?

When testing your fix the P4 counter was being updated in the sub job when it shouldn't have been.

JobA setup:
Polling time is * * * * *.
Build action is to call a Ant script containing: <sleep seconds="90"/>

1) I check a file in to trigger a build of JobA (parent):

Started by an SCM change
Using master perforce client: Jenkins_test
[workspace] $ "C:\\Program Files\\Perforce\\p4.exe" workspace -o Jenkins_test
Last build changeset: 141384
[workspace] $ "C:\\Program Files\\Perforce\\p4.exe" counter change
[workspace] $ "C:\\Program Files\\Perforce\\p4.exe" -s changes //personal/tfields/Live/Engine/Build/...@141385,@141395
Sync'ing workspace to changelist 141395 (forcing sync of unchanged files).
[workspace] $ "C:\\Program Files\\Perforce\\p4.exe" -s sync -f //personal/tfields/Live/Engine/Build/...@141395
Sync complete, took 654 ms
Updating counter TOMTOMTEST to 141395
[workspace] $ "C:\\Program Files\\Perforce\\p4.exe" counter TOMTOMTEST 141395
Triggering CL,Dt,R
Triggering SNC,Dt,Q
Finished: SUCCESS

2) CL,Dt,R sub job builds:

Started by upstream project "test" build number 32
Using master perforce client: Jenkins_test-C-CL-N-Dt-P-R
[R] $ "C:\\Program Files\\Perforce\\p4.exe" workspace -o Jenkins_test-C-CL-N-Dt-P-R
Last build changeset: 141384
[R] $ "C:\\Program Files\\Perforce\\p4.exe" counter change
Sync'ing workspace to changelist 141395 (forcing sync of unchanged files).
[R] $ "C:\\Program Files\\Perforce\\p4.exe" -s sync -f //personal/tfields/Live/Engine/Build/...@141395
Sync complete, took 710 ms
Updating counter TOMTOMTEST to 141395
[R] $ "C:\\Program Files\\Perforce\\p4.exe" counter TOMTOMTEST 141395
[R] $ cmd.exe /C '"ant.bat -file buildTest.xml -DP=R -DC=CL -DN=Dt delay3 && exit %%ERRORLEVEL%%"'

delay3:

BUILD SUCCESSFUL
Total time: 1 minute 30 seconds
Finished: SUCCESS

3) As soon as the CL,Dt,R sub job builds I check in another file. JobA will be added to the job queue but not start building until after the original JobA build has completed because I've only got 1 build executor thread.

4) SNC,Dt,Q sub job builds:

Started by upstream project "test" build number 32
Using master perforce client: Jenkins_test-C-SNC-N-Dt-P-Q
[Q] $ "C:\\Program Files\\Perforce\\p4.exe" workspace -o Jenkins_test-C-SNC-N-Dt-P-Q
Last build changeset: 141384
[Q] $ "C:\\Program Files\\Perforce\\p4.exe" counter change
[Q] $ "C:\\Program Files\\Perforce\\p4.exe" -s changes //personal/tfields/Live/Engine/Build/...@141396,@141398
Sync'ing workspace to changelist 141398 (forcing sync of unchanged files).
[Q] $ "C:\\Program Files\\Perforce\\p4.exe" -s sync -f //personal/tfields/Live/Engine/Build/...@141398
Sync complete, took 638 ms
Updating counter TOMTOMTEST to 141398
[Q] $ "C:\\Program Files\\Perforce\\p4.exe" counter TOMTOMTEST 141398
[Q] $ cmd.exe /C '"ant.bat -file buildTest.xml -DP=Q -DC=SNC -DN=Dt delay3 && exit %%ERRORLEVEL%%"'

delay3:

BUILD SUCCESSFUL
Total time: 1 minute 30 seconds
Finished: SUCCESS

In step 4 why is the counter being updated? It should still be 141395 shouldn't it?

Regards,
Tom.

The fix was never meant to prevent sub jobs from updating the counter. In most cases, the only thing that will be getting executed will be the sub jobs (unless you are running a touchstone build first). We will always want the counter to be updated.

The fix was supposed to sync every sub job to the same changeset.

I will take another look at it when I have the time.

Code changed in jenkins
User: Rob Petti
Path:
src/main/java/hudson/plugins/perforce/PerforceSCM.java
http://jenkins-ci.org/commit/perforce-plugin/03595c0e90be9daa53aa27a725de0b368992804c
Log:
JENKINS-10592 adding more logging for matrix builds, and fixing incorrect behavior

Integrated in plugins_perforce #126
JENKINS-10592 adding more logging for matrix builds, and fixing incorrect behavior

Rob Petti :
Files :

• src/main/java/hudson/plugins/perforce/PerforceSCM.java

Snapshot has been updated with a potential fix and some additional logging. Can you give it another try, please?

Hi Rob,

Thanks for the new update. That looks a lot better so I now get:

1) I check a file in to trigger a build of JobA (parent):

Started by an SCM change
Using master perforce client: Jenkins_test
[workspace] $ "C:\\Program Files\\Perforce\\p4.exe" workspace -o Jenkins_test
Last build changeset: 141604
[workspace] $ "C:\\Program Files\\Perforce\\p4.exe" counter change
[workspace] $ "C:\\Program Files\\Perforce\\p4.exe" -s changes //personal/tfields/Live/Engine/Build/...@141605,@141611
Sync'ing workspace to changelist 141611 (forcing sync of unchanged files).
[workspace] $ "C:\\Program Files\\Perforce\\p4.exe" -s sync -f //personal/tfields/Live/Engine/Build/...@141611
Sync complete, took 1087 ms
Updating counter TOMTOMTEST to 141611
[workspace] $ "C:\\Program Files\\Perforce\\p4.exe" counter TOMTOMTEST 141611
Triggering SNC,Dt,Q
Triggering CL,Dt,R
Finished: SUCCESS

2) CL,Dt,R sub job builds:

Started by upstream project "test" build number 36
Using master perforce client: Jenkins_test-C-CL-N-Dt-P-R
[R] $ "C:\\Program Files\\Perforce\\p4.exe" workspace -o Jenkins_test-C-CL-N-Dt-P-R
Last build changeset: 141604
[R] $ "C:\\Program Files\\Perforce\\p4.exe" counter change
This is a matrix run, trying to use change number from parent/siblings...
Latest change from parent is: 0
[R] $ "C:\\Program Files\\Perforce\\p4.exe" -s changes //personal/tfields/Live/Engine/Build/...@141605,@141611
Sync'ing workspace to changelist 141611 (forcing sync of unchanged files).
[R] $ "C:\\Program Files\\Perforce\\p4.exe" -s sync -f //personal/tfields/Live/Engine/Build/...@141611
Sync complete, took 815 ms
Updating counter TOMTOMTEST to 141611
[R] $ "C:\\Program Files\\Perforce\\p4.exe" counter TOMTOMTEST 141611
[R] $ cmd.exe /C '"ant.bat -file buildTest.xml -DP=R -DC=CL -DN=Dt delay3 && exit %%ERRORLEVEL%%"'

delay3:

BUILD SUCCESSFUL
Total time: 1 minute 30 seconds
Finished: SUCCESS

3) As soon as the CL,Dt,R sub job builds I check in another file. JobA will be added to the job queue but not start building until after the original JobA build has completed because I've only got 1 build executor thread.

4) SNC,Dt,Q sub job builds:

Started by upstream project "test" build number 36
Using master perforce client: Jenkins_test-C-SNC-N-Dt-P-Q
[Q] $ "C:\\Program Files\\Perforce\\p4.exe" workspace -o Jenkins_test-C-SNC-N-Dt-P-Q
Last build changeset: 141604
[Q] $ "C:\\Program Files\\Perforce\\p4.exe" counter change
This is a matrix run, trying to use change number from parent/siblings...
Latest change from parent is: 0
[Q] $ "C:\\Program Files\\Perforce\\p4.exe" -s changes //personal/tfields/Live/Engine/Build/...@141605,@141611
Sync'ing workspace to changelist 141611 (forcing sync of unchanged files).
[Q] $ "C:\\Program Files\\Perforce\\p4.exe" -s sync -f //personal/tfields/Live/Engine/Build/...@141611
Sync complete, took 1064 ms
Updating counter TOMTOMTEST to 141611
[Q] $ "C:\\Program Files\\Perforce\\p4.exe" counter TOMTOMTEST 141611
[Q] $ cmd.exe /C '"ant.bat -file buildTest.xml -DP=Q -DC=SNC -DN=Dt delay3 && exit %%ERRORLEVEL%%"'

delay3:

BUILD SUCCESSFUL
Total time: 1 minute 30 seconds
Finished: SUCCESS

Thanks a lot.

Tom.

Code changed in jenkins
User: Rob Petti
Path:
src/main/java/hudson/plugins/perforce/PerforceSCM.java
http://jenkins-ci.org/commit/perforce-plugin/4850a5aff2fd99eed237b5a5c35dc8b0218210f0
Log:
JENKINS-10592 fixing cosmetic log problem

the log would always output '0' for the changeset number from the parent job when running a matrix build. it'll should now output the value that's actually used

Integrated in plugins_perforce #127
JENKINS-10592 fixing cosmetic log problem

Rob Petti :
Files :

• src/main/java/hudson/plugins/perforce/PerforceSCM.java