Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-10592

Issues with P4 counters not working correctly in matrix builds

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • p4-plugin
    • None
    • Jenkins 1.424
      Perforce plugin 1.3.0

      Hi,

      I'm having problems getting the Perforce plugin counter system to work properly. This is my setup:

      JobA (matrix project): 
	P4 Counter = TOMLABEL
	Update counter to most recent changeset = ticked
	Build other projects = JobB

      	JobB (free-style project):
	P4 Counter = TOMLABEL
	Update counter to most recent changeset = not ticked

      So when I make a change in Perforce JobA is triggered. If I check the Console Output for the parent job it contains:

      Updating counter TOMLABEL to 140403
[workspace] $ "C:\\Program Files\\Perforce\\p4.exe" counter TOMLABEL 140403

      Each element of JobA's matrix then builds. In the Console output for those builds I see a lot of this:

      [Win32] $ "C:\\Program Files\\Perforce\\p4.exe" -s sync -f //personal/tfields/Live/Engine/Tools/program.exe@140403

      which is what I expect to see, ie. it's syncing to my counter.

      In between JobA finishing and JobB starting I check in another file which means the latest changelist Perforce knows about is now 140404.

      So JobA finally finishes building, succeeds and triggers JobB.

      All JobB does is print out the current value of TOMLABEL. So if I check the Console Output I would expect to see the value 140403. Unfortunately I don't. I get the value 140404 back.

      [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" workspace -o Jenkins_External
18:24:53  Last build changeset: 140392
18:24:53  [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" counter TOMLABEL
18:24:53  [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" -s changes //Jenkins_External/...@140393,@140404
18:24:53  [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" describe -s 140404
18:24:53  [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" -G where //personal/tfields/Live/Engine/External/NvTriStrip/NvTriStrip.cpp
18:24:53  [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" describe -s 140403
18:24:53  [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" -G where //personal/tfields/Live/Engine/Core/Engine.cpp
18:24:54  Buildfile: C:\JCI\jobs\External\workspace\build-jci.xml
18:24:54  
18:24:54  PrintInfo:
18:24:54       [exec] TOMLABEL = 140404

      Any ideas why this is happening. It seems like a bug to me.

      Regards,
      Tom.

      – Posted from Bugbox for Android

        1. PerforceSCM.java
          83 kB
        2. PerforceSCM.java
          83 kB

          [JENKINS-10592] Issues with P4 counters not working correctly in matrix builds

          Thomas Fields created issue -

          Rob Petti added a comment -

          That sounds more like a race condition to me. The second change would trigger job a again and update the counter before job b can grab it. If you increased the quiet period on job A, or set up a lock so A and B can't run at the same time, then I think that should fix the problem.

          Alternatively, you can pass the changeset down as a job parameter instead of a counter.

          – Posted from Bugbox for Android

          Rob Petti added a comment - That sounds more like a race condition to me. The second change would trigger job a again and update the counter before job b can grab it. If you increased the quiet period on job A, or set up a lock so A and B can't run at the same time, then I think that should fix the problem. Alternatively, you can pass the changeset down as a job parameter instead of a counter. – Posted from Bugbox for Android
          Rob Petti made changes -
          Assignee New: Rob Petti [ rpetti ]
          Description Original: Hi,

          I'm having problems getting the Perforce plugin counter system to work properly. This is my setup:

          {code}
          JobA (matrix project):
          P4 Counter = TOMLABEL
          Update counter to most recent changeset = ticked
          Build other projects = JobB
          {code}

          {code}
          JobB (free-style project):
          P4 Counter = TOMLABEL
          Update counter to most recent changeset = not ticked
          {code}

          So when I make a change in Perforce JobA is triggered. If I check the Console Output for the parent job it contains:

          {code}
          Updating counter TOMLABEL to 140403
          [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" counter TOMLABEL 140403
          {code}

          Each element of JobA's matrix then builds. In the Console output for those builds I see a lot of this:

          {code}
          [Win32] $ "C:\\Program Files\\Perforce\\p4.exe" -s sync -f //personal/tfields/Live/Engine/Tools/program.exe@140403
          {code}

          which is what I expect to see, ie. it's syncing to my counter.

          In between JobA finishing and JobB starting I check in another file which means the latest changelist Perforce knows about is now 140404.

          So JobA finally finishes building, succeeds and triggers JobB.

          All JobB does is print out the current value of TOMLABEL. So if I check the Console Output I would expect to see the value 140403. Unfortunately I don't. I get the value 140404 back.

          {code}
          [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" workspace -o Jenkins_External
          18:24:53 Last build changeset: 140392
          18:24:53 [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" counter TOMLABEL
          18:24:53 [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" -s changes //Jenkins_External/...@140393,@140404
          18:24:53 [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" describe -s 140404
          18:24:53 [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" -G where //personal/tfields/Live/Engine/External/NvTriStrip/NvTriStrip.cpp
          18:24:53 [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" describe -s 140403
          18:24:53 [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" -G where //personal/tfields/Live/Engine/Core/Engine.cpp
          18:24:54 Buildfile: C:\JCI\jobs\External\workspace\build-jci.xml
          18:24:54
          18:24:54 PrintInfo:
          18:24:54 [exec] TOMLABEL = 140404
          {code}

          Any ideas why this is happening. It seems like a bug to me.

          Regards,
          Tom.           		New: Hi,

          I'm having problems getting the Perforce plugin counter system to work properly. This is my setup:

          {code}
          JobA (matrix project):
          P4 Counter = TOMLABEL
          Update counter to most recent changeset = ticked
          Build other projects = JobB
          {code}

          {code}
          JobB (free-style project):
          P4 Counter = TOMLABEL
          Update counter to most recent changeset = not ticked
          {code}

          So when I make a change in Perforce JobA is triggered. If I check the Console Output for the parent job it contains:

          {code}
          Updating counter TOMLABEL to 140403
          [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" counter TOMLABEL 140403
          {code}

          Each element of JobA's matrix then builds. In the Console output for those builds I see a lot of this:

          {code}
          [Win32] $ "C:\\Program Files\\Perforce\\p4.exe" -s sync -f //personal/tfields/Live/Engine/Tools/program.exe@140403
          {code}

          which is what I expect to see, ie. it's syncing to my counter.

          In between JobA finishing and JobB starting I check in another file which means the latest changelist Perforce knows about is now 140404.

          So JobA finally finishes building, succeeds and triggers JobB.

          All JobB does is print out the current value of TOMLABEL. So if I check the Console Output I would expect to see the value 140403. Unfortunately I don't. I get the value 140404 back.

          {code}
          [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" workspace -o Jenkins_External
          18:24:53 Last build changeset: 140392
          18:24:53 [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" counter TOMLABEL
          18:24:53 [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" -s changes //Jenkins_External/...@140393,@140404
          18:24:53 [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" describe -s 140404
          18:24:53 [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" -G where //personal/tfields/Live/Engine/External/NvTriStrip/NvTriStrip.cpp
          18:24:53 [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" describe -s 140403
          18:24:53 [workspace] $ "C:\\Program Files\\Perforce\\p4.exe" -G where //personal/tfields/Live/Engine/Core/Engine.cpp
          18:24:54 Buildfile: C:\JCI\jobs\External\workspace\build-jci.xml
          18:24:54
          18:24:54 PrintInfo:
          18:24:54 [exec] TOMLABEL = 140404
          {code}

          Any ideas why this is happening. It seems like a bug to me.

          Regards,
          Tom.

          -- Posted from Bugbox for Android

          Thomas Fields added a comment -

          Hi Rob,

          I initially thought it was a race condition as well but with further tests I think I've got to the bottom of what is happening.

          For testing purposes my Jenkins setup has only 1 build executor thread.

          Consider this scenario:

          1) FileA is checked in.
          2) JobA is triggered, this queues up 4 child jobs because it's a matrix project. P4 counter is correct assigned latest changelist value.
          3) My 1 build executor is assigned a job and starts building (sub-job 1 of 4).
          4) The job from 3) completes so my build executor is assigned another job and starts building (sub-job 2 of 4).
          5) P4 counter is still correct.
          6) FileB is checked in which will trigger JobA again.
          7) I now see JobA has been added to the "Build Queue" but it can't start building because the build executor is still busy with sub-job 2 still.
          8) At this point the counter is incorrect. The value is the same as the CL containing FileB.

          It appears that the P4 counter is updated when JobA is added to the build queue, not when it is dispatched to actually build. This seems like a bug to me.

          Regards,
          Tom.

          Thomas Fields added a comment - Hi Rob, I initially thought it was a race condition as well but with further tests I think I've got to the bottom of what is happening. For testing purposes my Jenkins setup has only 1 build executor thread. Consider this scenario: 1) FileA is checked in. 2) JobA is triggered, this queues up 4 child jobs because it's a matrix project. P4 counter is correct assigned latest changelist value. 3) My 1 build executor is assigned a job and starts building (sub-job 1 of 4). 4) The job from 3) completes so my build executor is assigned another job and starts building (sub-job 2 of 4). 5) P4 counter is still correct. 6) FileB is checked in which will trigger JobA again. 7) I now see JobA has been added to the "Build Queue" but it can't start building because the build executor is still busy with sub-job 2 still. 8) At this point the counter is incorrect. The value is the same as the CL containing FileB. It appears that the P4 counter is updated when JobA is added to the build queue, not when it is dispatched to actually build. This seems like a bug to me. Regards, Tom.

          Rob Petti added a comment -

          No perforce operations are executed when a build is queued.

          From your description, the new file is checked in when between sub jobs. Wouldn't that explain why the counter is getting updated? Once sub job 3 starts, it will check out the latest and update the counter. A sub job is just a regular job from an SCM point of view, so unless you tell it otherwise, it will always check out the latest.

          Rob Petti added a comment - No perforce operations are executed when a build is queued. From your description, the new file is checked in when between sub jobs. Wouldn't that explain why the counter is getting updated? Once sub job 3 starts, it will check out the latest and update the counter. A sub job is just a regular job from an SCM point of view, so unless you tell it otherwise, it will always check out the latest.

          Thomas Fields added a comment -

          Hi Rob,

          So sub job from a matrix project will also update the counter? That seems odd to me. I'd expect the parent of the matrix job to set the counter then the sub jobs to actually use it, not update it again.

          How do I tell the sub jobs not to update the counter?

          Tom.

          Thomas Fields added a comment - Hi Rob, So sub job from a matrix project will also update the counter? That seems odd to me. I'd expect the parent of the matrix job to set the counter then the sub jobs to actually use it, not update it again. How do I tell the sub jobs not to update the counter? Tom.

          Rob Petti added a comment -

          Yes, every sub job bahaves like a regular job. The perforce plugin doesn't explicitly support matrix builds yet, so not all the functionality will work as you might expect.

          You can tell the sub jobs to not update the counter by disabling counter updates entirely. If you choose this route, you'll need to find some other way to pass the changeset to the downstream builds.

          Rob Petti added a comment - Yes, every sub job bahaves like a regular job. The perforce plugin doesn't explicitly support matrix builds yet, so not all the functionality will work as you might expect. You can tell the sub jobs to not update the counter by disabling counter updates entirely. If you choose this route, you'll need to find some other way to pass the changeset to the downstream builds.
          Rob Petti made changes -
          Link New: This issue depends on JENKINS-10606 [ JENKINS-10606 ]

          Thomas Fields added a comment -

          Hi Rob,

          I've been using the Perforce plugin with matrix jobs for a while now. Is this something that can be fixed in the Perforce plugin anytime soon?

          Thanks,
          Tom.

          Thomas Fields added a comment - Hi Rob, I've been using the Perforce plugin with matrix jobs for a while now. Is this something that can be fixed in the Perforce plugin anytime soon? Thanks, Tom.
          Rob Petti made changes -
          Summary Original: Issues with P4 counters not working correctly New: Issues with P4 counters not working correctly in matrix builds

            rpetti Rob Petti
            tfields Thomas Fields
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: