Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63879

Using pin to sync to changelist causes polling to no longer detect changes beyond pinned CL

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Karls Summary:

      Polled builds (pipeline) should sync to "@now" when polling occurred not "@now" when sync occurs.

      Usage case - On a busy system a long running job will pickup components of different ages.

      Example -

      • Poll finds latest change 123
      • Build is queued
      • Change 124 added to path1.
      • Build starts
      • Path1 synced at 124
      • Change 125 and 126 submitted to path2.
      • Path1 sync completes, path2 sync starts at 126

      The desired behavior is path1 and path2 should be synced at 123 (if that is not possible in the Jenkins infrastructure path 1 and path2 should be synced at 124).

       

      Original Description:

      HI we have setup a parallel pipeline and are using SyncOnlyImpl's pin to ensure all parallel stages will build on the same changelist.

       

      Our first implementation worked as expected.  pin value was empty at first, causing it to sync to head then we extracted this value and propagated to other stages through a field.

      However other requirements required us to have that change be accessible in other parts of the build pipeline so we start the pipeline with a simple p4 command to get the head CL which is then propagated through the rest of the pipeline normally rather than through a field.

      Sync behavior is as expected where all stages sync to desired CL BUT polling is now limited by this pin value and will always check for changes using the pinned value as an upper limit.

      Net effect if this is polling still takes place but never detect any changes effectively nullifying poll behavior beyond the first run.

       

      I would expect a way to sync to a specific CL that has no side effect in other parts of the system.  Either the pin behavior is broken, or an extra parameter is required beyond pin to separate the two intent : the CL at which we wish to "pin" this job, and the CL at which we wish to sync in the populate implementation.

        Attachments

          Activity

          newtopian Eric Daigneault created issue -
          Hide
          p4karl Karl Wirth added a comment -

          Hi Eric Daigneault - Please get me your pipeline code so I can test here. I'm wondering if disabling implicit sync may be a workaround.

          Show
          p4karl Karl Wirth added a comment - Hi Eric Daigneault - Please get me your pipeline code so I can test here. I'm wondering if disabling implicit sync may be a workaround.
          Hide
          p4karl Karl Wirth added a comment -

          Hi Eric Daigneault - Ignore that. Disabling implicit doe not change the behavior.

          P4Jenkins will only build if there is something to sync. This is intended because 'pin' is normally used with a build label and you only need to rebuild if the build label has changed. You dont want a job building if the label has not been updated.

          If I have two 'p4sync' steps in my pipeline script and only one of them is pinned the job will still build. From the polling log:

           

          # Pinned at 4262 
          P4: Polling with range: 4262,4262
          ... p4 changes -m20 //jenkins-master-PinPolledPipeline-Source-TEST/...@4262,4262 
          ... p4 repos -C 
          P4: Polling no changes found.
          
          # Not pinned
          P4: Polling with range: 4274,now
          ... p4 changes -m20 //jenkins-master-PinPolledPipeline-Source-TEST-FILES/...@4274,now 
          ... p4 change -o 4276 
          ... found change: 4276
          

          If after reading this you think its still not working as described please provide me more information about how you strcture everything. Are you syncing any new code in the job? Do you have multiple jobs or just steps? How are they connected to each other? Can you provide me the code?

           

          If it is working as described the workaround would be to trigger the jobs from perforce instead of using polling.

           

           

          Show
          p4karl Karl Wirth added a comment - Hi Eric Daigneault - Ignore that. Disabling implicit doe not change the behavior. P4Jenkins will only build if there is something to sync. This is intended because 'pin' is normally used with a build label and you only need to rebuild if the build label has changed. You dont want a job building if the label has not been updated. If I have two 'p4sync' steps in my pipeline script and only one of them is pinned the job will still build. From the polling log:   # Pinned at 4262 P4: Polling with range: 4262,4262 ... p4 changes -m20 //jenkins-master-PinPolledPipeline-Source-TEST/...@4262,4262 ... p4 repos -C P4: Polling no changes found. # Not pinned P4: Polling with range: 4274,now ... p4 changes -m20 //jenkins-master-PinPolledPipeline-Source-TEST-FILES/...@4274,now ... p4 change -o 4276 ... found change: 4276 If after reading this you think its still not working as described please provide me more information about how you strcture everything. Are you syncing any new code in the job? Do you have multiple jobs or just steps? How are they connected to each other? Can you provide me the code?   If it is working as described the workaround would be to trigger the jobs from perforce instead of using polling.    
          p4karl Karl Wirth made changes -
          Field Original Value New Value
          Assignee Karl Wirth [ p4karl ]
          p4karl Karl Wirth made changes -
          Labels P4_SUPPORT
          Hide
          newtopian Eric Daigneault added a comment - - edited

          Our pipeline resembles this pseudo seen below. The code is over-simplified and will likely not run as-is but it does give the essential of what is happening.

          Basically here we need to ensure that all stages in the parallel to build at the same CL. By default each will sync to head at the time they start, since each gets assigned to a different agent it's quite possible one or more is put in the queue to wait for available resources. During that time it's very likely that new commits occur thus having them all sync to head creates a "staggered" build across multiple changes which is undesirable.

          We used to do it in the way you suggests where one is not pinned, we struggled a bit to get the information across other stage but eventually made it work. however getting that changelist ID to propagate to other steps prooved to be challenging and it became much simpler to just fetch the change eagerly and pass it along the entire pipeline.

          Now we have averything working just dandy and all other steps that need access to the changelist can do so naturally... except polling no longer work because of this side effect. And since there are no obvious alternative ways to sync through the plugin that would not generate this side effect we are basically stuck. Temporary solution was to change the polling with a cron and force trigger the job each 30 mins or so.

          We experimented with perforce triggers and ran in another issue. Since the changelist is not (least wasnot at the time we tried) attached to the job being triggered we ran in situations where each commit woudl trigger a job but build system restrictions caused long queues. To make things worst when a job started it would pick head CL thus many jobs ended running at the same CL because of this which was very wasteful use of resources.

          Last thing we had not tried yet was to simply do without the populate and run the p4 sync directly with a p4.run.

          def pinnedchangelist = p4.run("changes", "-s", "submitted", "-m1", "${stream_location}/...")[0]['change']
          pipeline{
            options{
              skipDefaultCheckout()
            }
            parallel{
              stages{
                stage('someplatform'){
          	    agent('able-tagged'){}
                  step('sync'){
                    //see below for actual code
                  }
          		step('build'){
          		  ///
          		}
                }
                stage('someotherplatform'){
          	    agent('able-tagged'){}
                  step('sync'){
                    //see below for actual code
                  }
          		step('build'){
          		  ///
          		}
                }
                stage('yetanother'){
          	    agent('able-tagged'){}
                  step('sync'){
                    //see below for actual code
                  }
          		step('build'){
          		  ///
          		}
                }
              }
            }
          }
          

          the steps sync contains this code which does the actual sync

              def popImpl = [$class: "SyncOnlyImpl",
                      force: needs_force_sync(config),
                      have: true,
                      modtime: true,
                      parallel: null,
                      pin: pinnedChangelist,
                      quiet: false,
                      revert: false]
          
              echo "==  Workspace preparation complete, performing a sync ${popImpl}"
              checkout([
                      $class: 'PerforceScm',
                      credential: "${config.p4_credential}".toString(),
                      populate: popImpl,
                      workspace: ws
                      ])
          
          

          Edit:
          added skipDefaultCheckout() in option block at start of pipeline
          reformat pipeline code

          Show
          newtopian Eric Daigneault added a comment - - edited Our pipeline resembles this pseudo seen below. The code is over-simplified and will likely not run as-is but it does give the essential of what is happening. Basically here we need to ensure that all stages in the parallel to build at the same CL. By default each will sync to head at the time they start, since each gets assigned to a different agent it's quite possible one or more is put in the queue to wait for available resources. During that time it's very likely that new commits occur thus having them all sync to head creates a "staggered" build across multiple changes which is undesirable. We used to do it in the way you suggests where one is not pinned, we struggled a bit to get the information across other stage but eventually made it work. however getting that changelist ID to propagate to other steps prooved to be challenging and it became much simpler to just fetch the change eagerly and pass it along the entire pipeline. Now we have averything working just dandy and all other steps that need access to the changelist can do so naturally... except polling no longer work because of this side effect. And since there are no obvious alternative ways to sync through the plugin that would not generate this side effect we are basically stuck. Temporary solution was to change the polling with a cron and force trigger the job each 30 mins or so. We experimented with perforce triggers and ran in another issue. Since the changelist is not (least wasnot at the time we tried) attached to the job being triggered we ran in situations where each commit woudl trigger a job but build system restrictions caused long queues. To make things worst when a job started it would pick head CL thus many jobs ended running at the same CL because of this which was very wasteful use of resources. Last thing we had not tried yet was to simply do without the populate and run the p4 sync directly with a p4.run. def pinnedchangelist = p4.run( "changes" , "-s" , "submitted" , "-m1" , "${stream_location}/..." )[0][ 'change' ] pipeline{ options{ skipDefaultCheckout() } parallel{ stages{ stage( 'someplatform' ){ agent( 'able-tagged' ){} step( 'sync' ){ //see below for actual code } step( 'build' ){ /// } } stage( 'someotherplatform' ){ agent( 'able-tagged' ){} step( 'sync' ){ //see below for actual code } step( 'build' ){ /// } } stage( 'yetanother' ){ agent( 'able-tagged' ){} step( 'sync' ){ //see below for actual code } step( 'build' ){ /// } } } } } the steps sync contains this code which does the actual sync def popImpl = [$class: "SyncOnlyImpl" , force: needs_force_sync(config), have: true , modtime: true , parallel: null , pin: pinnedChangelist, quiet: false , revert: false ] echo "== Workspace preparation complete, performing a sync ${popImpl}" checkout([ $class: 'PerforceScm' , credential: "${config.p4_credential}" .toString(), populate: popImpl, workspace: ws ]) Edit: added skipDefaultCheckout() in option block at start of pipeline reformat pipeline code
          Hide
          p4karl Karl Wirth added a comment -

          Hi Eric Daigneault - A quick aplogy. I owe you a response on this one but I'm still working on it. Will get back to you when I have something.

          Show
          p4karl Karl Wirth added a comment - Hi Eric Daigneault - A quick aplogy. I owe you a response on this one but I'm still working on it. Will get back to you when I have something.
          Hide
          pgrohe Phil Grohe added a comment - - edited

          Hi Karl Wirth & Eric Daigneault

          Karl, for general context on how we are currently doing parallel builds & enforcing all parallel machines are synced to the same CL for a given build see this blog post from Riot Games, under the "Multiple Syncs in A Pipeline" section of the article:

          https://technology.riotgames.com/news/using-perforce-complex-jenkins-pipeline

          That's the core idea of what we're currently doing succesfully (with some tradeoffs and limitations that Eric is working on tackling with what he describes above).  Run a job with multiple seperate syncs occuring on parallel job parts & ensure they all sync to the same CL regardless of whether new CLs come into Perforce before some of the parallel parts begin (ex: waiting on an available agent).

          Apologies for the tangent but it gives an idea of what we're aiming for in a broader sense.

          Cheers.

          Show
          pgrohe Phil Grohe added a comment - - edited Hi Karl Wirth & Eric Daigneault Karl, for general context on how we are currently doing parallel builds & enforcing all parallel machines are synced to the same CL for a given build see this blog post from Riot Games, under the "Multiple Syncs in A Pipeline" section of the article: https://technology.riotgames.com/news/using-perforce-complex-jenkins-pipeline That's the core idea of what we're currently doing succesfully (with some tradeoffs and limitations that Eric is working on tackling with what he describes above).  Run a job with multiple seperate syncs occuring on parallel job parts & ensure they all sync to the same CL regardless of whether new CLs come into Perforce before some of the parallel parts begin (ex: waiting on an available agent). Apologies for the tangent but it gives an idea of what we're aiming for in a broader sense. Cheers.
          Hide
          arka_bb Arto Karppinen added a comment -

          I resolved the same problem by creating a separate polling pipeline, which just does the polling, and triggers the main pipeline with:

          build(job: 'main_pipeline_job', wait: false)

          Show
          arka_bb Arto Karppinen added a comment - I resolved the same problem by creating a separate polling pipeline, which just does the polling, and triggers the main pipeline with: build(job: 'main_pipeline_job', wait: false)
          p4karl Karl Wirth made changes -
          Description HI we have setup a parallel pipeline and are using SyncOnlyImpl's pin to ensure all parallel stages will build on the same changelist.

           

          Our first implementation worked as expected.  pin value was empty at first, causing it to sync to head then we extracted this value and propagated to other stages through a field.

          However other requirements required us to have that change be accessible in other parts of the build pipeline so we start the pipeline with a simple p4 command to get the head CL which is then propagated through the rest of the pipeline normally rather than through a field.

          Sync behavior is as expected where all stages sync to desired CL BUT polling is now limited by this pin value and will always check for changes using the pinned value as an upper limit.

          Net effect if this is polling still takes place but never detect any changes effectively nullifying poll behavior beyond the first run.

           

          I would expect a way to sync to a specific CL that has no side effect in other parts of the system.  Either the pin behavior is broken, or an extra parameter is required beyond pin to separate the two intent : the CL at which we wish to "pin" this job, and the CL at which we wish to sync in the populate implementation.
          *Karls Summary:*

          Polled builds (pipeline) should sync to "@now" when polling occurred not "@now" when sync occurs.

          _Usage case_ - On a busy system a long running job will pickup components of different ages.

          _Example_ -
           * Poll finds latest change 123
           * Build is queued
           * Change 124 added to path1.
           * Build starts
           * Path1 synced at 124
           * Change 125 and 126 submitted to path2.
           * Path1 sync completes, path2 sync starts at 126

          The desired behavior is Path2 should be synced at 124.

           

          *Original Description:*

          HI we have setup a parallel pipeline and are using SyncOnlyImpl's pin to ensure all parallel stages will build on the same changelist.

           

          Our first implementation worked as expected.  pin value was empty at first, causing it to sync to head then we extracted this value and propagated to other stages through a field.

          However other requirements required us to have that change be accessible in other parts of the build pipeline so we start the pipeline with a simple p4 command to get the head CL which is then propagated through the rest of the pipeline normally rather than through a field.

          Sync behavior is as expected where all stages sync to desired CL BUT polling is now limited by this pin value and will always check for changes using the pinned value as an upper limit.

          Net effect if this is polling still takes place but never detect any changes effectively nullifying poll behavior beyond the first run.

           

          I would expect a way to sync to a specific CL that has no side effect in other parts of the system.  Either the pin behavior is broken, or an extra parameter is required beyond pin to separate the two intent : the CL at which we wish to "pin" this job, and the CL at which we wish to sync in the populate implementation.
          p4karl Karl Wirth made changes -
          Labels P4_SUPPORT P4_A P4_VERIFY
          p4karl Karl Wirth made changes -
          Description *Karls Summary:*

          Polled builds (pipeline) should sync to "@now" when polling occurred not "@now" when sync occurs.

          _Usage case_ - On a busy system a long running job will pickup components of different ages.

          _Example_ -
           * Poll finds latest change 123
           * Build is queued
           * Change 124 added to path1.
           * Build starts
           * Path1 synced at 124
           * Change 125 and 126 submitted to path2.
           * Path1 sync completes, path2 sync starts at 126

          The desired behavior is Path2 should be synced at 124.

           

          *Original Description:*

          HI we have setup a parallel pipeline and are using SyncOnlyImpl's pin to ensure all parallel stages will build on the same changelist.

           

          Our first implementation worked as expected.  pin value was empty at first, causing it to sync to head then we extracted this value and propagated to other stages through a field.

          However other requirements required us to have that change be accessible in other parts of the build pipeline so we start the pipeline with a simple p4 command to get the head CL which is then propagated through the rest of the pipeline normally rather than through a field.

          Sync behavior is as expected where all stages sync to desired CL BUT polling is now limited by this pin value and will always check for changes using the pinned value as an upper limit.

          Net effect if this is polling still takes place but never detect any changes effectively nullifying poll behavior beyond the first run.

           

          I would expect a way to sync to a specific CL that has no side effect in other parts of the system.  Either the pin behavior is broken, or an extra parameter is required beyond pin to separate the two intent : the CL at which we wish to "pin" this job, and the CL at which we wish to sync in the populate implementation.
          *Karls Summary:*

          Polled builds (pipeline) should sync to "@now" when polling occurred not "@now" when sync occurs.

          _Usage case_ - On a busy system a long running job will pickup components of different ages.

          _Example_ -
           * Poll finds latest change 123
           * Build is queued
           * Change 124 added to path1.
           * Build starts
           * Path1 synced at 124
           * Change 125 and 126 submitted to path2.
           * Path1 sync completes, path2 sync starts at 126

          The desired behavior is Path2 should be synced at 123.

           

          *Original Description:*

          HI we have setup a parallel pipeline and are using SyncOnlyImpl's pin to ensure all parallel stages will build on the same changelist.

           

          Our first implementation worked as expected.  pin value was empty at first, causing it to sync to head then we extracted this value and propagated to other stages through a field.

          However other requirements required us to have that change be accessible in other parts of the build pipeline so we start the pipeline with a simple p4 command to get the head CL which is then propagated through the rest of the pipeline normally rather than through a field.

          Sync behavior is as expected where all stages sync to desired CL BUT polling is now limited by this pin value and will always check for changes using the pinned value as an upper limit.

          Net effect if this is polling still takes place but never detect any changes effectively nullifying poll behavior beyond the first run.

           

          I would expect a way to sync to a specific CL that has no side effect in other parts of the system.  Either the pin behavior is broken, or an extra parameter is required beyond pin to separate the two intent : the CL at which we wish to "pin" this job, and the CL at which we wish to sync in the populate implementation.
          p4karl Karl Wirth made changes -
          Description *Karls Summary:*

          Polled builds (pipeline) should sync to "@now" when polling occurred not "@now" when sync occurs.

          _Usage case_ - On a busy system a long running job will pickup components of different ages.

          _Example_ -
           * Poll finds latest change 123
           * Build is queued
           * Change 124 added to path1.
           * Build starts
           * Path1 synced at 124
           * Change 125 and 126 submitted to path2.
           * Path1 sync completes, path2 sync starts at 126

          The desired behavior is Path2 should be synced at 123.

           

          *Original Description:*

          HI we have setup a parallel pipeline and are using SyncOnlyImpl's pin to ensure all parallel stages will build on the same changelist.

           

          Our first implementation worked as expected.  pin value was empty at first, causing it to sync to head then we extracted this value and propagated to other stages through a field.

          However other requirements required us to have that change be accessible in other parts of the build pipeline so we start the pipeline with a simple p4 command to get the head CL which is then propagated through the rest of the pipeline normally rather than through a field.

          Sync behavior is as expected where all stages sync to desired CL BUT polling is now limited by this pin value and will always check for changes using the pinned value as an upper limit.

          Net effect if this is polling still takes place but never detect any changes effectively nullifying poll behavior beyond the first run.

           

          I would expect a way to sync to a specific CL that has no side effect in other parts of the system.  Either the pin behavior is broken, or an extra parameter is required beyond pin to separate the two intent : the CL at which we wish to "pin" this job, and the CL at which we wish to sync in the populate implementation.
          *Karls Summary:*

          Polled builds (pipeline) should sync to "@now" when polling occurred not "@now" when sync occurs.

          _Usage case_ - On a busy system a long running job will pickup components of different ages.

          _Example_ -
           * Poll finds latest change 123
           * Build is queued
           * Change 124 added to path1.
           * Build starts
           * Path1 synced at 124
           * Change 125 and 126 submitted to path2.
           * Path1 sync completes, path2 sync starts at 126

          The desired behavior is path1 and path2 should be synced at 123 (if that is not possible path 2 should be synced at 124).

           

          *Original Description:*

          HI we have setup a parallel pipeline and are using SyncOnlyImpl's pin to ensure all parallel stages will build on the same changelist.

           

          Our first implementation worked as expected.  pin value was empty at first, causing it to sync to head then we extracted this value and propagated to other stages through a field.

          However other requirements required us to have that change be accessible in other parts of the build pipeline so we start the pipeline with a simple p4 command to get the head CL which is then propagated through the rest of the pipeline normally rather than through a field.

          Sync behavior is as expected where all stages sync to desired CL BUT polling is now limited by this pin value and will always check for changes using the pinned value as an upper limit.

          Net effect if this is polling still takes place but never detect any changes effectively nullifying poll behavior beyond the first run.

           

          I would expect a way to sync to a specific CL that has no side effect in other parts of the system.  Either the pin behavior is broken, or an extra parameter is required beyond pin to separate the two intent : the CL at which we wish to "pin" this job, and the CL at which we wish to sync in the populate implementation.
          p4karl Karl Wirth made changes -
          Description *Karls Summary:*

          Polled builds (pipeline) should sync to "@now" when polling occurred not "@now" when sync occurs.

          _Usage case_ - On a busy system a long running job will pickup components of different ages.

          _Example_ -
           * Poll finds latest change 123
           * Build is queued
           * Change 124 added to path1.
           * Build starts
           * Path1 synced at 124
           * Change 125 and 126 submitted to path2.
           * Path1 sync completes, path2 sync starts at 126

          The desired behavior is path1 and path2 should be synced at 123 (if that is not possible path 2 should be synced at 124).

           

          *Original Description:*

          HI we have setup a parallel pipeline and are using SyncOnlyImpl's pin to ensure all parallel stages will build on the same changelist.

           

          Our first implementation worked as expected.  pin value was empty at first, causing it to sync to head then we extracted this value and propagated to other stages through a field.

          However other requirements required us to have that change be accessible in other parts of the build pipeline so we start the pipeline with a simple p4 command to get the head CL which is then propagated through the rest of the pipeline normally rather than through a field.

          Sync behavior is as expected where all stages sync to desired CL BUT polling is now limited by this pin value and will always check for changes using the pinned value as an upper limit.

          Net effect if this is polling still takes place but never detect any changes effectively nullifying poll behavior beyond the first run.

           

          I would expect a way to sync to a specific CL that has no side effect in other parts of the system.  Either the pin behavior is broken, or an extra parameter is required beyond pin to separate the two intent : the CL at which we wish to "pin" this job, and the CL at which we wish to sync in the populate implementation.
          *Karls Summary:*

          Polled builds (pipeline) should sync to "@now" when polling occurred not "@now" when sync occurs.

          _Usage case_ - On a busy system a long running job will pickup components of different ages.

          _Example_ -
           * Poll finds latest change 123
           * Build is queued
           * Change 124 added to path1.
           * Build starts
           * Path1 synced at 124
           * Change 125 and 126 submitted to path2.
           * Path1 sync completes, path2 sync starts at 126

          The desired behavior is path1 and path2 should be synced at 123 (if that is not possible in the Jenkins infrastructure path 1 and path2 should be synced at 124).

           

          *Original Description:*

          HI we have setup a parallel pipeline and are using SyncOnlyImpl's pin to ensure all parallel stages will build on the same changelist.

           

          Our first implementation worked as expected.  pin value was empty at first, causing it to sync to head then we extracted this value and propagated to other stages through a field.

          However other requirements required us to have that change be accessible in other parts of the build pipeline so we start the pipeline with a simple p4 command to get the head CL which is then propagated through the rest of the pipeline normally rather than through a field.

          Sync behavior is as expected where all stages sync to desired CL BUT polling is now limited by this pin value and will always check for changes using the pinned value as an upper limit.

          Net effect if this is polling still takes place but never detect any changes effectively nullifying poll behavior beyond the first run.

           

          I would expect a way to sync to a specific CL that has no side effect in other parts of the system.  Either the pin behavior is broken, or an extra parameter is required beyond pin to separate the two intent : the CL at which we wish to "pin" this job, and the CL at which we wish to sync in the populate implementation.
          p4karl Karl Wirth made changes -
          Labels P4_A P4_VERIFY P4_A
          p4karl Karl Wirth made changes -
          Assignee Karl Wirth [ p4karl ]
          Hide
          pgrohe Phil Grohe added a comment -

          Any news on this Karl Wirth ?

          Show
          pgrohe Phil Grohe added a comment - Any news on this Karl Wirth ?
          Hide
          p4karl Karl Wirth added a comment -

          I'm am updating this to highlight to the devs. Still a major cause of frustration.

          Show
          p4karl Karl Wirth added a comment - I'm am updating this to highlight to the devs. Still a major cause of frustration.
          p4karl Karl Wirth made changes -
          Priority Critical [ 2 ] Blocker [ 1 ]
          Hide
          pgrohe Phil Grohe added a comment - - edited

          I'd like to refine / clarify the use-case at the top of the issue description to match the way this issue is affecting myself & Eric Daigneault

          Parallel Pipeline Where All Parallel Job Parts Sync Same Perforce Path

          Polled builds (pipeline) should sync to "@now" when polling occurred not "@now" when sync occurs.

          Usage case - On a busy system a job with long-running parallel parts each syncing Perforce Path1,  will sync Path1 at different ages on the different parallel parts of the Jennkins job if new CLs are submitted while some parallel parts are still waiting in queue for available agents (or have begun but have not yet reached their sync step)

          Example - Each triggered build job has parallel parts Part1 and Part2

          • Poll finds latest change 123 on Path1
          • Build triggered by polling & added to Jenkins queue to wait for available agents.  Perforce @now for Path1 is equal to CL 123
          • Change 124 is submitted to Path1
          • Agent1 becomes available. Build job Part1 starts on Agent1. Sync to @now occurs on Agent1 when @now of Path1 is equal to CL 124
          • Job Part2 is still waiting in the Jenkins queue for an available agent.
          • Change 125 and 126 submitted to Path1
          • Agent2 becomes available. Build Job Part2 stars on Agent2. Sync to @now occurs on Agent2 when @now of Path1 is equal to CL 126
          • For a single build job, triggered when @now was equal to 123, the parallel parts are no longer coordinated.

          The desired behavior would be for all parallel parts of a job to sync at the same Perforce @now age.  Ideally this would be the @now age that initially triggered the job vis polling.

          What is missing IMO is a built-in plugin feature/option that allows injecting/passing the @now changelist number that caused SCM polling to trigger down into the triggered job itself.  This way once the polling triggers a job, the @now age that the parallel job parts will actually sync to becomes a fixed value, not dependent on when agents become available for that job & depedendent on what was submitted since the polling queued the job.

          Show
          pgrohe Phil Grohe added a comment - - edited I'd like to refine / clarify the use-case at the top of the issue description to match the way this issue is affecting myself & Eric Daigneault Parallel Pipeline Where All Parallel Job Parts Sync Same Perforce Path Polled builds (pipeline) should sync to "@now" when polling occurred not "@now" when sync occurs. Usage case - On a busy system a job with long-running parallel parts each syncing Perforce Path1,  will sync Path1 at different ages on the different parallel parts of the Jennkins job if new CLs are submitted while some parallel parts are still waiting in queue for available agents (or have begun but have not yet reached their sync step) Example - Each triggered build job has parallel parts Part1 and Part2 Poll finds latest change 123 on Path1 Build triggered by polling & added to Jenkins queue to wait for available agents.  Perforce @now for Path1 is equal to CL 123 Change 124 is submitted to Path1 Agent1 becomes available. Build job Part1 starts on Agent1. Sync to @now occurs on Agent1 when @now of Path1 is equal to CL 124 Job Part2 is still waiting in the Jenkins queue for an available agent. Change 125 and 126 submitted to Path1 Agent2 becomes available. Build Job Part2 stars on Agent2. Sync to @now occurs on Agent2 when @now of Path1 is equal to CL 126 For a single build job, triggered when @now was equal to 123, the parallel parts are no longer coordinated. The desired behavior would be for all parallel parts of a job to sync at the same Perforce @now age.  Ideally this would be the @now age that initially triggered the job vis polling. What is missing IMO is a built-in plugin feature/option that allows injecting/passing the @now changelist number that caused SCM polling to trigger down into the triggered job itself.  This way once the polling triggers a job, the @now age that the parallel job parts will actually sync to becomes a fixed value, not dependent on when agents become available for that job & depedendent on what was submitted since the polling queued the job.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            newtopian Eric Daigneault
            Votes:
            3 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated: