Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63879

Using pin to sync to changelist causes polling to no longer detect changes beyond pinned CL

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Karls Summary:

      Polled builds (pipeline) should sync to "@now" when polling occurred not "@now" when sync occurs.

      Usage case - On a busy system a long running job will pickup components of different ages.

      Example -

      • Poll finds latest change 123
      • Build is queued
      • Change 124 added to path1.
      • Build starts
      • Path1 synced at 124
      • Change 125 and 126 submitted to path2.
      • Path1 sync completes, path2 sync starts at 126

      The desired behavior is path1 and path2 should be synced at 123 (if that is not possible in the Jenkins infrastructure path 1 and path2 should be synced at 124).

       

      Original Description:

      HI we have setup a parallel pipeline and are using SyncOnlyImpl's pin to ensure all parallel stages will build on the same changelist.

       

      Our first implementation worked as expected.  pin value was empty at first, causing it to sync to head then we extracted this value and propagated to other stages through a field.

      However other requirements required us to have that change be accessible in other parts of the build pipeline so we start the pipeline with a simple p4 command to get the head CL which is then propagated through the rest of the pipeline normally rather than through a field.

      Sync behavior is as expected where all stages sync to desired CL BUT polling is now limited by this pin value and will always check for changes using the pinned value as an upper limit.

      Net effect if this is polling still takes place but never detect any changes effectively nullifying poll behavior beyond the first run.

       

      I would expect a way to sync to a specific CL that has no side effect in other parts of the system.  Either the pin behavior is broken, or an extra parameter is required beyond pin to separate the two intent : the CL at which we wish to "pin" this job, and the CL at which we wish to sync in the populate implementation.

        Attachments

          Activity

          Hide
          p4karl Karl Wirth added a comment -

          Hi Eric Daigneault - Please get me your pipeline code so I can test here. I'm wondering if disabling implicit sync may be a workaround.

          Show
          p4karl Karl Wirth added a comment - Hi Eric Daigneault - Please get me your pipeline code so I can test here. I'm wondering if disabling implicit sync may be a workaround.
          Hide
          p4karl Karl Wirth added a comment -

          Hi Eric Daigneault - Ignore that. Disabling implicit doe not change the behavior.

          P4Jenkins will only build if there is something to sync. This is intended because 'pin' is normally used with a build label and you only need to rebuild if the build label has changed. You dont want a job building if the label has not been updated.

          If I have two 'p4sync' steps in my pipeline script and only one of them is pinned the job will still build. From the polling log:

           

          # Pinned at 4262 
          P4: Polling with range: 4262,4262
          ... p4 changes -m20 //jenkins-master-PinPolledPipeline-Source-TEST/...@4262,4262 
          ... p4 repos -C 
          P4: Polling no changes found.
          
          # Not pinned
          P4: Polling with range: 4274,now
          ... p4 changes -m20 //jenkins-master-PinPolledPipeline-Source-TEST-FILES/...@4274,now 
          ... p4 change -o 4276 
          ... found change: 4276
          

          If after reading this you think its still not working as described please provide me more information about how you strcture everything. Are you syncing any new code in the job? Do you have multiple jobs or just steps? How are they connected to each other? Can you provide me the code?

           

          If it is working as described the workaround would be to trigger the jobs from perforce instead of using polling.

           

           

          Show
          p4karl Karl Wirth added a comment - Hi Eric Daigneault - Ignore that. Disabling implicit doe not change the behavior. P4Jenkins will only build if there is something to sync. This is intended because 'pin' is normally used with a build label and you only need to rebuild if the build label has changed. You dont want a job building if the label has not been updated. If I have two 'p4sync' steps in my pipeline script and only one of them is pinned the job will still build. From the polling log:   # Pinned at 4262 P4: Polling with range: 4262,4262 ... p4 changes -m20 //jenkins-master-PinPolledPipeline-Source-TEST/...@4262,4262 ... p4 repos -C P4: Polling no changes found. # Not pinned P4: Polling with range: 4274,now ... p4 changes -m20 //jenkins-master-PinPolledPipeline-Source-TEST-FILES/...@4274,now ... p4 change -o 4276 ... found change: 4276 If after reading this you think its still not working as described please provide me more information about how you strcture everything. Are you syncing any new code in the job? Do you have multiple jobs or just steps? How are they connected to each other? Can you provide me the code?   If it is working as described the workaround would be to trigger the jobs from perforce instead of using polling.    
          Hide
          newtopian Eric Daigneault added a comment - - edited

          Our pipeline resembles this pseudo seen below. The code is over-simplified and will likely not run as-is but it does give the essential of what is happening.

          Basically here we need to ensure that all stages in the parallel to build at the same CL. By default each will sync to head at the time they start, since each gets assigned to a different agent it's quite possible one or more is put in the queue to wait for available resources. During that time it's very likely that new commits occur thus having them all sync to head creates a "staggered" build across multiple changes which is undesirable.

          We used to do it in the way you suggests where one is not pinned, we struggled a bit to get the information across other stage but eventually made it work. however getting that changelist ID to propagate to other steps prooved to be challenging and it became much simpler to just fetch the change eagerly and pass it along the entire pipeline.

          Now we have averything working just dandy and all other steps that need access to the changelist can do so naturally... except polling no longer work because of this side effect. And since there are no obvious alternative ways to sync through the plugin that would not generate this side effect we are basically stuck. Temporary solution was to change the polling with a cron and force trigger the job each 30 mins or so.

          We experimented with perforce triggers and ran in another issue. Since the changelist is not (least wasnot at the time we tried) attached to the job being triggered we ran in situations where each commit woudl trigger a job but build system restrictions caused long queues. To make things worst when a job started it would pick head CL thus many jobs ended running at the same CL because of this which was very wasteful use of resources.

          Last thing we had not tried yet was to simply do without the populate and run the p4 sync directly with a p4.run.

          def pinnedchangelist = p4.run("changes", "-s", "submitted", "-m1", "${stream_location}/...")[0]['change']
          pipeline{
            options{
              skipDefaultCheckout()
            }
            parallel{
              stages{
                stage('someplatform'){
          	    agent('able-tagged'){}
                  step('sync'){
                    //see below for actual code
                  }
          		step('build'){
          		  ///
          		}
                }
                stage('someotherplatform'){
          	    agent('able-tagged'){}
                  step('sync'){
                    //see below for actual code
                  }
          		step('build'){
          		  ///
          		}
                }
                stage('yetanother'){
          	    agent('able-tagged'){}
                  step('sync'){
                    //see below for actual code
                  }
          		step('build'){
          		  ///
          		}
                }
              }
            }
          }
          

          the steps sync contains this code which does the actual sync

              def popImpl = [$class: "SyncOnlyImpl",
                      force: needs_force_sync(config),
                      have: true,
                      modtime: true,
                      parallel: null,
                      pin: pinnedChangelist,
                      quiet: false,
                      revert: false]
          
              echo "==  Workspace preparation complete, performing a sync ${popImpl}"
              checkout([
                      $class: 'PerforceScm',
                      credential: "${config.p4_credential}".toString(),
                      populate: popImpl,
                      workspace: ws
                      ])
          
          

          Edit:
          added skipDefaultCheckout() in option block at start of pipeline
          reformat pipeline code

          Show
          newtopian Eric Daigneault added a comment - - edited Our pipeline resembles this pseudo seen below. The code is over-simplified and will likely not run as-is but it does give the essential of what is happening. Basically here we need to ensure that all stages in the parallel to build at the same CL. By default each will sync to head at the time they start, since each gets assigned to a different agent it's quite possible one or more is put in the queue to wait for available resources. During that time it's very likely that new commits occur thus having them all sync to head creates a "staggered" build across multiple changes which is undesirable. We used to do it in the way you suggests where one is not pinned, we struggled a bit to get the information across other stage but eventually made it work. however getting that changelist ID to propagate to other steps prooved to be challenging and it became much simpler to just fetch the change eagerly and pass it along the entire pipeline. Now we have averything working just dandy and all other steps that need access to the changelist can do so naturally... except polling no longer work because of this side effect. And since there are no obvious alternative ways to sync through the plugin that would not generate this side effect we are basically stuck. Temporary solution was to change the polling with a cron and force trigger the job each 30 mins or so. We experimented with perforce triggers and ran in another issue. Since the changelist is not (least wasnot at the time we tried) attached to the job being triggered we ran in situations where each commit woudl trigger a job but build system restrictions caused long queues. To make things worst when a job started it would pick head CL thus many jobs ended running at the same CL because of this which was very wasteful use of resources. Last thing we had not tried yet was to simply do without the populate and run the p4 sync directly with a p4.run. def pinnedchangelist = p4.run( "changes" , "-s" , "submitted" , "-m1" , "${stream_location}/..." )[0][ 'change' ] pipeline{ options{ skipDefaultCheckout() } parallel{ stages{ stage( 'someplatform' ){ agent( 'able-tagged' ){} step( 'sync' ){ //see below for actual code } step( 'build' ){ /// } } stage( 'someotherplatform' ){ agent( 'able-tagged' ){} step( 'sync' ){ //see below for actual code } step( 'build' ){ /// } } stage( 'yetanother' ){ agent( 'able-tagged' ){} step( 'sync' ){ //see below for actual code } step( 'build' ){ /// } } } } } the steps sync contains this code which does the actual sync def popImpl = [$class: "SyncOnlyImpl" , force: needs_force_sync(config), have: true , modtime: true , parallel: null , pin: pinnedChangelist, quiet: false , revert: false ] echo "== Workspace preparation complete, performing a sync ${popImpl}" checkout([ $class: 'PerforceScm' , credential: "${config.p4_credential}" .toString(), populate: popImpl, workspace: ws ]) Edit: added skipDefaultCheckout() in option block at start of pipeline reformat pipeline code
          Hide
          p4karl Karl Wirth added a comment -

          Hi Eric Daigneault - A quick aplogy. I owe you a response on this one but I'm still working on it. Will get back to you when I have something.

          Show
          p4karl Karl Wirth added a comment - Hi Eric Daigneault - A quick aplogy. I owe you a response on this one but I'm still working on it. Will get back to you when I have something.
          Hide
          pgrohe Phil Grohe added a comment - - edited

          Hi Karl Wirth & Eric Daigneault

          Karl, for general context on how we are currently doing parallel builds & enforcing all parallel machines are synced to the same CL for a given build see this blog post from Riot Games, under the "Multiple Syncs in A Pipeline" section of the article:

          https://technology.riotgames.com/news/using-perforce-complex-jenkins-pipeline

          That's the core idea of what we're currently doing succesfully (with some tradeoffs and limitations that Eric is working on tackling with what he describes above).  Run a job with multiple seperate syncs occuring on parallel job parts & ensure they all sync to the same CL regardless of whether new CLs come into Perforce before some of the parallel parts begin (ex: waiting on an available agent).

          Apologies for the tangent but it gives an idea of what we're aiming for in a broader sense.

          Cheers.

          Show
          pgrohe Phil Grohe added a comment - - edited Hi Karl Wirth & Eric Daigneault Karl, for general context on how we are currently doing parallel builds & enforcing all parallel machines are synced to the same CL for a given build see this blog post from Riot Games, under the "Multiple Syncs in A Pipeline" section of the article: https://technology.riotgames.com/news/using-perforce-complex-jenkins-pipeline That's the core idea of what we're currently doing succesfully (with some tradeoffs and limitations that Eric is working on tackling with what he describes above).  Run a job with multiple seperate syncs occuring on parallel job parts & ensure they all sync to the same CL regardless of whether new CLs come into Perforce before some of the parallel parts begin (ex: waiting on an available agent). Apologies for the tangent but it gives an idea of what we're aiming for in a broader sense. Cheers.
          Hide
          arka_bb Arto Karppinen added a comment -

          I resolved the same problem by creating a separate polling pipeline, which just does the polling, and triggers the main pipeline with:

          build(job: 'main_pipeline_job', wait: false)

          Show
          arka_bb Arto Karppinen added a comment - I resolved the same problem by creating a separate polling pipeline, which just does the polling, and triggers the main pipeline with: build(job: 'main_pipeline_job', wait: false)

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            newtopian Eric Daigneault
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated: