Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63879

Using pin to sync to changelist causes polling to no longer detect changes beyond pinned CL

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Karls Summary:

      Polled builds (pipeline) should sync to "@now" when polling occurred not "@now" when sync occurs.

      Usage case - On a busy system a long running job will pickup components of different ages.

      Example -

      • Poll finds latest change 123
      • Build is queued
      • Change 124 added to path1.
      • Build starts
      • Path1 synced at 124
      • Change 125 and 126 submitted to path2.
      • Path1 sync completes, path2 sync starts at 126

      The desired behavior is path1 and path2 should be synced at 123 (if that is not possible in the Jenkins infrastructure path 1 and path2 should be synced at 124).

       

      Original Description:

      HI we have setup a parallel pipeline and are using SyncOnlyImpl's pin to ensure all parallel stages will build on the same changelist.

       

      Our first implementation worked as expected.  pin value was empty at first, causing it to sync to head then we extracted this value and propagated to other stages through a field.

      However other requirements required us to have that change be accessible in other parts of the build pipeline so we start the pipeline with a simple p4 command to get the head CL which is then propagated through the rest of the pipeline normally rather than through a field.

      Sync behavior is as expected where all stages sync to desired CL BUT polling is now limited by this pin value and will always check for changes using the pinned value as an upper limit.

      Net effect if this is polling still takes place but never detect any changes effectively nullifying poll behavior beyond the first run.

       

      I would expect a way to sync to a specific CL that has no side effect in other parts of the system.  Either the pin behavior is broken, or an extra parameter is required beyond pin to separate the two intent : the CL at which we wish to "pin" this job, and the CL at which we wish to sync in the populate implementation.

        Attachments

          Activity

          Hide
          pgrohe Phil Grohe added a comment - - edited

          Hi Karl Wirth & Eric Daigneault

          Karl, for general context on how we are currently doing parallel builds & enforcing all parallel machines are synced to the same CL for a given build see this blog post from Riot Games, under the "Multiple Syncs in A Pipeline" section of the article:

          https://technology.riotgames.com/news/using-perforce-complex-jenkins-pipeline

          That's the core idea of what we're currently doing succesfully (with some tradeoffs and limitations that Eric is working on tackling with what he describes above).  Run a job with multiple seperate syncs occuring on parallel job parts & ensure they all sync to the same CL regardless of whether new CLs come into Perforce before some of the parallel parts begin (ex: waiting on an available agent).

          Apologies for the tangent but it gives an idea of what we're aiming for in a broader sense.

          Cheers.

          Show
          pgrohe Phil Grohe added a comment - - edited Hi Karl Wirth & Eric Daigneault Karl, for general context on how we are currently doing parallel builds & enforcing all parallel machines are synced to the same CL for a given build see this blog post from Riot Games, under the "Multiple Syncs in A Pipeline" section of the article: https://technology.riotgames.com/news/using-perforce-complex-jenkins-pipeline That's the core idea of what we're currently doing succesfully (with some tradeoffs and limitations that Eric is working on tackling with what he describes above).  Run a job with multiple seperate syncs occuring on parallel job parts & ensure they all sync to the same CL regardless of whether new CLs come into Perforce before some of the parallel parts begin (ex: waiting on an available agent). Apologies for the tangent but it gives an idea of what we're aiming for in a broader sense. Cheers.
          Hide
          arka_bb Arto Karppinen added a comment -

          I resolved the same problem by creating a separate polling pipeline, which just does the polling, and triggers the main pipeline with:

          build(job: 'main_pipeline_job', wait: false)

          Show
          arka_bb Arto Karppinen added a comment - I resolved the same problem by creating a separate polling pipeline, which just does the polling, and triggers the main pipeline with: build(job: 'main_pipeline_job', wait: false)
          Hide
          pgrohe Phil Grohe added a comment -

          Any news on this Karl Wirth ?

          Show
          pgrohe Phil Grohe added a comment - Any news on this Karl Wirth ?
          Hide
          p4karl Karl Wirth added a comment -

          I'm am updating this to highlight to the devs. Still a major cause of frustration.

          Show
          p4karl Karl Wirth added a comment - I'm am updating this to highlight to the devs. Still a major cause of frustration.
          Hide
          pgrohe Phil Grohe added a comment - - edited

          I'd like to refine / clarify the use-case at the top of the issue description to match the way this issue is affecting myself & Eric Daigneault

          Parallel Pipeline Where All Parallel Job Parts Sync Same Perforce Path

          Polled builds (pipeline) should sync to "@now" when polling occurred not "@now" when sync occurs.

          Usage case - On a busy system a job with long-running parallel parts each syncing Perforce Path1,  will sync Path1 at different ages on the different parallel parts of the Jennkins job if new CLs are submitted while some parallel parts are still waiting in queue for available agents (or have begun but have not yet reached their sync step)

          Example - Each triggered build job has parallel parts Part1 and Part2

          • Poll finds latest change 123 on Path1
          • Build triggered by polling & added to Jenkins queue to wait for available agents.  Perforce @now for Path1 is equal to CL 123
          • Change 124 is submitted to Path1
          • Agent1 becomes available. Build job Part1 starts on Agent1. Sync to @now occurs on Agent1 when @now of Path1 is equal to CL 124
          • Job Part2 is still waiting in the Jenkins queue for an available agent.
          • Change 125 and 126 submitted to Path1
          • Agent2 becomes available. Build Job Part2 stars on Agent2. Sync to @now occurs on Agent2 when @now of Path1 is equal to CL 126
          • For a single build job, triggered when @now was equal to 123, the parallel parts are no longer coordinated.

          The desired behavior would be for all parallel parts of a job to sync at the same Perforce @now age.  Ideally this would be the @now age that initially triggered the job vis polling.

          What is missing IMO is a built-in plugin feature/option that allows injecting/passing the @now changelist number that caused SCM polling to trigger down into the triggered job itself.  This way once the polling triggers a job, the @now age that the parallel job parts will actually sync to becomes a fixed value, not dependent on when agents become available for that job & depedendent on what was submitted since the polling queued the job.

          Show
          pgrohe Phil Grohe added a comment - - edited I'd like to refine / clarify the use-case at the top of the issue description to match the way this issue is affecting myself & Eric Daigneault Parallel Pipeline Where All Parallel Job Parts Sync Same Perforce Path Polled builds (pipeline) should sync to "@now" when polling occurred not "@now" when sync occurs. Usage case - On a busy system a job with long-running parallel parts each syncing Perforce Path1,  will sync Path1 at different ages on the different parallel parts of the Jennkins job if new CLs are submitted while some parallel parts are still waiting in queue for available agents (or have begun but have not yet reached their sync step) Example - Each triggered build job has parallel parts Part1 and Part2 Poll finds latest change 123 on Path1 Build triggered by polling & added to Jenkins queue to wait for available agents.  Perforce @now for Path1 is equal to CL 123 Change 124 is submitted to Path1 Agent1 becomes available. Build job Part1 starts on Agent1. Sync to @now occurs on Agent1 when @now of Path1 is equal to CL 124 Job Part2 is still waiting in the Jenkins queue for an available agent. Change 125 and 126 submitted to Path1 Agent2 becomes available. Build Job Part2 stars on Agent2. Sync to @now occurs on Agent2 when @now of Path1 is equal to CL 126 For a single build job, triggered when @now was equal to 123, the parallel parts are no longer coordinated. The desired behavior would be for all parallel parts of a job to sync at the same Perforce @now age.  Ideally this would be the @now age that initially triggered the job vis polling. What is missing IMO is a built-in plugin feature/option that allows injecting/passing the @now changelist number that caused SCM polling to trigger down into the triggered job itself.  This way once the polling triggers a job, the @now age that the parallel job parts will actually sync to becomes a fixed value, not dependent on when agents become available for that job & depedendent on what was submitted since the polling queued the job.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            newtopian Eric Daigneault
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated: