Major We have a multibranch pipeline which after being triggered immediately splits into 6 stages for various linux and windows platforms. The sequence of each of the pipelines goes like this:
- Obtain branch source (checkout scm)
- Obtain platform specific thirdparties (p4sync)
- Obtain common thirdparties (p4sync)
- Build/Test/Archive/Etc
The problem I am having is pinning the thirdparty syncs to the same changelist as the branch, because each instance of a sync can set the P4_CHANGELIST environment variable (which seems to be updated globally) to a different value, so unless I perform a single sync ahead of all the others just to get the changelist and store it, I have a race condition.
What I would like to do is be able to put this at the start of my script, before I enter the pipeline block:
env.GLOBAL_P4_CHANGELIST = scm.getChangelist()
Which would ensure I have a global reference to the changelist that triggered the build to use for pinning the subsequent thirdparty syncs, that isn't overridden by any subsequent uses of p4sync.
I see that PerforceScm has a property P4Ref revision. Can this be exposed in the manner I'd like?
My current workaround consists of either wrapping an initial checkout in a lock step so that I can set the global variable after a first sync has completed (this delays all other parallel stages by the time it takes to sync, which can be up to 10 minutes currently)
lock("$env.JOB_NAME-GLOBAL_P4_CHANGELIST"){ if ( !env.GLOBAL_P4_CHANGELIST ){ checkout scm env.GLOBAL_P4_CHANGELIST = "$env.P4_CHANGELIST" env.GLOBAL_P4_CHANGELIST_STAGE = "$env.STAGE_NAME" } echo "GLOBAL_P4_CHANGELIST = $env.GLOBAL_P4_CHANGELIST" echo "GLOBAL_P4_CHANGELIST_STAGE = $env.GLOBAL_P4_CHANGELIST_STAGE" } if ( ! (env.STAGE_NAME == $env.GLOBAL_P4_CHANGELIST_STAGE) ){ checkout scm }
or by trying to use a lock on updating the variable immediately after a checkout scm, which still has a race condition.
[JENKINS-55315] Obtain target changelist from PerforceScm object in pipeline
Alisdair Robertson
created issue -
Alisdair Robertson
made changes -
| Description |
Original:
We have a multibranch pipeline which after being triggered immediately splits into 6 stages for various linux and windows platforms. The sequence of each of the pipelines goes like this: - Obtain branch source ({{checkout scm}}) - Obtain platform specific thirdparties ({{p4sync}}) - Obtain common thirdparties ({{p4sync}}) - Build/Test/Archive/Etc The problem I am having is pinning the thirdparty syncs to the same changelist as the branch, because each instance of a sync can set the {{P4_CHANGELIST}} environment variable (which seems to be updated globally) to a different value, so unless I perform a single sync ahead of all the others just to get the changelist and store it, I have a race condition. What I would like to do is be able to put this at the start of my script, before I enter the pipeline block: {code} env.GLOBAL_P4_CHANGELIST = scm.getChangelist() {code} Which would ensure I have a global reference to the changelist that triggered the build to use for pinning the subsequent thirdparty syncs, that isn't overridden by any subsequent uses of {{p4sync}}. I see that {{PerforceScm}} has a property {{P4Ref revision}}. Can this be exposed in the manner I'd like? My current workaround consists of either wrapping an initial checkout in a {{lock}} step so that I can set the global variable after a first sync has completed (this delays all other parallel stages by the time it takes to sync, which can be up to 10 minutes currently) {code:groovy} lock("$env.JOB_NAME-GLOBAL_P4_CHANGELIST"){ if ( !env.GLOBAL_P4_CHANGELIST ){ checkout source_control_management env.GLOBAL_P4_CHANGELIST = "$env.P4_CHANGELIST" env.GLOBAL_P4_CHANGELIST_STAGE = "$env.STAGE_NAME" } echo "GLOBAL_P4_CHANGELIST = $env.GLOBAL_P4_CHANGELIST" echo "GLOBAL_P4_CHANGELIST_STAGE = $env.GLOBAL_P4_CHANGELIST_STAGE" } if ( ! (env.STAGE_NAME == $env.GLOBAL_P4_CHANGELIST_STAGE) ){ checkout source_control_management } {code} or by trying to use a lock on updating the variable immediately after a {{checkout scm}}, which still has a race condition. |
New:
We have a multibranch pipeline which after being triggered immediately splits into 6 stages for various linux and windows platforms. The sequence of each of the pipelines goes like this: - Obtain branch source ({{checkout scm}}) - Obtain platform specific thirdparties ({{p4sync}}) - Obtain common thirdparties ({{p4sync}}) - Build/Test/Archive/Etc The problem I am having is pinning the thirdparty syncs to the same changelist as the branch, because each instance of a sync can set the {{P4_CHANGELIST}} environment variable (which seems to be updated globally) to a different value, so unless I perform a single sync ahead of all the others just to get the changelist and store it, I have a race condition. What I would like to do is be able to put this at the start of my script, before I enter the pipeline block: {code:java} env.GLOBAL_P4_CHANGELIST = scm.getChangelist() {code} Which would ensure I have a global reference to the changelist that triggered the build to use for pinning the subsequent thirdparty syncs, that isn't overridden by any subsequent uses of {{p4sync}}. I see that {{PerforceScm}} has a property {{P4Ref revision}}. Can this be exposed in the manner I'd like? My current workaround consists of either wrapping an initial checkout in a {{lock}} step so that I can set the global variable after a first sync has completed (this delays all other parallel stages by the time it takes to sync, which can be up to 10 minutes currently) {code} lock("$env.JOB_NAME-GLOBAL_P4_CHANGELIST"){ if ( !env.GLOBAL_P4_CHANGELIST ){ checkout scm env.GLOBAL_P4_CHANGELIST = "$env.P4_CHANGELIST" env.GLOBAL_P4_CHANGELIST_STAGE = "$env.STAGE_NAME" } echo "GLOBAL_P4_CHANGELIST = $env.GLOBAL_P4_CHANGELIST" echo "GLOBAL_P4_CHANGELIST_STAGE = $env.GLOBAL_P4_CHANGELIST_STAGE" } if ( ! (env.STAGE_NAME == $env.GLOBAL_P4_CHANGELIST_STAGE) ){ checkout scm } {code} or by trying to use a lock on updating the variable immediately after a {{checkout scm}}, which still has a race condition. |
Karl Wirth
made changes -
| Assignee | New: Karl Wirth [ p4karl ] |
Karl Wirth
made changes -
| Labels | New: P4_SUPPORT |
Karl Wirth
added a comment - There are a couple of ways to approach this. A temporary workaround for now is to cache the highest changelist across the depot (or part) instead of the sync mentioned above.
For example:
def ws = [$class: 'StreamWorkspaceImpl', charset: 'none', format: env.JOB_NAME + '-get-CL', pinHost: false, streamName: '//streams/main'] // Create P4 object def p4 = p4(credential: 'JenkinsMaster', workspace: ws) def Result = p4.run('changes','-m1','//...') GLOBAL_CHANGELIST = Result[0].change;
However we also see merrit in the plugin doing this so I'm passing it onto the developers.
Karl Wirth
added a comment - There are a couple of ways to approach this. A temporary workaround for now is to cache the highest changelist across the depot (or part) instead of the sync mentioned above.
For example:
def ws = [$class: 'StreamWorkspaceImpl' ,
charset: 'none' , format: env.JOB_NAME + '-get-CL' , pinHost: false ,
streamName: ' //streams/main' ]
// Create P4 object
def p4 = p4(credential: 'JenkinsMaster' , workspace: ws)
def Result = p4.run( 'changes' , '-m1' , ' //...' )
GLOBAL_CHANGELIST = Result[0].change;
However we also see merrit in the plugin doing this so I'm passing it onto the developers.
Karl Wirth
made changes -
| Assignee | Original: Karl Wirth [ p4karl ] |
Karl Wirth
made changes -
| Labels | Original: P4_SUPPORT | New: P4_VERIFY |
Paul Allen
made changes -
| Labels | Original: P4_VERIFY | New: P4_C |
Paul Allen
made changes -
| Labels | Original: P4_C | New: P4_DOC |
Hi alisdair_robertson,
Thanks for the suggestion. I can see that this would be useful so I will discuss with the developers.