Code changed in jenkins
User: Paul Allen
Path:
src/main/java/org/jenkinsci/plugins/p4/PerforceScm.java
src/main/java/org/jenkinsci/plugins/p4/client/CleanupNotifier.java
src/main/java/org/jenkinsci/plugins/p4/publish/PublishNotifier.java
src/main/java/org/jenkinsci/plugins/p4/publish/PublishNotifierStep.java
src/main/java/org/jenkinsci/plugins/p4/tasks/RemoveClientTask.java
http://jenkins-ci.org/commit/p4-plugin/880b2a4a498d8aa72def30643949c1a3a0843f3a
Log:
Set environment to expand client name for Publish and Remove Client.

JENKINS-43378

Please try latest snapshot build to test fix.

https://jenkins.ci.cloudbees.com/job/plugins/job/p4-plugin/583/

Ready for release.

Thanks for the fix.

The names of the workspaces are now correct, %40 replaces @ and perforce workspaces are no longer appended with clone twice, e.g. .clone2.clone2.

But I'm getting lots of strange issues which can change everytime I build, such as files in the view mapping not being downloaded, files not in the view mapping being downloaded, getting changes from @null, files needing to be resolved and other odd behaviours.

Some context - I'm submitting 2+ files at the same time from within p4v. This triggers a script which for each file triggers a jenkins build. I've checked the parameters passed from the trigger script and they are still correct.

If I only submit 1 file or remove the ability to run concurrent builds, everything works as expected.

Also, the post-build Perforce delete client still doesn't seem to work, at least not everytime.

Do you use two different clients, one for Populate (syncing the files) and one for Publish (Shelve/Submitting the files)?  I normally recommend using the same name formatter but with a prefix of "-publish" for the latter.

The Publish workspace view should be as narrow as possible (either set the View map or use a Virtual Stream).

What reconcile options do you have set?

Are you using FreeStyle Job or Pipeline DSL?

I've modified my job to use 2 clients, one for populating and one for publishing.

If I run a single job it works, as you'd expect, but the publish client is not deleted from Perforce. Only the original populate client is deleted.

If I then run 2 builds concurrently I end up with more clashes such as files not being retrieved during population.

An example of an error that I receive is:
P4: Template workspace not found: jenkins-master-asset-generation-populate
P4: Unable to setup workspace: hudson.AbortException: P4: Undefined workspace: jenkins-master-asset-generation-populate.clone2
ERROR: Step ‘Perforce: Cleanup’ failed: P4: Unable to setup workspace: hudson.AbortException: P4: Undefined workspace: jenkins-master-asset-generation-populate.clone2
[WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done
Finished: FAILURE
At the time of this error, jenkins-master-asset-generation-populate had already been deleted as the first build had finished. jenkins-master-asset-generation-populate.clone2 did exist.

I'm using a freestyle job that receives some parameters containing information on a changed file and then converting that image from dds format to png format before submitting the new file back to Perforce. I've attached images of my config for both the populate task and publish task. An example of the view mapping I am using for population after variable insertion is:
//JenkinsTest/Games/test-project/assets/image.dds //jenkins-master-asset-generation-populate/image.dds
An example of the view mapping for the publish task is:
//JenkinsTest/Games/test-project/assets/... //jenkins-master-asset-generation-publish/...
-//JenkinsTest/Games/test-project/assets/....dds //jenkins-master-asset-generation-populate/....dds
-//JenkinsTest/Games/test-project/assets/....wav //jenkins-master-asset-generation-populate/....wav
(The wav line is because this task can also convert wav sounds to mp3 format)

The Post-Build Action 'Perforce: Cleanup' was only designed to remove the client used for populating the Jenkins workspace.  However I can see you might want to delete the Publish client.  The order of events is obviously important as the Post-Build 'Perforce: Cleanup' must occur after the publish and label actions.

I am unable to reproduce your specific issue, but think I can see the problem. If the initial (primary) client is deleted during the template operation for the second executor. I'm not sure how to deal with that other than aborting the second executor or trying to fall back to the initial primary name.

Why do you want to delete the clients between builds?

Jenkins -> Manage Jenkins -> Configure System -> 'Perforce: OnDelete Workspace Options'

Uncheck 'Delete Perforce client', but Check 'Delete Workspace files'

This way the client metadata will still be present for the 2nd (in flight) build.

I originally wanted to delete the clients to keep Perforce tidy while no jobs are running but it's not necessary for them to be deleted. I've done as you said and unticked the delete client box.

I've submitted 2 files in the same changelist which has resulted in the following Perforce clients being created:

• jenkins-master-asset-generation-populate

• jenkins-master-asset-generation-populate.clone2
• jenkins-master-asset-generation-populate.clone2.clone2
• jenkins-master-asset-generation-publish
• jenkins-master-asset-generation-publish.clone2

Already you can see an issue with a .clone2.clone2 client being created.

In this case, both jobs finished successfully but only the "second" job actually populated its workspace with what was mapped. The "first" job never got any files and this is some of the console output for that job:

 
P4 Task: reverting all pending and shelved revisions.
... p4 revert null/... +
 
... rm [abandoned files]
duration: (61ms)

P4 Task: cleaning workspace to match have list.
... p4 reconcile -w -f null/... +
 
duration: (64ms)

P4 Task: syncing files at change: 673
... p4 sync -q null/...@673 +
 
duration: (59ms)
 

The null references seem to be the problem.

If it would be helpful I can give you the entire console output for the builds.

I was looking at the logs and confused to see that the clean up happens first?  In addition I do not recognise the line:

[WS-CLEANUP] Deleting project workspace...

Is this a plugin or advanced Jenkins option?

I thought you were using the 'Perforce:Cleanup' step as a Post Build action.

The 
[WS-CLEANUP] Deleting project workspace...
message is the vanilla Jenkins workspace cleanup that deletes the local files.

This step occurs at both the start and end because I have "Delete workspace before build starts" ticked in the "Build Environment" section and also "Delete workspace when build is done" as the very last step in the "Post-build Actions".

It looks like you are using the ws-cleanup plugin (not included by default).

I have not tested this plugin's compatibility with the p4-plugin, please try remove the 'before build' cleanup.

Sorry it is included by default, but I still can't find it for pipeline only freestyle?

This is a freestyle job.

I've un-ticked every unnecessary box and removed all build and post-build steps to reduce the possibility of any conflicts.

When I run a single build I'm now left with a single dds image in the asset-generation workspace folder.

When I run 2 builds, 2 folders are created but only 1 of them contains a dds image.

In my test I have echoed the filename it should be mapping for each build and this is correct. But if I look at the mapping of the workspace in P4V both of the workspaces are mapped to the same file.

//JenkinsTest/Games/test-project/assets/bottom_game/barrel.dds //jenkins-master-asset-generation-populate/barrel.dds

//JenkinsTest/Games/test-project/assets/bottom_game/barrel.dds //jenkins-master-asset-generation-populate.clone2/barrel.dds

which it generated from:

${DEPOT_DIR}/${FILENAME} //jenkins-${NODE_NAME}-${JOB_NAME}-populate/${FILENAME}

${FILENAME} was backdrop.dds on build 1 and barrel.dds on build 2.

When the workspace is cloned during a concurrent build the 'expanded' view is copied from the parent.  So I would have expected:

Executor 1:  workspace(ws), view(//depot/A.dds //ws/A.dds)

Executor 2:  workspace(ws.clone2), view(//depot/A.dds //ws/A.dds)

You might find removing ${FILENAME} and using a view of ${DEPOT_DIR}/....dds might work better.  Your use of ${FILENAME} is an interesting variation I had not considered. 

Just done some testing and if I change my view mapping to:

${DEPOT_DIR}/....dds

or 

${DEPOT_DIR}/...

build 1 does not download any files but build 2 downloads correctly. I'm not using a specific file anywhere now so both instances should be identical.

Issue is currently down to the workspace directory location being null.

Should be c:/program files/jenkins/workspace/etc/

 
P4 Task: reverting all pending and shelved revisions.
... p4 revert null/... +
 
... rm [abandoned files]
duration: (62ms)

P4 Task: cleaning workspace to match have list.
... p4 reconcile -w -f null/... +
 
duration: (85ms)

P4 Task: syncing files at change: 775
... p4 sync -q null/...@775 +
 
duration: (59ms)

The concurrent logic has changed since 1.6.x.  Issue Postponed.

(Please open a new issue and reference this one if required)