This is our build pipeline:
1) Compile - checks files out of Perforce, cleans, configures for the target architecture and compiles. Archives the workspace for susequent steps in the pipeline.
2) Zip - builds the .ear file, verifies it, builds a complete shippable .zip of all project deliverables.
3) Quick test - runs a relatively short test suite.
2a) Zip is "promoted" if all tests in QuickTest pass. This delivers the Zip to the System test team (outside of Jenkins).
Now we didn't want each step to sync the files down from Perforce and compile afresh, the clone-workspace plugin seemed ideal for us. Unfortunately there are a number of problems with it including:
a) Only the latest workspace from step(1) is maintained. So if another change triggers a Compile whilst Zip is in progress, it could be that (3) Quick test actually tests that subsequent build and not the one it is supposed to be testing. This has regularly caused us considerable confusion as the jobs were out-of-step.
b) It takes so long ... our project is somewhat monolithic, including all binaries required within the workspace. The workspace.zip archive is about 1.5Gb and creating it takes around 50 minutes using clone-workspace-plugin vs. only 10 minutes for the actual compile, that's unacceptable. Un-zipping at the start of the steps (2) and (3) is not quite so bad but still 10-20 minutes. It seems to be something to do with the way data is piped between master and slave.
The solution we now have working is to replace this plugin by a shell script. This script takes 2 minutes to archive the workspace at the end of job (1) and 2 minutes at to un-archive it at the start of the other steps. It is attached to this report to illustrate our use case which we would like this plugin to support.
Use the script as follows:
At the end of job (1) invoke the script using "Execute Shell":
${JENKINS_BIN}/clone-workspace PACK "${BUILD_TAG}"
Optional INCLUDE= and EXCLUDE= filters can be used to limit the contents of the archive. We actually create 2 archives, one is much smaller containing just the information required for the "promote" step (2a) to work.
Also add a Post-Build action "Trigger parameterized build on other projects" to start job (2) using the parameter:
UPSTREAM_BUILD_TAG=${BUILD_TAG}
Jobs (2) and (3) are parameterized, accepting UPSTREAM_BUILD_TAG. The first task that each of these downstream jobs runs is:
Execute Shell
${JENKINS_BIN}/clone-workspace UNPACK "${UPSTREAM_BUILD_TAG}"
The archive dir must exist on the master node and be referenced by the JOBS_ARCHIVE environment variable.
The last job in the pipeline removes the redundant archive by invoking:
${JENKINS_BIN}/clone-workspace CLEAN "${UPSTREAM_BUILD_TAG}"
We also purge all archives from it overnight using a "cron" job, just to be sure.
Now the downstream jobs don't show the change history; that was solved by re-instating the clone-workspace-plugin but configured to archive just one file.
I did some tests, to archive a workspace from a slave to the master using "tar | ssh" takes around 2 minutes 15 seconds (2 min 10 with -z compression).
clone-workspace plugin took 22 minutes to do it from the same workspace on the same node.