-
Bug
-
Resolution: Unresolved
-
Major
-
None
copy-to-slave does nothing when the job runs on the master. This is the wrong behavior if the job is a multi-configuration project, because these jobs run with a modified workspace, depending on the parameter of the current job.
What the plugin should do when the job is running on the master, is to check if the file in the 'default' workspace is identical to the target file (it won't be in a multi-config build) and if not, copy it to the new workspace.
For example, the job on master has this workspace:
$ pwd
.../hudson/jobs/bootstrap-toolchain/workspace
$ ls
buildconf config-env.yml config.yml my-bootstrap.sh TARGET
where TARGET is the first category for the parametrization. The job will never run in this 'workspace' dir, but somewhere deep in the TARGET subdir. So copy-to-slave should copy the file from this workspace into the deep 'multi-config' workspace.
For example, when my job runs, these variables are set:
JOB_NAME=bootstrap-toolchain/TARGET=gnulinux,arch=amd64,branch=master
WORKSPACE=.../hudson/jobs/bootstrap-toolchain/workspace/TARGET/gnulinux/arch/amd64/branch/master
I hope this report is clear about the issue.
[JENKINS-8346] Copy to slave incompatible with multi configuration project
psoetens
added a comment - Well, if you click on 'New Job' there will be a radio button for 'Build multi-configuration project'. I don't have an option for 'matrix project', although in the config page of the created project, it's called 'Configuration Matrix'.
The point of a multi-config project is that it works in a deep subdir of the project's workspace. Somehow, the other tools, like the version control tools, know that they have to check-out to that directory, but your plugin doesn't even try, it just does the 'master' check instead of checking if source and target file are the same.
What I'd like to be the behavior of this plugin is similar to this script (bash syntax):
for slave in $nodes; do
- always copy if: slave is not master OR source and target file are not the same file.
if [ $slave -neq "master" -o -not $HUDSON/jobs/$JOB_NAME/$file_to_copy -ef $HOME/$WORKSPACE/$file_to_copy ]; then
scp $HUDSON/jobs/$JOB_NAME/$file_to_copy $slave:$WORKSPACE
fi
done
So to use jobs/$JOB_NAME as source directory and to use $WORKSPACE as target directory.
The JOB_NAME in the original post shows how it is actually encoded,so you'd need to strip everything after the last '/'.
Peter
psoetens
added a comment - Well, if you click on 'New Job' there will be a radio button for 'Build multi-configuration project'. I don't have an option for 'matrix project', although in the config page of the created project, it's called 'Configuration Matrix'.
The point of a multi-config project is that it works in a deep subdir of the project's workspace. Somehow, the other tools, like the version control tools, know that they have to check-out to that directory, but your plugin doesn't even try, it just does the 'master' check instead of checking if source and target file are the same.
What I'd like to be the behavior of this plugin is similar to this script (bash syntax):
for slave in $nodes; do
always copy if: slave is not master OR source and target file are not the same file.
if [ $slave -neq "master" -o -not $HUDSON/jobs/$JOB_NAME/$file_to_copy -ef $HOME/$WORKSPACE/$file_to_copy ]; then
scp $HUDSON/jobs/$JOB_NAME/$file_to_copy $slave:$WORKSPACE
fi
done
So to use jobs/$JOB_NAME as source directory and to use $WORKSPACE as target directory.
The JOB_NAME in the original post shows how it is actually encoded,so you'd need to strip everything after the last '/'.
Peter 
Romain Seguy
added a comment - Thanks for the precision, I think I'll have to dive into the way Hudson handles that.
In fact, when you say
you're absolutely right, but it's on purpose/required: Hudson offers tons of methods to get the current workspace and so on, "but" (this should not be "but" but "as a consequence" in a normal case) these methods return pointer to the actual location (on the slave, etc.).the other tools, like the version control tools, know that they have to check-out to that directory, but your plugin doesn't even try, it just does the 'master' check instead of checking if source and target file are the same
The only one way to get the workspace on the master, for example when doing a copy-to-slave, is then to build the pointer manually (by appending /workspace to the job name). Which means: All possible cases handled by Hudson have to be manually handled in the Copy To Slave plugin.
Romain Seguy
added a comment - Thanks for the precision, I think I'll have to dive into the way Hudson handles that.
In fact, when you say the other tools, like the version control tools, know that they have to check-out to that directory, but your plugin doesn't even try, it just does the 'master' check instead of checking if source and target file are the same you're absolutely right, but it's on purpose/required: Hudson offers tons of methods to get the current workspace and so on, "but" (this should not be "but" but "as a consequence" in a normal case) these methods return pointer to the actual location (on the slave, etc.).
The only one way to get the workspace on the master, for example when doing a copy-to-slave, is then to build the pointer manually (by appending /workspace to the job name). Which means: All possible cases handled by Hudson have to be manually handled in the Copy To Slave plugin. 
Brad Leonard
added a comment - I am running into this problem when trying to copy from slave workspace to master workspace in a matrix (multi-configuration) project.
The error as seen in console output on slave is:
[copy-to-slave] Copying '***.msi', excluding nothing, from 'file:/C:/jenkins/workspace/project-name/label/windows/' on 'hudson.slaves.DumbSlave@e5e309d5' to 'file:/Users/build/.jenkins/jobs/project-name/configurations/axis-label/windows/workspace/' on the master.
ERROR: Publisher com.michelin.cio.hudson.plugins.copytoslave.CopyToMasterNotifier aborted due to exception
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at hudson.remoting.Request$1.get(Request.java:218)
at hudson.remoting.Request$1.get(Request.java:184)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55)
at com.michelin.cio.hudson.plugins.copytoslave.MyFilePath.copyRecursiveTo(MyFilePath.java:145)
at com.michelin.cio.hudson.plugins.copytoslave.CopyToSlaveUtils.hudson5977(CopyToSlaveUtils.java:102)
at com.michelin.cio.hudson.plugins.copytoslave.CopyToMasterNotifier.perform(CopyToMasterNotifier.java:81)
at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:694)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:669)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:647)
at hudson.model.Build$RunnerImpl.cleanUp(Build.java:171)
at hudson.model.Run.run(Run.java:1448)
at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:230)
Each unsuccessful build creates an empty directory in the slaves workspace titled 'copy-to-slave8439381225529586145.dir'. The directories listed at the top of the console output do in fact exist.
Brad Leonard
added a comment - I am running into this problem when trying to copy from slave workspace to master workspace in a matrix (multi-configuration) project.
The error as seen in console output on slave is:
[copy-to-slave] Copying '***.msi', excluding nothing, from 'file:/C:/jenkins/workspace/project-name/label/windows/' on 'hudson.slaves.DumbSlave@e5e309d5' to 'file:/Users/build/.jenkins/jobs/project-name/configurations/axis-label/windows/workspace/' on the master.
ERROR: Publisher com.michelin.cio.hudson.plugins.copytoslave.CopyToMasterNotifier aborted due to exception
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at hudson.remoting.Request$1.get(Request.java:218)
at hudson.remoting.Request$1.get(Request.java:184)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55)
at com.michelin.cio.hudson.plugins.copytoslave.MyFilePath.copyRecursiveTo(MyFilePath.java:145)
at com.michelin.cio.hudson.plugins.copytoslave.CopyToSlaveUtils.hudson5977(CopyToSlaveUtils.java:102)
at com.michelin.cio.hudson.plugins.copytoslave.CopyToMasterNotifier.perform(CopyToMasterNotifier.java:81)
at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:694)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:669)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:647)
at hudson.model.Build$RunnerImpl.cleanUp(Build.java:171)
at hudson.model.Run.run(Run.java:1448)
at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:230)
Each unsuccessful build creates an empty directory in the slaves workspace titled 'copy-to-slave8439381225529586145.dir'. The directories listed at the top of the console output do in fact exist. 
Andy Bigos
added a comment - I have a similar problem with matrix configs. I'm copying the changelog.xml file into my workspace. So for a simple build I have:
jobs/$JOB_NAME/builds/$BUILD_ID/changelog.xml
with path set relative to HUDSON_HOME. This works for a simple project, however for a matrix project the $JOB_NAME gets expanded to the matrix element job name - this means the copy will fail as the source directory is not on the master. In the case of a matrix projects I think the plugin needs to use the 'base' project path, not the path for a matrix element.
Can you confirm this is an issue? I can log it as new issue if you think it's different to the current issue.
Thanks
Andy
Andy Bigos
added a comment -
I have a similar problem with matrix configs. I'm copying the changelog.xml file into my workspace. So for a simple build I have:
jobs/$JOB_NAME/builds/$BUILD_ID/changelog.xml
with path set relative to HUDSON_HOME. This works for a simple project, however for a matrix project the $JOB_NAME gets expanded to the matrix element job name - this means the copy will fail as the source directory is not on the master. In the case of a matrix projects I think the plugin needs to use the 'base' project path, not the path for a matrix element.
Can you confirm this is an issue? I can log it as new issue if you think it's different to the current issue.
Thanks
Andy 
Well, I can't say it's clear to me...
One first thing: Does multi-configuration project == matrix project? If yes, then I'll need more details since I don't use this kind of jobs.