Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-25346

"Copy files back to master node" doesn't copy to workspace

      I've a matrix job that uses the "Copy To Slave" plugin to "Copy files back to master node" as a post-build action. I then use the PostBuildScript plugin to run on the master node and process these files.

      However, even though Copy-to-Slave succeeds, the files aren't in the master workspace. Looking at the build output on the slaves shows:

      [copy-to-slave] Copying 'build/**/*', excluding nothing, from 'file:/home/foobar/jenkins/workspace/MyJob/label/foobar-ci-linux/' on 'hudson.slaves.DumbSlave@564c7adb' to 'file:/Users/foobar/.jenkins/jobs/MyJob/configurations/axis-label/foobar-ci-linux/workspace/' on the master.
      

      The path

      /Users/foobar/.jenkins/jobs/MyJob/configurations/axis-label/foobar-ci-linux/workspace/
      

      is not the workspace. The workspace is actually here:

      /Users/foobar/.jenkins/.jenkins/workspace/MyJob/
      

      Why does Copy-To-Slave copy the files to this weird path?

          [JENKINS-25346] "Copy files back to master node" doesn't copy to workspace

          Daniel Beck added a comment -

          Daniel Beck added a comment - This is a long obsolete default workspace location that was hard-coded into the plugin: https://github.com/jenkinsci/copy-to-slave-plugin/blob/master/src/main/java/com/michelin/cio/hudson/plugins/copytoslave/CopyToSlaveUtils.java#L57

          Dan Jarvis added a comment -

          I sponsored a $100 bounty on fixing this issue:
          https://freedomsponsors.org/issue/623/copy-files-back-to-master-node-doesnt-copy-to-workspace

          Please add to the bounty or fix the issue in the next 90 days to claim the money!

          Dan Jarvis added a comment - I sponsored a $100 bounty on fixing this issue: https://freedomsponsors.org/issue/623/copy-files-back-to-master-node-doesnt-copy-to-workspace Please add to the bounty or fix the issue in the next 90 days to claim the money!

          Vivekanand SV added a comment - - edited

          @Daniel, I have done the following code based on your comment ...

          < ----- DIFF START ---- >

          LINE 57

          DEL projectWorkspaceOnMaster = new FilePath(new File(freeStyleProject.getRootDir(), "workspace"));
          ADD projectWorkspaceOnMaster = freeStyleProject.getWorkspace();

          LINE 61

          DEL projectWorkspaceOnMaster = new FilePath(new File(project.getRootDir(), "workspace"));
          ADD projectWorkspaceOnMaster = project.getWorkspace();

          < ----- DIFF END ---- >

          Can I know your review comments for this please ?

          Vivekanand SV added a comment - - edited @Daniel, I have done the following code based on your comment ... < ----- DIFF START ---- > LINE 57 DEL projectWorkspaceOnMaster = new FilePath(new File(freeStyleProject.getRootDir(), "workspace")); ADD projectWorkspaceOnMaster = freeStyleProject.getWorkspace(); LINE 61 DEL projectWorkspaceOnMaster = new FilePath(new File(project.getRootDir(), "workspace")); ADD projectWorkspaceOnMaster = project.getWorkspace(); < ----- DIFF END ---- > Can I know your review comments for this please ?

          Daniel Beck added a comment -

          https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/AbstractProject.java#L498

          @deprecated as of 1.319

          This will not guarantee to point to the master node anyway.


          I'd try Jenkins.getInstance().getWorkspaceFor(project) as that'll return a workspace on the master node. Not sure whether it's correct in all cases in the context of this plugin as I don't use it.

          Daniel Beck added a comment - https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/AbstractProject.java#L498 @deprecated as of 1.319 This will not guarantee to point to the master node anyway. I'd try Jenkins.getInstance().getWorkspaceFor(project) as that'll return a workspace on the master node. Not sure whether it's correct in all cases in the context of this plugin as I don't use it.

          Vivekanand SV added a comment -

          I'll give it a try, thanks.

          Vivekanand SV added a comment - I'll give it a try, thanks.

          Vivekanand SV added a comment -

          @Daniel, getWorkspaceFor() accepts only an implementation of TopLevelItem, so this will work in case of a free style project, what about other cases ? I tried Jenkins javadoc for some time but I couldn't get a clue.

          Sorry for posting these questions, I don't have much knowledge about Jenkins code so looking for all help.

          Vivekanand SV added a comment - @Daniel, getWorkspaceFor() accepts only an implementation of TopLevelItem, so this will work in case of a free style project, what about other cases ? I tried Jenkins javadoc for some time but I couldn't get a clue. Sorry for posting these questions, I don't have much knowledge about Jenkins code so looking for all help.

          Vivekanand SV added a comment -

          @ Aaron, @Dan, @Daniel ... I fixed this and have the changes ready for release, before I release I need your feedback on the paths I chose (asking you people as you might have some real use-cases which can help me decide this)

          Assuming the following factors,

          Multiconfiguration Project name - C2S_Test_MultiConf
          Axis 1 label - machine
          Axis 1 Label values - Lin_Test1 and master

          Axis 2 label - temp_axis
          Axis 2 label values - 1 and 2

          the files copied on the master will be @
          /home/test/jenkins/workspace/C2S_Test_MultiConf/machine/Lin_Test1/temp_axis/1
          /home/test/jenkins/workspace/C2S_Test_MultiConf/machine/Lin_Test1/temp_axis/2

          other machine is master itself, so they wont be copied.

          I thought of copying the files into "/home/test/jenkins/workspace/C2S_Test_MultiConf/" itself, but i was thinking these files should be kept separate for each configuration.

          For freestyle job, workspace will be the default one "/home/test/jenkins/workspace/C2S_Test_FreeStyle/" where "C2S_Test_FreeStyle" is the project name.

          Coming to Maven style jobs, if you guys can give me example paths I can add that for those.

          Vivekanand SV added a comment - @ Aaron, @Dan, @Daniel ... I fixed this and have the changes ready for release, before I release I need your feedback on the paths I chose (asking you people as you might have some real use-cases which can help me decide this) Assuming the following factors, Multiconfiguration Project name - C2S_Test_MultiConf Axis 1 label - machine Axis 1 Label values - Lin_Test1 and master Axis 2 label - temp_axis Axis 2 label values - 1 and 2 the files copied on the master will be @ /home/test/jenkins/workspace/C2S_Test_MultiConf/machine/Lin_Test1/temp_axis/1 /home/test/jenkins/workspace/C2S_Test_MultiConf/machine/Lin_Test1/temp_axis/2 other machine is master itself, so they wont be copied. I thought of copying the files into "/home/test/jenkins/workspace/C2S_Test_MultiConf/" itself, but i was thinking these files should be kept separate for each configuration. For freestyle job, workspace will be the default one "/home/test/jenkins/workspace/C2S_Test_FreeStyle/" where "C2S_Test_FreeStyle" is the project name. Coming to Maven style jobs, if you guys can give me example paths I can add that for those.

          Daniel Beck added a comment -

          @Daniel, getWorkspaceFor() accepts only an implementation of TopLevelItem, so this will work in case of a free style project, what about other cases ? I tried Jenkins javadoc for some time but I couldn't get a clue.

          All projects generally are top-level items. It won't cover matrix axes and similar cases though, where project types divide their workspaces further.

          What you could try is to determine the TopLevelItem ancestor of the current build's project, and get the workspace of that on the current node (e.g. /jenkins/workspace/foo). If that's a prefix of the current actual workspace (/jenkins/workspace/foo/bar/baz), just append the rest (/bar/baz) to the workspace path on master

          It seems to me that the design of this feature is broken if you need to know about details like that, and an approximation and/or specifically adding support for job types is the best you can do.

          Daniel Beck added a comment - @Daniel, getWorkspaceFor() accepts only an implementation of TopLevelItem, so this will work in case of a free style project, what about other cases ? I tried Jenkins javadoc for some time but I couldn't get a clue. All projects generally are top-level items. It won't cover matrix axes and similar cases though, where project types divide their workspaces further. What you could try is to determine the TopLevelItem ancestor of the current build's project, and get the workspace of that on the current node (e.g. /jenkins/workspace/foo). If that's a prefix of the current actual workspace (/jenkins/workspace/foo/bar/baz), just append the rest (/bar/baz) to the workspace path on master It seems to me that the design of this feature is broken if you need to know about details like that, and an approximation and/or specifically adding support for job types is the best you can do.

          Vivekanand SV added a comment -

          What you could try is to determine the TopLevelItem ancestor of the current build's project, and get the workspace of that on the current node (e.g. /jenkins/workspace/foo). If that's a prefix of the current actual workspace (/jenkins/workspace/foo/bar/baz), just append the rest (/bar/baz) to the workspace path on master

          Yes, thats what I end up doing as stated in the comment before your reply above.

          Vivekanand SV added a comment - What you could try is to determine the TopLevelItem ancestor of the current build's project, and get the workspace of that on the current node (e.g. /jenkins/workspace/foo). If that's a prefix of the current actual workspace (/jenkins/workspace/foo/bar/baz), just append the rest (/bar/baz) to the workspace path on master Yes, thats what I end up doing as stated in the comment before your reply above.

          Daniel Beck added a comment -

          Yes, thats what I end up doing as stated in the comment before your reply above.

          The approach I describe in that paragraph doesn't require any particular knowledge about matrix axes or matrix projects; it wasn't clear from your comment whether yours does.

          Daniel Beck added a comment - Yes, thats what I end up doing as stated in the comment before your reply above. The approach I describe in that paragraph doesn't require any particular knowledge about matrix axes or matrix projects; it wasn't clear from your comment whether yours does.

          Vivekanand SV added a comment -

          Actually, yes. "machine/Lin_Test1/temp_axis/1" and "machine/Lin_Test1/temp_axis/2" parts of the path, from my comment, is generated after checking the prefix

          Vivekanand SV added a comment - Actually, yes. "machine/Lin_Test1/temp_axis/1" and "machine/Lin_Test1/temp_axis/2" parts of the path, from my comment, is generated after checking the prefix

          Daniel Beck added a comment -

          Oh, in that case I just didn't understand what you wrote. So you can interpret my recommendation as agreement to what you wrote.

          Daniel Beck added a comment - Oh, in that case I just didn't understand what you wrote. So you can interpret my recommendation as agreement to what you wrote.

          Vivekanand SV added a comment -

          Its okay, I wrote that way for the others to check the output path generated, I wanted to know if they agree to the paths that I am using now for different types of jobs.

          Vivekanand SV added a comment - Its okay, I wrote that way for the others to check the output path generated, I wanted to know if they agree to the paths that I am using now for different types of jobs.

          Dan Jarvis added a comment -

          Vivek, the target locations on the master that you described for the Multiconfiguration Project look good to me. I look forward to testing the new release of the plugin!

          Dan Jarvis added a comment - Vivek, the target locations on the master that you described for the Multiconfiguration Project look good to me. I look forward to testing the new release of the plugin!

          Code changed in jenkins
          User: Vivekanand S V
          Path:
          src/main/java/com/michelin/cio/hudson/plugins/copytoslave/CopyToSlaveUtils.java
          http://jenkins-ci.org/commit/copy-to-slave-plugin/6482c22c3a313c0ba8c83c553b1711a3d40c081e
          Log:
          [FIXED JENKINS-25346] - Addresses the workspace issue where files copied back to master are not in correct path.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vivekanand S V Path: src/main/java/com/michelin/cio/hudson/plugins/copytoslave/CopyToSlaveUtils.java http://jenkins-ci.org/commit/copy-to-slave-plugin/6482c22c3a313c0ba8c83c553b1711a3d40c081e Log: [FIXED JENKINS-25346] - Addresses the workspace issue where files copied back to master are not in correct path.

          Vivekanand SV added a comment -

          @Dan, you can cancel the bounty, thats not needed to make me work, I was (and still am) busy with my personal life, that gives me no time to look here

          Vivekanand SV added a comment - @Dan, you can cancel the bounty, thats not needed to make me work, I was (and still am) busy with my personal life, that gives me no time to look here

          Dan Jarvis added a comment - - edited

          I think you deserve the bounty as you made the fix we asked for and given me a lot of assistance. I think you need to claim the bounty through the website and then I will gladly pay it. Thanks!

          Dan Jarvis added a comment - - edited I think you deserve the bounty as you made the fix we asked for and given me a lot of assistance. I think you need to claim the bounty through the website and then I will gladly pay it. Thanks!

            svvivek Vivekanand SV
            paleozogt Aaron Simmons
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: