Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-25346

"Copy files back to master node" doesn't copy to workspace

    XMLWordPrintable

Details

    Description

      I've a matrix job that uses the "Copy To Slave" plugin to "Copy files back to master node" as a post-build action. I then use the PostBuildScript plugin to run on the master node and process these files.

      However, even though Copy-to-Slave succeeds, the files aren't in the master workspace. Looking at the build output on the slaves shows:

      [copy-to-slave] Copying 'build/**/*', excluding nothing, from 'file:/home/foobar/jenkins/workspace/MyJob/label/foobar-ci-linux/' on 'hudson.slaves.DumbSlave@564c7adb' to 'file:/Users/foobar/.jenkins/jobs/MyJob/configurations/axis-label/foobar-ci-linux/workspace/' on the master.
      

      The path

      /Users/foobar/.jenkins/jobs/MyJob/configurations/axis-label/foobar-ci-linux/workspace/
      

      is not the workspace. The workspace is actually here:

      /Users/foobar/.jenkins/.jenkins/workspace/MyJob/
      

      Why does Copy-To-Slave copy the files to this weird path?

      Attachments

        Activity

          paleozogt Aaron Simmons created issue -
          paleozogt Aaron Simmons made changes -
          Field Original Value New Value
          Description I've a matrix job that uses the "Copy To Slave" plugin to "Copy files back to master node" as a post-build action. I then use the PostBuildScript plugin to run on the master node to process these files.

          However, even though Copy-to-Slave succeeds, the files aren't in the master workspace. Look at the build output on the slaves shows:
          {code}
          [copy-to-slave] Copying 'build/**/*', excluding nothing, from 'file:/home/foobar/jenkins/workspace/JenkinsSandbox/label/foobar-ci-linux/' on 'hudson.slaves.DumbSlave@564c7adb' to 'file:/Users/foobar/.jenkins/jobs/JenkinsSandbox/configurations/axis-label/foobar-ci-linux/workspace/' on the master.
          {code}

          The path
          {code}
          /Users/foobar/.jenkins/jobs/JenkinsSandbox/configurations/axis-label/foobar-ci-linux/workspace/
          {code}
          is not the workspace. The workspace is actually here:
          {code}
          /Users/foobar/.jenkins/.jenkins/workspace/JenkinsSandbox/
          {code}

          Why does Copy-To-Slave copy the files to this weird path?

          I've a matrix job that uses the "Copy To Slave" plugin to "Copy files back to master node" as a post-build action. I then use the PostBuildScript plugin to run on the master node and process these files.

          However, even though Copy-to-Slave succeeds, the files aren't in the master workspace. Look at the build output on the slaves shows:
          {code}
          [copy-to-slave] Copying 'build/**/*', excluding nothing, from 'file:/home/foobar/jenkins/workspace/JenkinsSandbox/label/foobar-ci-linux/' on 'hudson.slaves.DumbSlave@564c7adb' to 'file:/Users/foobar/.jenkins/jobs/JenkinsSandbox/configurations/axis-label/foobar-ci-linux/workspace/' on the master.
          {code}

          The path
          {code}
          /Users/foobar/.jenkins/jobs/JenkinsSandbox/configurations/axis-label/foobar-ci-linux/workspace/
          {code}
          is not the workspace. The workspace is actually here:
          {code}
          /Users/foobar/.jenkins/.jenkins/workspace/JenkinsSandbox/
          {code}

          Why does Copy-To-Slave copy the files to this weird path?

          paleozogt Aaron Simmons made changes -
          Description I've a matrix job that uses the "Copy To Slave" plugin to "Copy files back to master node" as a post-build action. I then use the PostBuildScript plugin to run on the master node and process these files.

          However, even though Copy-to-Slave succeeds, the files aren't in the master workspace. Look at the build output on the slaves shows:
          {code}
          [copy-to-slave] Copying 'build/**/*', excluding nothing, from 'file:/home/foobar/jenkins/workspace/JenkinsSandbox/label/foobar-ci-linux/' on 'hudson.slaves.DumbSlave@564c7adb' to 'file:/Users/foobar/.jenkins/jobs/JenkinsSandbox/configurations/axis-label/foobar-ci-linux/workspace/' on the master.
          {code}

          The path
          {code}
          /Users/foobar/.jenkins/jobs/JenkinsSandbox/configurations/axis-label/foobar-ci-linux/workspace/
          {code}
          is not the workspace. The workspace is actually here:
          {code}
          /Users/foobar/.jenkins/.jenkins/workspace/JenkinsSandbox/
          {code}

          Why does Copy-To-Slave copy the files to this weird path?

          I've a matrix job that uses the "Copy To Slave" plugin to "Copy files back to master node" as a post-build action. I then use the PostBuildScript plugin to run on the master node and process these files.

          However, even though Copy-to-Slave succeeds, the files aren't in the master workspace. Looking at the build output on the slaves shows:
          {code}
          [copy-to-slave] Copying 'build/**/*', excluding nothing, from 'file:/home/foobar/jenkins/workspace/JenkinsSandbox/label/foobar-ci-linux/' on 'hudson.slaves.DumbSlave@564c7adb' to 'file:/Users/foobar/.jenkins/jobs/JenkinsSandbox/configurations/axis-label/foobar-ci-linux/workspace/' on the master.
          {code}

          The path
          {code}
          /Users/foobar/.jenkins/jobs/JenkinsSandbox/configurations/axis-label/foobar-ci-linux/workspace/
          {code}
          is not the workspace. The workspace is actually here:
          {code}
          /Users/foobar/.jenkins/.jenkins/workspace/JenkinsSandbox/
          {code}

          Why does Copy-To-Slave copy the files to this weird path?

          paleozogt Aaron Simmons made changes -
          Description I've a matrix job that uses the "Copy To Slave" plugin to "Copy files back to master node" as a post-build action. I then use the PostBuildScript plugin to run on the master node and process these files.

          However, even though Copy-to-Slave succeeds, the files aren't in the master workspace. Looking at the build output on the slaves shows:
          {code}
          [copy-to-slave] Copying 'build/**/*', excluding nothing, from 'file:/home/foobar/jenkins/workspace/JenkinsSandbox/label/foobar-ci-linux/' on 'hudson.slaves.DumbSlave@564c7adb' to 'file:/Users/foobar/.jenkins/jobs/JenkinsSandbox/configurations/axis-label/foobar-ci-linux/workspace/' on the master.
          {code}

          The path
          {code}
          /Users/foobar/.jenkins/jobs/JenkinsSandbox/configurations/axis-label/foobar-ci-linux/workspace/
          {code}
          is not the workspace. The workspace is actually here:
          {code}
          /Users/foobar/.jenkins/.jenkins/workspace/JenkinsSandbox/
          {code}

          Why does Copy-To-Slave copy the files to this weird path?

          I've a matrix job that uses the "Copy To Slave" plugin to "Copy files back to master node" as a post-build action. I then use the PostBuildScript plugin to run on the master node and process these files.

          However, even though Copy-to-Slave succeeds, the files aren't in the master workspace. Looking at the build output on the slaves shows:
          {code}
          [copy-to-slave] Copying 'build/**/*', excluding nothing, from 'file:/home/foobar/jenkins/workspace/MyJob/label/foobar-ci-linux/' on 'hudson.slaves.DumbSlave@564c7adb' to 'file:/Users/foobar/.jenkins/jobs/MyJob/configurations/axis-label/foobar-ci-linux/workspace/' on the master.
          {code}

          The path
          {code}
          /Users/foobar/.jenkins/jobs/MyJob/configurations/axis-label/foobar-ci-linux/workspace/
          {code}
          is not the workspace. The workspace is actually here:
          {code}
          /Users/foobar/.jenkins/.jenkins/workspace/MyJob/
          {code}

          Why does Copy-To-Slave copy the files to this weird path?

          danielbeck Daniel Beck added a comment - This is a long obsolete default workspace location that was hard-coded into the plugin: https://github.com/jenkinsci/copy-to-slave-plugin/blob/master/src/main/java/com/michelin/cio/hudson/plugins/copytoslave/CopyToSlaveUtils.java#L57
          daj Dan Jarvis added a comment -

          I sponsored a $100 bounty on fixing this issue:
          https://freedomsponsors.org/issue/623/copy-files-back-to-master-node-doesnt-copy-to-workspace

          Please add to the bounty or fix the issue in the next 90 days to claim the money!

          daj Dan Jarvis added a comment - I sponsored a $100 bounty on fixing this issue: https://freedomsponsors.org/issue/623/copy-files-back-to-master-node-doesnt-copy-to-workspace Please add to the bounty or fix the issue in the next 90 days to claim the money!
          svvivek Vivekanand SV added a comment - - edited

          @Daniel, I have done the following code based on your comment ...

          < ----- DIFF START ---- >

          LINE 57

          DEL projectWorkspaceOnMaster = new FilePath(new File(freeStyleProject.getRootDir(), "workspace"));
          ADD projectWorkspaceOnMaster = freeStyleProject.getWorkspace();

          LINE 61

          DEL projectWorkspaceOnMaster = new FilePath(new File(project.getRootDir(), "workspace"));
          ADD projectWorkspaceOnMaster = project.getWorkspace();

          < ----- DIFF END ---- >

          Can I know your review comments for this please ?

          svvivek Vivekanand SV added a comment - - edited @Daniel, I have done the following code based on your comment ... < ----- DIFF START ---- > LINE 57 DEL projectWorkspaceOnMaster = new FilePath(new File(freeStyleProject.getRootDir(), "workspace")); ADD projectWorkspaceOnMaster = freeStyleProject.getWorkspace(); LINE 61 DEL projectWorkspaceOnMaster = new FilePath(new File(project.getRootDir(), "workspace")); ADD projectWorkspaceOnMaster = project.getWorkspace(); < ----- DIFF END ---- > Can I know your review comments for this please ?
          danielbeck Daniel Beck added a comment -

          https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/AbstractProject.java#L498

          @deprecated as of 1.319

          This will not guarantee to point to the master node anyway.


          I'd try Jenkins.getInstance().getWorkspaceFor(project) as that'll return a workspace on the master node. Not sure whether it's correct in all cases in the context of this plugin as I don't use it.

          danielbeck Daniel Beck added a comment - https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/AbstractProject.java#L498 @deprecated as of 1.319 This will not guarantee to point to the master node anyway. I'd try Jenkins.getInstance().getWorkspaceFor(project) as that'll return a workspace on the master node. Not sure whether it's correct in all cases in the context of this plugin as I don't use it.
          svvivek Vivekanand SV added a comment -

          I'll give it a try, thanks.

          svvivek Vivekanand SV added a comment - I'll give it a try, thanks.
          svvivek Vivekanand SV added a comment -

          @Daniel, getWorkspaceFor() accepts only an implementation of TopLevelItem, so this will work in case of a free style project, what about other cases ? I tried Jenkins javadoc for some time but I couldn't get a clue.

          Sorry for posting these questions, I don't have much knowledge about Jenkins code so looking for all help.

          svvivek Vivekanand SV added a comment - @Daniel, getWorkspaceFor() accepts only an implementation of TopLevelItem, so this will work in case of a free style project, what about other cases ? I tried Jenkins javadoc for some time but I couldn't get a clue. Sorry for posting these questions, I don't have much knowledge about Jenkins code so looking for all help.
          svvivek Vivekanand SV added a comment -

          @ Aaron, @Dan, @Daniel ... I fixed this and have the changes ready for release, before I release I need your feedback on the paths I chose (asking you people as you might have some real use-cases which can help me decide this)

          Assuming the following factors,

          Multiconfiguration Project name - C2S_Test_MultiConf
          Axis 1 label - machine
          Axis 1 Label values - Lin_Test1 and master

          Axis 2 label - temp_axis
          Axis 2 label values - 1 and 2

          the files copied on the master will be @
          /home/test/jenkins/workspace/C2S_Test_MultiConf/machine/Lin_Test1/temp_axis/1
          /home/test/jenkins/workspace/C2S_Test_MultiConf/machine/Lin_Test1/temp_axis/2

          other machine is master itself, so they wont be copied.

          I thought of copying the files into "/home/test/jenkins/workspace/C2S_Test_MultiConf/" itself, but i was thinking these files should be kept separate for each configuration.

          For freestyle job, workspace will be the default one "/home/test/jenkins/workspace/C2S_Test_FreeStyle/" where "C2S_Test_FreeStyle" is the project name.

          Coming to Maven style jobs, if you guys can give me example paths I can add that for those.

          svvivek Vivekanand SV added a comment - @ Aaron, @Dan, @Daniel ... I fixed this and have the changes ready for release, before I release I need your feedback on the paths I chose (asking you people as you might have some real use-cases which can help me decide this) Assuming the following factors, Multiconfiguration Project name - C2S_Test_MultiConf Axis 1 label - machine Axis 1 Label values - Lin_Test1 and master Axis 2 label - temp_axis Axis 2 label values - 1 and 2 the files copied on the master will be @ /home/test/jenkins/workspace/C2S_Test_MultiConf/machine/Lin_Test1/temp_axis/1 /home/test/jenkins/workspace/C2S_Test_MultiConf/machine/Lin_Test1/temp_axis/2 other machine is master itself, so they wont be copied. I thought of copying the files into "/home/test/jenkins/workspace/C2S_Test_MultiConf/" itself, but i was thinking these files should be kept separate for each configuration. For freestyle job, workspace will be the default one "/home/test/jenkins/workspace/C2S_Test_FreeStyle/" where "C2S_Test_FreeStyle" is the project name. Coming to Maven style jobs, if you guys can give me example paths I can add that for those.
          danielbeck Daniel Beck added a comment -

          @Daniel, getWorkspaceFor() accepts only an implementation of TopLevelItem, so this will work in case of a free style project, what about other cases ? I tried Jenkins javadoc for some time but I couldn't get a clue.

          All projects generally are top-level items. It won't cover matrix axes and similar cases though, where project types divide their workspaces further.

          What you could try is to determine the TopLevelItem ancestor of the current build's project, and get the workspace of that on the current node (e.g. /jenkins/workspace/foo). If that's a prefix of the current actual workspace (/jenkins/workspace/foo/bar/baz), just append the rest (/bar/baz) to the workspace path on master

          It seems to me that the design of this feature is broken if you need to know about details like that, and an approximation and/or specifically adding support for job types is the best you can do.

          danielbeck Daniel Beck added a comment - @Daniel, getWorkspaceFor() accepts only an implementation of TopLevelItem, so this will work in case of a free style project, what about other cases ? I tried Jenkins javadoc for some time but I couldn't get a clue. All projects generally are top-level items. It won't cover matrix axes and similar cases though, where project types divide their workspaces further. What you could try is to determine the TopLevelItem ancestor of the current build's project, and get the workspace of that on the current node (e.g. /jenkins/workspace/foo). If that's a prefix of the current actual workspace (/jenkins/workspace/foo/bar/baz), just append the rest (/bar/baz) to the workspace path on master It seems to me that the design of this feature is broken if you need to know about details like that, and an approximation and/or specifically adding support for job types is the best you can do.
          svvivek Vivekanand SV added a comment -

          What you could try is to determine the TopLevelItem ancestor of the current build's project, and get the workspace of that on the current node (e.g. /jenkins/workspace/foo). If that's a prefix of the current actual workspace (/jenkins/workspace/foo/bar/baz), just append the rest (/bar/baz) to the workspace path on master

          Yes, thats what I end up doing as stated in the comment before your reply above.

          svvivek Vivekanand SV added a comment - What you could try is to determine the TopLevelItem ancestor of the current build's project, and get the workspace of that on the current node (e.g. /jenkins/workspace/foo). If that's a prefix of the current actual workspace (/jenkins/workspace/foo/bar/baz), just append the rest (/bar/baz) to the workspace path on master Yes, thats what I end up doing as stated in the comment before your reply above.
          danielbeck Daniel Beck added a comment -

          Yes, thats what I end up doing as stated in the comment before your reply above.

          The approach I describe in that paragraph doesn't require any particular knowledge about matrix axes or matrix projects; it wasn't clear from your comment whether yours does.

          danielbeck Daniel Beck added a comment - Yes, thats what I end up doing as stated in the comment before your reply above. The approach I describe in that paragraph doesn't require any particular knowledge about matrix axes or matrix projects; it wasn't clear from your comment whether yours does.
          svvivek Vivekanand SV added a comment -

          Actually, yes. "machine/Lin_Test1/temp_axis/1" and "machine/Lin_Test1/temp_axis/2" parts of the path, from my comment, is generated after checking the prefix

          svvivek Vivekanand SV added a comment - Actually, yes. "machine/Lin_Test1/temp_axis/1" and "machine/Lin_Test1/temp_axis/2" parts of the path, from my comment, is generated after checking the prefix
          danielbeck Daniel Beck added a comment -

          Oh, in that case I just didn't understand what you wrote. So you can interpret my recommendation as agreement to what you wrote.

          danielbeck Daniel Beck added a comment - Oh, in that case I just didn't understand what you wrote. So you can interpret my recommendation as agreement to what you wrote.
          svvivek Vivekanand SV added a comment -

          Its okay, I wrote that way for the others to check the output path generated, I wanted to know if they agree to the paths that I am using now for different types of jobs.

          svvivek Vivekanand SV added a comment - Its okay, I wrote that way for the others to check the output path generated, I wanted to know if they agree to the paths that I am using now for different types of jobs.
          daj Dan Jarvis added a comment -

          Vivek, the target locations on the master that you described for the Multiconfiguration Project look good to me. I look forward to testing the new release of the plugin!

          daj Dan Jarvis added a comment - Vivek, the target locations on the master that you described for the Multiconfiguration Project look good to me. I look forward to testing the new release of the plugin!

          Code changed in jenkins
          User: Vivekanand S V
          Path:
          src/main/java/com/michelin/cio/hudson/plugins/copytoslave/CopyToSlaveUtils.java
          http://jenkins-ci.org/commit/copy-to-slave-plugin/6482c22c3a313c0ba8c83c553b1711a3d40c081e
          Log:
          [FIXED JENKINS-25346] - Addresses the workspace issue where files copied back to master are not in correct path.

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vivekanand S V Path: src/main/java/com/michelin/cio/hudson/plugins/copytoslave/CopyToSlaveUtils.java http://jenkins-ci.org/commit/copy-to-slave-plugin/6482c22c3a313c0ba8c83c553b1711a3d40c081e Log: [FIXED JENKINS-25346] - Addresses the workspace issue where files copied back to master are not in correct path.
          scm_issue_link SCM/JIRA link daemon made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]
          svvivek Vivekanand SV added a comment -

          @Dan, you can cancel the bounty, thats not needed to make me work, I was (and still am) busy with my personal life, that gives me no time to look here

          svvivek Vivekanand SV added a comment - @Dan, you can cancel the bounty, thats not needed to make me work, I was (and still am) busy with my personal life, that gives me no time to look here
          daj Dan Jarvis added a comment - - edited

          I think you deserve the bounty as you made the fix we asked for and given me a lot of assistance. I think you need to claim the bounty through the website and then I will gladly pay it. Thanks!

          daj Dan Jarvis added a comment - - edited I think you deserve the bounty as you made the fix we asked for and given me a lot of assistance. I think you need to claim the bounty through the website and then I will gladly pay it. Thanks!
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 159266 ] JNJira + In-Review [ 196036 ]

          People

            svvivek Vivekanand SV
            paleozogt Aaron Simmons
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: