Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-57955

Clean and sync incorrectly follows junction/directory symlinks on Windows

    • Icon: Bug Bug
    • Resolution: Won't Fix
    • Icon: Major Major
    • p4-plugin
    • Windows Host and Agent, Jenkins ver. 2.176.1 (Pipeline in SCM)
      P4-Plugin 1.10.0

      I sync my client workspace in three environments Windows, Mac and Linux

      My step after syncing the workspace is to set up a symlink in each environment that points to a local third party dependencies folder used by the build.

      I'm not wishing to add my third party dependencies folder to source control as the folder is about 18GB and would vastly increase the sync time and unnecessary as this folder doesn't change.

       

      After adding the symlink and running the build, on the second run... the p4 plugin issues a force clean workspace before it syncs the workspace again.

       

      On Mac and Linux, the third party symlink is deleted and recreated each time...

      However on windows, instead of removing the link, it follows in to the link and deletes the contents of the dependencies folder, when all I am looking to to is unlink it.

       

      On windows the command I am using to create the link is:

      mklink /J "%WORKSPACE%\3rdparty" "C:\Jenkins\3rdparty"

       

      And on Mac and Linux its:

      ln -s /Jenkins/workspace/ 3rdparty

       

      I notice that the cleanup command is an "rm -rf" which I am guessing must be handled differently on the windows agent.

       

      it must be the recursive flag that's entering the folder.

      maybe whats needed is the ability to run a custom removal command. (This will also fix another issue with the fact that the Mac needs to run the rm command with sudo if not running as root user, even if it has permissions to delete and modify the workspace folder, however the plugin only issues "rm -rf" not "sudo rm -rf". which took me a while to convince my IT department to enable the root user on the mac)

       

      Thanks a lot as this one is a massive pain for me and can't seem to be able to work around it.

       

       

       

          [JENKINS-57955] Clean and sync incorrectly follows junction/directory symlinks on Windows

          Karl Wirth added a comment -

          Hi kierzo - Thanks for reporting this. I'll check to see if creating a link instead of a junction changes the behavior.

          Do you need the force clean workspace? If not then the workaround may be to use a different sync option such as 'Sync Only':

               https://www.perforce.com/manuals/jenkins/Content/P4Jenkins/populate-sync-only.html?Highlight=%22Sync%20Only%22

          Karl Wirth added a comment - Hi kierzo - Thanks for reporting this. I'll check to see if creating a link instead of a junction changes the behavior. Do you need the force clean workspace? If not then the workaround may be to use a different sync option such as 'Sync Only':      https://www.perforce.com/manuals/jenkins/Content/P4Jenkins/populate-sync-only.html?Highlight=%22Sync%20Only%22

          Karl Wirth added a comment -

          Hi kierzo, I have been able to reproduce this behavior. Changing it to 'mklink /D' does not solve the problem.

           

          If it works for you, using 'sync only' is one workaround. Would another be to run 'rmdir' on the junction before the end of the job then recreate it next time?

           

          Reproduction steps:

          1. Create a file on the build machine called 'C:\filestore\Jenkins\temp\symlink_target1\file1'.

          2. Add a file to '//depot/win_symlink/...'.

          3. Create a pipeline job with the following config:

          pipeline {
            agent { label 'Windows10' }
            stages {
              stage("Repro") {
                steps {
                  script {
                      checkout perforce(credential: 'JenkinsMaster', populate: forceClean(have: false, parallel: [enable: false, minbytes: '1024', minfiles: '1', threads: '4'], pin: '', quiet: false), workspace: manualSpec(charset: 'none', cleanup: false, name: 'jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}', pinHost: false, spec: clientSpec(allwrite: false, backup: true, changeView: '', clobber: true, compress: false, line: 'LOCAL', locked: false, modtime: false, rmdir: false, serverID: '', streamName: '', type: 'WRITABLE', view: '//depot/win_symlink/... 
                      bat 'dir /b /s'
                      bat 'mklink /J "linkeddir" "C:\\filestore\\Jenkins\\temp\\symlink_target1"'
                      bat 'dir /b /s'
                  }
                }
              }
            }
          }
          

          4. Build the job.

          5. Build the job again.

          Result seen - 'C:\filestore\Jenkins\temp\symlink_target1\file1' is deleted.

          Expected result - 'C:\filestore\Jenkins\temp\symlink_target1\file1' is not deleted.

          Karl Wirth added a comment - Hi kierzo , I have been able to reproduce this behavior. Changing it to 'mklink /D' does not solve the problem.   If it works for you, using 'sync only' is one workaround. Would another be to run 'rmdir' on the junction before the end of the job then recreate it next time?   Reproduction steps: 1. Create a file on the build machine called 'C:\filestore\Jenkins\temp\symlink_target1\file1'. 2. Add a file to '//depot/win_symlink/...'. 3. Create a pipeline job with the following config: pipeline { agent { label 'Windows10' } stages { stage( "Repro" ) { steps { script { checkout perforce(credential: 'JenkinsMaster' , populate: forceClean(have: false , parallel: [enable: false , minbytes: '1024' , minfiles: '1' , threads: '4' ], pin: '', quiet: false ), workspace: manualSpec(charset: ' none ', cleanup: false , name: ' jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER} ', pinHost: false , spec: clientSpec(allwrite: false , backup: true , changeView: ' ', clobber: true , compress: false , line: ' LOCAL ', locked: false , modtime: false , rmdir: false , serverID: ' ', streamName: ' ', type: ' WRITABLE ', view: ' //depot/win_symlink/... bat 'dir /b /s' bat 'mklink /J "linkeddir" "C:\\filestore\\Jenkins\\temp\\symlink_target1" ' bat 'dir /b /s' } } } } } 4. Build the job. 5. Build the job again. Result seen - 'C:\filestore\Jenkins\temp\symlink_target1\file1' is deleted. Expected result - 'C:\filestore\Jenkins\temp\symlink_target1\file1' is not deleted.

          Kieron Kierzo added a comment -

          Hi Karl,

          Thanks for your detailed response.

          I have added it to a always run in a post step, the issue is that its not always reliable to run.

          e.g. I've had a power-cut , had java run out of heap space, network disconnects, switching from jnpl3 to jnpl4, plugins doing odd stuff and users making breaking changes to the pipeline script affecting it getting to the end.

          Its not common but not as uncommon as id like, but it only needs to happen once and the build misses getting to the end and then the next run removes the build dependencies.

           

          Thanks a lot for your time.

          Kieron Kierzo added a comment - Hi Karl, Thanks for your detailed response. I have added it to a always run in a post step, the issue is that its not always reliable to run. e.g. I've had a power-cut , had java run out of heap space, network disconnects, switching from jnpl3 to jnpl4, plugins doing odd stuff and users making breaking changes to the pipeline script affecting it getting to the end. Its not common but not as uncommon as id like, but it only needs to happen once and the build misses getting to the end and then the next run removes the build dependencies.   Thanks a lot for your time.

          Karl Wirth added a comment -

          Hi kierzo, Thanks and I'm happy to help. I'll make sure the developers see the feedback about the the workaround not being robust.

          Karl Wirth added a comment - Hi kierzo , Thanks and I'm happy to help. I'll make sure the developers see the feedback about the the workaround not being robust.

          Paul Allen added a comment -

          Released in 1.10.2

          Paul Allen added a comment - Released in 1.10.2

          Kieron Kierzo added a comment -

          This still is not fixed unfortunately p4karl i'm on version 1.10.3

          Kieron Kierzo added a comment - This still is not fixed unfortunately p4karl i'm on version 1.10.3

          Paul Allen added a comment -

          Possible issue when cleaning up symbolic links on Windows. 

          Paul Allen added a comment - Possible issue when cleaning up symbolic links on Windows. 

          Paul Allen added a comment -

          P4Java's reconcile  supports 'symlinks' not 'junctions'.  Please use:

          mklink /D "%WORKSPACE%\3rdparty" "C:\Jenkins\3rdparty" 

          'D' for a Directory Symlink.

          Paul Allen added a comment - P4Java's reconcile  supports 'symlinks' not 'junctions'.  Please use: mklink /D "%WORKSPACE%\3rdparty" "C:\Jenkins\3rdparty" 'D' for a Directory Symlink.

          Paul Allen added a comment -

          Closing - will not support 'junctions'.

          Paul Allen added a comment - Closing - will not support 'junctions'.

            msmeeth Matthew Smeeth
            kierzo Kieron Kierzo
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: