I see that issue happening on long paths. A workaround was to use drive mapping in order to reduce the length but it doesn't solve the issue and we came to the limit of path shortening possibilities.

      Here's a stacktrace from a Matrix job:
      https://qa.nuxeo.org/jenkins/job/nuxeo-master-fullbuild-part2-distribution-multios/218/Slave=MULTIDB_WINDOWS

      08:59:26 Building remotely on tweedledum in workspace C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS
      08:59:26 Checkout:MULTIDB_WINDOWS / C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS - hudson.remoting.Channel@4a58a509:tweedledum
      08:59:26 Using strategy: Default
      08:59:26 Last Built Revision: Revision 3676821964588c85aa8b71e288c743c24edd00a3 (origin/master)
      08:59:27 Cloning the remote Git repository
      08:59:27 Cloning repository git://github.com/nuxeo/nuxeo-distribution.git
      08:59:27 git --version
      08:59:27 git version 1.7.6.msysgit.0
      08:59:27 ERROR: Failed to clean the workspace
      08:59:27 java.io.IOException: Unable to delete C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management - files in dir: [C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management\pom.properties, C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management\pom.xml]
      08:59:27 	at hudson.Util.deleteFile(Util.java:238)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:289)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.FilePath$11.invoke(FilePath.java:910)
      08:59:27 	at hudson.FilePath$11.invoke(FilePath.java:908)
      08:59:27 	at hudson.FilePath.act(FilePath.java:842)
      08:59:27 	at hudson.FilePath.act(FilePath.java:824)
      08:59:27 	at hudson.FilePath.deleteRecursive(FilePath.java:908)
      08:59:27 	at hudson.plugins.git.GitAPI.clone(GitAPI.java:239)
      08:59:27 	at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1040)
      08:59:27 	at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:982)
      08:59:27 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2200)
      08:59:27 	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      08:59:27 	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      08:59:27 	at hudson.remoting.Request$2.run(Request.java:326)
      08:59:27 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      08:59:27 	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      08:59:27 	at java.util.concurrent.FutureTask.run(Unknown Source)
      08:59:27 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      08:59:27 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      08:59:27 	at hudson.remoting.Engine$1$1.run(Engine.java:60)
      08:59:27 	at java.lang.Thread.run(Unknown Source)
      08:59:27 ERROR: Error cloning remote repo 'origin' : Failed to delete workspace
      08:59:27 hudson.plugins.git.GitException: Failed to delete workspace
      08:59:27 	at hudson.plugins.git.GitAPI.clone(GitAPI.java:242)
      08:59:27 	at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1040)
      08:59:27 	at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:982)
      08:59:27 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2200)
      08:59:27 	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      08:59:27 	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      08:59:27 	at hudson.remoting.Request$2.run(Request.java:326)
      08:59:27 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      08:59:27 	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      08:59:27 	at java.util.concurrent.FutureTask.run(Unknown Source)
      08:59:27 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      08:59:27 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      08:59:27 	at hudson.remoting.Engine$1$1.run(Engine.java:60)
      08:59:27 	at java.lang.Thread.run(Unknown Source)
      08:59:27 Caused by: java.io.IOException: Unable to delete C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management - files in dir: [C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management\pom.properties, C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management\pom.xml]
      08:59:27 	at hudson.Util.deleteFile(Util.java:238)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:289)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.FilePath$11.invoke(FilePath.java:910)
      08:59:27 	at hudson.FilePath$11.invoke(FilePath.java:908)
      08:59:27 	at hudson.FilePath.act(FilePath.java:842)
      08:59:27 	at hudson.FilePath.act(FilePath.java:824)
      08:59:27 	at hudson.FilePath.deleteRecursive(FilePath.java:908)
      08:59:27 	at hudson.plugins.git.GitAPI.clone(GitAPI.java:239)
      08:59:27 	... 13 more
      08:59:27 Trying next repository
      08:59:27 ERROR: Could not clone repository
      08:59:27 FATAL: Could not clone
      08:59:27 hudson.plugins.git.GitException: Could not clone
      08:59:27 	at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1052)
      08:59:27 	at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:982)
      08:59:27 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2200)
      08:59:27 	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      08:59:27 	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      08:59:27 	at hudson.remoting.Request$2.run(Request.java:326)
      08:59:27 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      08:59:27 	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      08:59:27 	at java.util.concurrent.FutureTask.run(Unknown Source)
      08:59:27 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      08:59:27 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      08:59:27 	at hudson.remoting.Engine$1$1.run(Engine.java:60)
      08:59:27 	at java.lang.Thread.run(Unknown Source)
      

      It is still possible to manually delete the files.

      FYI, we encountered such a seemingly similar issue in a Python script using Python internal delTree implementation, and it has been resolved by using Windows Shell command: "rmdir /s /q path/to/file" instead.

      Using a matrix job is increasing the path length.
      We would also need to use a different workspace path when under Windows.

          [JENKINS-15418] Fails to clean windows workspace with long path

          Julien Carsique created issue -

          Mark Waite added a comment -

          We've seen a general problem on Windows file systems with paths which exceed about 250 characters. The problems were visible in our MSBuild scripts and affected other programs as well. We finally decided that it was not worth fighting what appeared to be Microsoft file system limitations, so we forced ourselves to use shorter paths. The sample path in your output is longer than 250 characters, so you may be encountering the same problem we encountered.

          You might find Jeff Attwood's comments on the topic interesting.

          http://www.codinghorror.com/blog/2006/11/filesystem-paths-how-long-is-too-long.html

          Mark Waite added a comment - We've seen a general problem on Windows file systems with paths which exceed about 250 characters. The problems were visible in our MSBuild scripts and affected other programs as well. We finally decided that it was not worth fighting what appeared to be Microsoft file system limitations, so we forced ourselves to use shorter paths. The sample path in your output is longer than 250 characters, so you may be encountering the same problem we encountered. You might find Jeff Attwood's comments on the topic interesting. http://www.codinghorror.com/blog/2006/11/filesystem-paths-how-long-is-too-long.html
          Julien Carsique made changes -
          Labels Original: git matrix path windows New: matrix path windows

          Hi,

          Thanks for that interesting link.
          You're right and we already worked on shortening our paths. We also wrote a few scripts which automatically map the current path on a new drive letter in order to reduce the risk to encounter that ridiculous Windows issue. But anything we do is only delaying the issue. And it came back with that job.

          The point here is:

          • the job is a "matrix" job so the working dir path is longer (C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS). Reducing too much the job name is an issue when you have 400 jobs.
          • in the currently failing path, the only part we could still reduce is not very long: nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss.

          The deletion fails in Java File.delete() so there's no quick solution for that issue. But we could imagine some workarounds to reduce the times it's happening:

          • Jenkins could mount the working directory on a new drive letter,
          • Jenkins could propose to use a specific short path when on Windows (ie c:\ws\buildNumber or c:\ws\jobName\buildNumber),
          • when failing to delete a file on Windows, the code could try to fallback on calling a delete windows command (rmdir /s /q path/to/file),
          • ...
            I can't see a "clean" solution but anything that could help to shorten the path when working on windows would help...

          Julien Carsique added a comment - Hi, Thanks for that interesting link. You're right and we already worked on shortening our paths. We also wrote a few scripts which automatically map the current path on a new drive letter in order to reduce the risk to encounter that ridiculous Windows issue. But anything we do is only delaying the issue. And it came back with that job. The point here is: the job is a "matrix" job so the working dir path is longer ( C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS ). Reducing too much the job name is an issue when you have 400 jobs. in the currently failing path, the only part we could still reduce is not very long: nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss . The deletion fails in Java File.delete() so there's no quick solution for that issue. But we could imagine some workarounds to reduce the times it's happening: Jenkins could mount the working directory on a new drive letter, Jenkins could propose to use a specific short path when on Windows (ie c:\ws\buildNumber or c:\ws\jobName\buildNumber ), when failing to delete a file on Windows, the code could try to fallback on calling a delete windows command ( rmdir /s /q path/to/file ), ... I can't see a "clean" solution but anything that could help to shorten the path when working on windows would help...
          Julien Carsique made changes -
          Component/s New: core [ 15593 ]
          Component/s Original: git [ 15543 ]
          URL Original: https://qa.nuxeo.org/jenkins/job/nuxeo-master-fullbuild-part2-distribution-multios
          Description Original: I see that issue happening on long paths. A workaround was to use drive mapping in order to reduce the length but it doesn't solve the issue and we came to the limit of path shortening possibilities.

          Here's a stacktrace from a Matrix job:
          https://qa.nuxeo.org/jenkins/job/nuxeo-master-fullbuild-part2-distribution-multios/218/Slave=MULTIDB_WINDOWS
          {code}08:59:26 Building remotely on tweedledum in workspace C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS
          08:59:26 Checkout:MULTIDB_WINDOWS / C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS - hudson.remoting.Channel@4a58a509:tweedledum
          08:59:26 Using strategy: Default
          08:59:26 Last Built Revision: Revision 3676821964588c85aa8b71e288c743c24edd00a3 (origin/master)
          08:59:27 Cloning the remote Git repository
          08:59:27 Cloning repository git://github.com/nuxeo/nuxeo-distribution.git
          08:59:27 git --version
          08:59:27 git version 1.7.6.msysgit.0
          08:59:27 ERROR: Failed to clean the workspace
          08:59:27 java.io.IOException: Unable to delete C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management - files in dir: [C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management\pom.properties, C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management\pom.xml]
          08:59:27 at hudson.Util.deleteFile(Util.java:238)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:289)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.FilePath$11.invoke(FilePath.java:910)
          08:59:27 at hudson.FilePath$11.invoke(FilePath.java:908)
          08:59:27 at hudson.FilePath.act(FilePath.java:842)
          08:59:27 at hudson.FilePath.act(FilePath.java:824)
          08:59:27 at hudson.FilePath.deleteRecursive(FilePath.java:908)
          08:59:27 at hudson.plugins.git.GitAPI.clone(GitAPI.java:239)
          08:59:27 at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1040)
          08:59:27 at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:982)
          08:59:27 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2200)
          08:59:27 at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          08:59:27 at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          08:59:27 at hudson.remoting.Request$2.run(Request.java:326)
          08:59:27 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          08:59:27 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          08:59:27 at java.util.concurrent.FutureTask.run(Unknown Source)
          08:59:27 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
          08:59:27 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          08:59:27 at hudson.remoting.Engine$1$1.run(Engine.java:60)
          08:59:27 at java.lang.Thread.run(Unknown Source)
          08:59:27 ERROR: Error cloning remote repo 'origin' : Failed to delete workspace
          08:59:27 hudson.plugins.git.GitException: Failed to delete workspace
          08:59:27 at hudson.plugins.git.GitAPI.clone(GitAPI.java:242)
          08:59:27 at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1040)
          08:59:27 at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:982)
          08:59:27 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2200)
          08:59:27 at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          08:59:27 at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          08:59:27 at hudson.remoting.Request$2.run(Request.java:326)
          08:59:27 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          08:59:27 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          08:59:27 at java.util.concurrent.FutureTask.run(Unknown Source)
          08:59:27 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
          08:59:27 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          08:59:27 at hudson.remoting.Engine$1$1.run(Engine.java:60)
          08:59:27 at java.lang.Thread.run(Unknown Source)
          08:59:27 Caused by: java.io.IOException: Unable to delete C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management - files in dir: [C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management\pom.properties, C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management\pom.xml]
          08:59:27 at hudson.Util.deleteFile(Util.java:238)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:289)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.FilePath$11.invoke(FilePath.java:910)
          08:59:27 at hudson.FilePath$11.invoke(FilePath.java:908)
          08:59:27 at hudson.FilePath.act(FilePath.java:842)
          08:59:27 at hudson.FilePath.act(FilePath.java:824)
          08:59:27 at hudson.FilePath.deleteRecursive(FilePath.java:908)
          08:59:27 at hudson.plugins.git.GitAPI.clone(GitAPI.java:239)
          08:59:27 ... 13 more
          08:59:27 Trying next repository
          08:59:27 ERROR: Could not clone repository
          08:59:27 FATAL: Could not clone
          08:59:27 hudson.plugins.git.GitException: Could not clone
          08:59:27 at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1052)
          08:59:27 at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:982)
          08:59:27 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2200)
          08:59:27 at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          08:59:27 at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          08:59:27 at hudson.remoting.Request$2.run(Request.java:326)
          08:59:27 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          08:59:27 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          08:59:27 at java.util.concurrent.FutureTask.run(Unknown Source)
          08:59:27 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
          08:59:27 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          08:59:27 at hudson.remoting.Engine$1$1.run(Engine.java:60)
          08:59:27 at java.lang.Thread.run(Unknown Source)
          {code}

          It is still possible to manually delete the files.

          FYI, we encountered such a seemingly similar issue in a Python script using Python internal delTree implementation, and it has been resolved by using Windows Shell command: "rmdir /s /q path/to/file" instead.

          Also, find attached the job's config.xml if it can help to look at the parameters.
          New: I see that issue happening on long paths. A workaround was to use drive mapping in order to reduce the length but it doesn't solve the issue and we came to the limit of path shortening possibilities.

          Here's a stacktrace from a Matrix job:
          https://qa.nuxeo.org/jenkins/job/nuxeo-master-fullbuild-part2-distribution-multios/218/Slave=MULTIDB_WINDOWS
          {code}08:59:26 Building remotely on tweedledum in workspace C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS
          08:59:26 Checkout:MULTIDB_WINDOWS / C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS - hudson.remoting.Channel@4a58a509:tweedledum
          08:59:26 Using strategy: Default
          08:59:26 Last Built Revision: Revision 3676821964588c85aa8b71e288c743c24edd00a3 (origin/master)
          08:59:27 Cloning the remote Git repository
          08:59:27 Cloning repository git://github.com/nuxeo/nuxeo-distribution.git
          08:59:27 git --version
          08:59:27 git version 1.7.6.msysgit.0
          08:59:27 ERROR: Failed to clean the workspace
          08:59:27 java.io.IOException: Unable to delete C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management - files in dir: [C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management\pom.properties, C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management\pom.xml]
          08:59:27 at hudson.Util.deleteFile(Util.java:238)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:289)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.FilePath$11.invoke(FilePath.java:910)
          08:59:27 at hudson.FilePath$11.invoke(FilePath.java:908)
          08:59:27 at hudson.FilePath.act(FilePath.java:842)
          08:59:27 at hudson.FilePath.act(FilePath.java:824)
          08:59:27 at hudson.FilePath.deleteRecursive(FilePath.java:908)
          08:59:27 at hudson.plugins.git.GitAPI.clone(GitAPI.java:239)
          08:59:27 at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1040)
          08:59:27 at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:982)
          08:59:27 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2200)
          08:59:27 at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          08:59:27 at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          08:59:27 at hudson.remoting.Request$2.run(Request.java:326)
          08:59:27 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          08:59:27 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          08:59:27 at java.util.concurrent.FutureTask.run(Unknown Source)
          08:59:27 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
          08:59:27 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          08:59:27 at hudson.remoting.Engine$1$1.run(Engine.java:60)
          08:59:27 at java.lang.Thread.run(Unknown Source)
          08:59:27 ERROR: Error cloning remote repo 'origin' : Failed to delete workspace
          08:59:27 hudson.plugins.git.GitException: Failed to delete workspace
          08:59:27 at hudson.plugins.git.GitAPI.clone(GitAPI.java:242)
          08:59:27 at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1040)
          08:59:27 at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:982)
          08:59:27 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2200)
          08:59:27 at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          08:59:27 at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          08:59:27 at hudson.remoting.Request$2.run(Request.java:326)
          08:59:27 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          08:59:27 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          08:59:27 at java.util.concurrent.FutureTask.run(Unknown Source)
          08:59:27 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
          08:59:27 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          08:59:27 at hudson.remoting.Engine$1$1.run(Engine.java:60)
          08:59:27 at java.lang.Thread.run(Unknown Source)
          08:59:27 Caused by: java.io.IOException: Unable to delete C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management - files in dir: [C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management\pom.properties, C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management\pom.xml]
          08:59:27 at hudson.Util.deleteFile(Util.java:238)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:289)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.Util.deleteContentsRecursive(Util.java:200)
          08:59:27 at hudson.Util.deleteRecursive(Util.java:280)
          08:59:27 at hudson.FilePath$11.invoke(FilePath.java:910)
          08:59:27 at hudson.FilePath$11.invoke(FilePath.java:908)
          08:59:27 at hudson.FilePath.act(FilePath.java:842)
          08:59:27 at hudson.FilePath.act(FilePath.java:824)
          08:59:27 at hudson.FilePath.deleteRecursive(FilePath.java:908)
          08:59:27 at hudson.plugins.git.GitAPI.clone(GitAPI.java:239)
          08:59:27 ... 13 more
          08:59:27 Trying next repository
          08:59:27 ERROR: Could not clone repository
          08:59:27 FATAL: Could not clone
          08:59:27 hudson.plugins.git.GitException: Could not clone
          08:59:27 at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1052)
          08:59:27 at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:982)
          08:59:27 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2200)
          08:59:27 at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          08:59:27 at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          08:59:27 at hudson.remoting.Request$2.run(Request.java:326)
          08:59:27 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          08:59:27 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          08:59:27 at java.util.concurrent.FutureTask.run(Unknown Source)
          08:59:27 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
          08:59:27 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          08:59:27 at hudson.remoting.Engine$1$1.run(Engine.java:60)
          08:59:27 at java.lang.Thread.run(Unknown Source)
          {code}

          It is still possible to manually delete the files.

          FYI, we encountered such a seemingly similar issue in a Python script using Python internal delTree implementation, and it has been resolved by using Windows Shell command: "rmdir /s /q path/to/file" instead.

          Using a matrix job is increasing the path length.
          We would also need to use a different workspace path when under Windows.
          Priority Original: Major [ 3 ] New: Minor [ 4 ]
          Summary Original: Git fails to clean windows workspace with long path New: Fails to clean windows workspace with long path

          onemanbucket added a comment -

          We're also running into the windows MAX_PATH problem.

          I tried calling File.delete with the unicode prefix (\\?\C:\...) but it didn't work.

          onemanbucket added a comment - We're also running into the windows MAX_PATH problem. I tried calling File.delete with the unicode prefix (\\?\C:\...) but it didn't work.

          onemanbucket added a comment -

          I have a fix for the issue, but it needs review. The two main issues are:
          1) safe guard that the pivot point in the path is in the workspace so we don't delete too much
          2) should the code go in deleteRecursive

          What it does is move the subtree close to the MAX_PATH limit to a temp directory and try to delete from there.

          Commit at: https://github.com/onemanbucket/jenkins/commit/0ab18187eb60dd89cf5759eb231c326cb31c866b

          onemanbucket added a comment - I have a fix for the issue, but it needs review. The two main issues are: 1) safe guard that the pivot point in the path is in the workspace so we don't delete too much 2) should the code go in deleteRecursive What it does is move the subtree close to the MAX_PATH limit to a temp directory and try to delete from there. Commit at: https://github.com/onemanbucket/jenkins/commit/0ab18187eb60dd89cf5759eb231c326cb31c866b

          Julien Carsique added a comment - Interesting solution. I commented your commit with my tests results: https://github.com/onemanbucket/jenkins/commit/0ab18187eb60dd89cf5759eb231c326cb31c866b#commitcomment-2033773

          onemanbucket added a comment -

          Hi, I've done some more research now, and it turns out the jre actually supports using unicode prefixes, and all win api calls are correct. The problem is in hudsons Kernel32Utils that doesn't call GetFileAttributesW correctly (the path should be prefixed as far as I can tell). This makes deleteRecursive() mistakenly think that the first directory after MAX_PATH is a file, and since delete only works on empty directories the subsequent delete call fails.

          I'll post a test case and a patch in a bit.

          onemanbucket added a comment - Hi, I've done some more research now, and it turns out the jre actually supports using unicode prefixes, and all win api calls are correct. The problem is in hudsons Kernel32Utils that doesn't call GetFileAttributesW correctly (the path should be prefixed as far as I can tell). This makes deleteRecursive() mistakenly think that the first directory after MAX_PATH is a file, and since delete only works on empty directories the subsequent delete call fails. I'll post a test case and a patch in a bit.

          onemanbucket added a comment -

          onemanbucket added a comment - https://github.com/jenkinsci/jenkins/pull/593

            ndeloof Nicolas De Loof
            jcarsique Julien Carsique
            Votes:
            1 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: