Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21369

Git client fails to remove files/directories with "git clean -fdx"

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • git-client-plugin
    • None
    • OS: Windows 2008 R2
      Java: 1.6.0_26
      Jenkins: 1.547
      Git client: Newest (1.6.1)
      Other plugins: All are updated, like Git (2.0.1)

      Jenkins is running as Windows service, with Local system account

      This all has worked fine previously, but latest updates for some reason there have been these kind of issues, as described below.

      1. Start Jenkins service
      2. Trigger ex. 20 builds
      3. Wait and see if one of them fails to build
      4. See if the reason is Jenkins issue, not msbuild/unit test
      5. Rest of builds that starts after failed one, will fail also with same output as below
      6. Wipe workspaces manually
      7. Builds will most likely start working again

      Triggered by Gerrit: http://xxx002:8080/4195
      [EnvInject] - Loading node environment variables.
      Building on master in workspace r:\jenkins-workspaces
      Fetching changes from the remote Git repository
      Fetching upstream changes from ssh://hudson@xxxx002:29418/xxx.git
      Checking out Revision 45d62ed078309607e0b6a24b842ff33c93f3b740 (feature/WI3076)
      Cleaning workspace
      Resetting working tree
      FATAL: Command "git clean -fdx" returned status code 1:
      stdout: Removing Source/xxx/zzz/Areas
      Removing Source/xxx/zzz/Content
      Removing Source/xxx/zzz/Scripts
      Removing Source/xxx/zzz/Views

      stderr: warning: failed to remove Source/xxx/zzz/Areas
      warning: failed to remove Source/xxx/zzz/Content
      warning: failed to remove Source/xxx/zzz/Scripts
      warning: failed to remove Source/xxx/zzz/Views

      hudson.plugins.git.GitException: Command "git clean -fdx" returned status code 1:
      stdout: Removing Source/xxx/zzz/Areas
      Removing Source/xxx/zzz/Content
      Removing Source/xxx/zzz/Scripts
      Removing Source/xxx/zzz/Views

      stderr: warning: failed to remove Source/xxx/zzz/Areas
      warning: failed to remove Source/xxx/zzz/Content
      warning: failed to remove Source/xxx/zzz/Scripts
      warning: failed to remove Source/xxx/zzz/Views

      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1086)
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1063)
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:900)
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:910)
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.clean(CliGitAPIImpl.java:367)
      at hudson.plugins.git.GitAPI.clean(GitAPI.java:251)
      at hudson.plugins.git.extensions.impl.CleanCheckout.onCheckoutCompleted(CleanCheckout.java:28)
      at hudson.plugins.git.GitSCM.checkout(GitSCM.java:890)
      at hudson.model.AbstractProject.checkout(AbstractProject.java:1415)
      at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:652)
      at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
      at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:561)
      at hudson.model.Run.execute(Run.java:1678)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:231)

          [JENKINS-21369] Git client fails to remove files/directories with "git clean -fdx"

          Mark Waite added a comment -

          Can you provide more details that would make this a repeatable bug?

          The stack trace shows you are using the command line git implementation. That implementation has no known problems with leaving files open. If you can provide a repeatable case, it has a much better chance of being investigated.

          The JGit implementation has known problems with leaving files open (see JENKINS-19994).

          There is also a risk that some other program is keeping the files open. There were cases in the past where a virus scanner prevented a directory from being deleted because it was keeping one or more files open from the directory.

          Mark Waite added a comment - Can you provide more details that would make this a repeatable bug? The stack trace shows you are using the command line git implementation. That implementation has no known problems with leaving files open. If you can provide a repeatable case, it has a much better chance of being investigated. The JGit implementation has known problems with leaving files open (see JENKINS-19994 ). There is also a risk that some other program is keeping the files open. There were cases in the past where a virus scanner prevented a directory from being deleted because it was keeping one or more files open from the directory.

          Mark Waite added a comment -

          Closing as "Cannot Reproduce" after a month of waiting for a response from the original submitter.

          Mark Waite added a comment - Closing as "Cannot Reproduce" after a month of waiting for a response from the original submitter.

          Thomas COAN added a comment -

          Hi,
          I face same issue :
          It was working really until yesterday on a specific job (no changes on jenkins configuration for several weeks)

          remark :
          Other jobs are still working for the moment (with the same command run on their own git repo).

          =====================
          Error with StackTrace
          =====================
          Manually triggered by user anonymous for Gerrit: https://MY_GIT:8080/607665
          Building in workspace <http://MY_JENKINS:8080/jenkins/job/AR_VA_GerritVoter/ws/>
          > git.exe rev-parse --is-inside-work-tree
          Fetching changes from the remote Git repository
          > git.exe config remote.origin.url ssh://MY_GIT:29418/sandbox/MY.git
          Fetching upstream changes from ssh://MY_GIT:29418/sandbox/MY.git
          > git.exe --version
          > git.exe fetch --tags --progress ssh://MY_GIT:29418/sandbox/MY.git refs/changes/65/607665/1
          > git.exe rev-parse "c68be008cc694ffc7fa87e28baaf86245b696c1a^

          {commit}"
          Checking out Revision c68be008cc694ffc7fa87e28baaf86245b696c1a (master)
          > git.exe config core.sparsecheckout
          > git.exe checkout -f c68be008cc694ffc7fa87e28baaf86245b696c1a
          > git.exe rev-parse "FETCH_HEAD^{commit}

          "
          > git.exe rev-list 445b38d8c3bbf8d5ef30f811b2c148fb356308ce
          > git.exe tag -a -f -m Jenkins Build #359 jenkins-AR_VA_GerritVoter-359
          Cleaning workspace
          > git.exe rev-parse --verify HEAD
          Resetting working tree
          > git.exe reset --hard
          > git.exe clean -fdx
          FATAL: Command "git.exe clean -fdx" returned status code 1:
          stdout: Removing com.sap.hilo.desktop.overlay/target/

          stderr: warning: failed to remove com.sap.hilo.desktop.overlay/target/

          hudson.plugins.git.GitException: Command "git.exe clean -fdx" returned status code 1:
          stdout: Removing com.sap.hilo.desktop.overlay/target/

          stderr: warning: failed to remove com.sap.hilo.desktop.overlay/target/

          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1406)
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1382)
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1378)
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1092)
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1102)
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.clean(CliGitAPIImpl.java:492)
          at hudson.plugins.git.GitAPI.clean(GitAPI.java:253)
          at hudson.plugins.git.extensions.impl.CleanCheckout.onCheckoutCompleted(CleanCheckout.java:28)
          at hudson.plugins.git.GitSCM.checkout(GitSCM.java:908)
          at hudson.model.AbstractProject.checkout(AbstractProject.java:1254)
          at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:624)
          at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:530)
          at hudson.model.Run.execute(Run.java:1732)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          at hudson.model.ResourceController.execute(ResourceController.java:88)
          at hudson.model.Executor.run(Executor.java:234)

          =====================
          === manual Check ===
          =====================
          I have tryed to run the git command manually, and it seems to be the same problem :
          C:\.jenkins\jobs\AR_VA_GerritVoter\workspace\gitRepo>git clean -fdx
          Removing com.sap.hilo.desktop.overlay/target/
          warning: failed to remove com.sap.hilo.desktop.overlay/target/

          Note that /target folder is in .gitignore (for more than 1 month)

          Thomas COAN added a comment - Hi, I face same issue : It was working really until yesterday on a specific job (no changes on jenkins configuration for several weeks) remark : Other jobs are still working for the moment (with the same command run on their own git repo). ===================== Error with StackTrace ===================== Manually triggered by user anonymous for Gerrit: https://MY_GIT:8080/607665 Building in workspace < http://MY_JENKINS:8080/jenkins/job/AR_VA_GerritVoter/ws/ > > git.exe rev-parse --is-inside-work-tree Fetching changes from the remote Git repository > git.exe config remote.origin.url ssh://MY_GIT:29418/sandbox/MY.git Fetching upstream changes from ssh://MY_GIT:29418/sandbox/MY.git > git.exe --version > git.exe fetch --tags --progress ssh://MY_GIT:29418/sandbox/MY.git refs/changes/65/607665/1 > git.exe rev-parse "c68be008cc694ffc7fa87e28baaf86245b696c1a^ {commit}" Checking out Revision c68be008cc694ffc7fa87e28baaf86245b696c1a (master) > git.exe config core.sparsecheckout > git.exe checkout -f c68be008cc694ffc7fa87e28baaf86245b696c1a > git.exe rev-parse "FETCH_HEAD^{commit} " > git.exe rev-list 445b38d8c3bbf8d5ef30f811b2c148fb356308ce > git.exe tag -a -f -m Jenkins Build #359 jenkins-AR_VA_GerritVoter-359 Cleaning workspace > git.exe rev-parse --verify HEAD Resetting working tree > git.exe reset --hard > git.exe clean -fdx FATAL: Command "git.exe clean -fdx" returned status code 1: stdout: Removing com.sap.hilo.desktop.overlay/target/ stderr: warning: failed to remove com.sap.hilo.desktop.overlay/target/ hudson.plugins.git.GitException: Command "git.exe clean -fdx" returned status code 1: stdout: Removing com.sap.hilo.desktop.overlay/target/ stderr: warning: failed to remove com.sap.hilo.desktop.overlay/target/ at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1406) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1382) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1378) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1092) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1102) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.clean(CliGitAPIImpl.java:492) at hudson.plugins.git.GitAPI.clean(GitAPI.java:253) at hudson.plugins.git.extensions.impl.CleanCheckout.onCheckoutCompleted(CleanCheckout.java:28) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:908) at hudson.model.AbstractProject.checkout(AbstractProject.java:1254) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:624) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:530) at hudson.model.Run.execute(Run.java:1732) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:234) ===================== === manual Check === ===================== I have tryed to run the git command manually, and it seems to be the same problem : C:\.jenkins\jobs\AR_VA_GerritVoter\workspace\gitRepo>git clean -fdx Removing com.sap.hilo.desktop.overlay/target/ warning: failed to remove com.sap.hilo.desktop.overlay/target/ Note that /target folder is in .gitignore (for more than 1 month)

          Mark Waite added a comment -

          Since your problem directory is the "target" directory, I suspect that one or more programs are holding that directory, or a file in that directory open. Windows will not allow a file to be removed which is open by any process. Windows will not allow a directory to be removed which includes any open file.

          One simple technique to test the theory is to stop the slave agent and restart it.

          Another simple technique is to restart the slave computer.

          The Windows file removal poliicy is much more challenging than the Unix file removal policy, so another alternative is to prefer Unix based build machines instead of Windows machines.

          Mark Waite added a comment - Since your problem directory is the "target" directory, I suspect that one or more programs are holding that directory, or a file in that directory open. Windows will not allow a file to be removed which is open by any process. Windows will not allow a directory to be removed which includes any open file. One simple technique to test the theory is to stop the slave agent and restart it. Another simple technique is to restart the slave computer. The Windows file removal poliicy is much more challenging than the Unix file removal policy, so another alternative is to prefer Unix based build machines instead of Windows machines.

          Thomas COAN added a comment -

          Thanks Mark,
          I have tryed to run the job after a reboot => no old process holding the folder => same problem !
          in batch mode (del /F/Q [...]\target , it fails => no error, but no delete!)

          However I have investigated on the commit that generates the problem, and I have found a new file that generates a path greater than 255 characters in my target folder.
          Could it be the root cause ?

          I will check with the dev if he can change it and keep you in touch

          I cannot switch on a unix machine (not owned by myself !)

          Thomas COAN added a comment - Thanks Mark, I have tryed to run the job after a reboot => no old process holding the folder => same problem ! in batch mode (del /F/Q [...] \target , it fails => no error, but no delete!) However I have investigated on the commit that generates the problem, and I have found a new file that generates a path greater than 255 characters in my target folder. Could it be the root cause ? I will check with the dev if he can change it and keep you in touch I cannot switch on a unix machine (not owned by myself !)

          Daniel Beck added a comment -

          Issue report needs to show that this is a problem with Jenkins. The Sysinternals tools (e.g. Process Explorer) could help determining which process is holding a file handle.

          Daniel Beck added a comment - Issue report needs to show that this is a problem with Jenkins. The Sysinternals tools (e.g. Process Explorer) could help determining which process is holding a file handle.

          Daniel Beck added a comment -

          FWIW I've repeatedly had to remind my users to log out of the build server instead of just closing the RDC session – make sure there are no sessions with e.g. a cmd.exe in that directory open.

          Daniel Beck added a comment - FWIW I've repeatedly had to remind my users to log out of the build server instead of just closing the RDC session – make sure there are no sessions with e.g. a cmd.exe in that directory open.

          Daniel Beck added a comment -

          Sorry, didn't see the comment posted just before mine. It's likely that the long path is the problem in which case this is Not a Defect in Jenkins.

          A few suggestions if you're right around ~260 chars and your developer is being stubborn:

          • Use a short Jenkins (master/slave) root directory, e.g. D:\J
          • Change the Workspace Root Directory on the master completely (Configure System; Advanced), e.g. D:\WS if building on the master
          • Set the hudson.model.Slave.workspaceRoot system property to something shorter than workspace if building on a slave
          • Set the hudson.matrix.MatrixConfiguration.useShortWorkspaceName system property to true for any Multi-configuration projects
          • Configure a custom workspace in Advanced Project Options

          Daniel Beck added a comment - Sorry, didn't see the comment posted just before mine. It's likely that the long path is the problem in which case this is Not a Defect in Jenkins. A few suggestions if you're right around ~260 chars and your developer is being stubborn: Use a short Jenkins (master/slave) root directory, e.g. D:\J Change the Workspace Root Directory on the master completely (Configure System; Advanced), e.g. D:\WS if building on the master Set the hudson.model.Slave.workspaceRoot system property to something shorter than workspace if building on a slave Set the hudson.matrix.MatrixConfiguration.useShortWorkspaceName system property to true for any Multi-configuration projects Configure a custom workspace in Advanced Project Options

          Thomas COAN added a comment - - edited

          Ok problem comes from the length of windows path > 255.
          Problem is solved for us => should be a jenkins restriction

          thanks Daniel,
          I will check your proposition but we have some restrictions as some jobs are generated by an automate (=> to do in our side for the choice of folder names)

          Thomas COAN added a comment - - edited Ok problem comes from the length of windows path > 255. Problem is solved for us => should be a jenkins restriction thanks Daniel, I will check your proposition but we have some restrictions as some jobs are generated by an automate (=> to do in our side for the choice of folder names)

            ndeloof Nicolas De Loof
            hannupiki Hannu Piki
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: