Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-15103

jenkins git does not release pack files, prevents wiping workspace

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • git-plugin
    • None
    • Windows XP Slave with msysgit v 1.7.11
      Windows 7 32-Bit SP1 with msysgit 1.7.9
      Windows Server 2008 with msysgit 1.8.0-preview20121012

      When building a job with a git branch specifier that ends with a wildcard (ie origin/feature/* or origin/feature*) jenkins never releases its hold on .git/object/pack/.pack files. This causes the next build to not be able to wipe the workspace. This is not a problem if the branch specifier does not end with a wildcard.

      The *.pack file cannot be deleted until the javaw.exe process (JNLP connection to jenkins) is killed.

      The only way I have been able to get by is to not wipe the workspace before builds.

      This is the stack trace that this causes...

      Using strategy: Default
      Last Built Revision: Revision cfd6d3ea8c621905444d4e84b4718bf5d40cbe59 (origin/feature/jeff-test)
      Wiping out workspace first.
      hudson.util.IOException2: remote file operation failed: E:\jenkins\workspace\VoiceFirst-WebServices-feature at hudson.remoting.Channel@3007615c:VHS-Lab-YosWin0
      at hudson.FilePath.act(FilePath.java:838)
      at hudson.FilePath.act(FilePath.java:824)
      at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:1044)
      at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1200)
      at hudson.model.AbstractProject.checkout(AbstractProject.java:1248)
      at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:589)
      at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
      at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:494)
      at hudson.model.Run.execute(Run.java:1484)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:236)
      Caused by: java.io.IOException: Unable to delete E:\jenkins\workspace\VoiceFirst-WebServices-feature\.git\objects\pack\pack-7afe54f6a971bd2518f1713140acd04dee21882c.pack
      at hudson.Util.deleteFile(Util.java:237)
      at hudson.Util.deleteRecursive(Util.java:287)
      at hudson.Util.deleteContentsRecursive(Util.java:198)
      at hudson.Util.deleteRecursive(Util.java:278)
      at hudson.Util.deleteContentsRecursive(Util.java:198)
      at hudson.Util.deleteRecursive(Util.java:278)
      at hudson.Util.deleteContentsRecursive(Util.java:198)
      at hudson.Util.deleteRecursive(Util.java:278)
      at hudson.Util.deleteContentsRecursive(Util.java:198)
      at hudson.FilePath$12.invoke(FilePath.java:922)
      at hudson.FilePath$12.invoke(FilePath.java:920)
      at hudson.FilePath.act(FilePath.java:842)
      at hudson.FilePath.act(FilePath.java:824)
      at hudson.FilePath.deleteContents(FilePath.java:920)
      at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1057)
      at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1044)
      at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2180)
      at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      at hudson.remoting.Request$2.run(Request.java:287)
      at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at hudson.remoting.Engine$1$1.run(Engine.java:60)
      at java.lang.Thread.run(Thread.java:662)

          [JENKINS-15103] jenkins git does not release pack files, prevents wiping workspace

          Kevin Burge added a comment - - edited

          The work around works for me, but I had to remove "Wipe out workspace after build" post build step as well as restart the slaves. I did not have the "Wipe out workspace before build" option enabled.

          Edit: of course, I won't really know until I re-run the build.

          Kevin Burge added a comment - - edited The work around works for me, but I had to remove "Wipe out workspace after build" post build step as well as restart the slaves. I did not have the "Wipe out workspace before build" option enabled. Edit: of course, I won't really know until I re-run the build.

          Omid RA added a comment - - edited

          I am also seeing this happening. My setup is as follows:

          • Jenkins 1.506 (all plugins up to date as of 2013/03/20)
          • Build master and slaves are all RedHat 5 machines.
          • Using git for all SCM
          • Currently using "Wipe out workspace before build" option

          If I log into any build slave and use the "lsof" command to see what processes have files open, I see that the java instance of the build slave has files open in the .git/objects/pack/ directory of several jobs.

          I have to restart all build-slaves that have files open before I can run the job again.

          EDIT: I also want to say that I'm seeing this even when the branch specifier is explicit: "origin/master". no wildcard is being used...

          EDIT2: So, I've figured out how to work around my issue. If I add this JVM option to each slave, everything works fine:
          -Dorg.jenkinsci.plugins.gitclient.Git.useCLI=true

          Omid RA added a comment - - edited I am also seeing this happening. My setup is as follows: Jenkins 1.506 (all plugins up to date as of 2013/03/20) Build master and slaves are all RedHat 5 machines. Using git for all SCM Currently using "Wipe out workspace before build" option If I log into any build slave and use the "lsof" command to see what processes have files open, I see that the java instance of the build slave has files open in the .git/objects/pack/ directory of several jobs. I have to restart all build-slaves that have files open before I can run the job again. EDIT: I also want to say that I'm seeing this even when the branch specifier is explicit: "origin/master". no wildcard is being used... EDIT2: So, I've figured out how to work around my issue. If I add this JVM option to each slave, everything works fine: -Dorg.jenkinsci.plugins.gitclient.Git.useCLI=true

          This issue really affects our projects, we had lot of projects failing because of this issue. Please consider fixing it soon.

          Abdelrahman Elogeel added a comment - This issue really affects our projects, we had lot of projects failing because of this issue. Please consider fixing it soon.

          Mark Waite added a comment -

          Since Omid was able to confirm that the problem is resolved with useCLI=true, and useCLI=true is now the default with git-client-plugin version 1.0.5, I believe this issue is resolved. I was unable to duplicate the issue with many different jobs on a Windows computer.

          Mark Waite added a comment - Since Omid was able to confirm that the problem is resolved with useCLI=true, and useCLI=true is now the default with git-client-plugin version 1.0.5, I believe this issue is resolved. I was unable to duplicate the issue with many different jobs on a Windows computer.

          Mark Waite added a comment -

          I used the latest released git client plugin (1.0.5) and the latest git plugin (1.3.0) with two different Windows machines (one Windows 6.1 kernel [Windows Home Server 2011], one Windows 6.2 kernel [Windows 8]) and was unable to duplicate this problem. As far as I can tell, it is resolved.

          Mark Waite added a comment - I used the latest released git client plugin (1.0.5) and the latest git plugin (1.3.0) with two different Windows machines (one Windows 6.1 kernel [Windows Home Server 2011] , one Windows 6.2 kernel [Windows 8] ) and was unable to duplicate this problem. As far as I can tell, it is resolved.

          Mark Waite added a comment - - edited

          This was working correctly in Git plugin 1.5.0 so long as the command line implementation was used. It no longer works correctly in Git plugin 2.0, even when the command line implementation is used. Some portion of the Git plugin or Git client plugin seems to be opening a pack file and not closing it.

          Refer to JENKINS-19994 for a bug which may be a duplicate of this bug.

          Mark Waite added a comment - - edited This was working correctly in Git plugin 1.5.0 so long as the command line implementation was used. It no longer works correctly in Git plugin 2.0, even when the command line implementation is used. Some portion of the Git plugin or Git client plugin seems to be opening a pack file and not closing it. Refer to JENKINS-19994 for a bug which may be a duplicate of this bug.

          Mark Waite added a comment - - edited

          The automated tests executed by "mvn install" on the git-client-plugin repository seem to show the same problem. When those tests are run on Linux, they pass. When the same tests are run on Windows, several of the tests fail with the message "Failed to clean up temp dirs"

          The tests which are failing include:

          test_checkoutBranchFailure(org.jenkinsci.plugins.gitclient.JGitAPIImplTest): Failed to clean up temp dirs
          test_setSubmoduleUrl(org.jenkinsci.plugins.gitclient.JGitAPIImplTest): Failed to clean up temp dirs
          test_show_revision_for_single_commit(org.jenkinsci.plugins.gitclient.JGitAPIImplTest): Failed to clean up temp dirs
          test_getAllLogEntries(org.jenkinsci.plugins.gitclient.JGitAPIImplTest): Failed to clean up temp dirs
          test_revListAll(org.jenkinsci.plugins.gitclient.JGitAPIImplTest): Failed to clean up temp dirs
          test_fetch_with_updated_tag(org.jenkinsci.plugins.gitclient.JGitAPIImplTest): Failed to clean up temp dirs

          Within the failure details, the message "IOException: Unable to delete ... pack" is listed.

          Note that the same unit tests pass for the command line implementation. I think that makes the current behavior different from the originally reported behavior. The original report required that there was a certain format for the branch to build, while the unit test failure shows that it will fail for any case which uses JGit as the git executable instead of using the command line as the git executable.

          Unfortunately, those test failures only happen in the JGit implementation, not in the CLI implementation. This bug report was submitted confirmed as a problem with both the JGit and the CLI implementations, so fixing the JGit specific test failures will not be enough.

          Mark Waite added a comment - - edited The automated tests executed by "mvn install" on the git-client-plugin repository seem to show the same problem. When those tests are run on Linux, they pass. When the same tests are run on Windows, several of the tests fail with the message "Failed to clean up temp dirs" The tests which are failing include: test_checkoutBranchFailure(org.jenkinsci.plugins.gitclient.JGitAPIImplTest): Failed to clean up temp dirs test_setSubmoduleUrl(org.jenkinsci.plugins.gitclient.JGitAPIImplTest): Failed to clean up temp dirs test_show_revision_for_single_commit(org.jenkinsci.plugins.gitclient.JGitAPIImplTest): Failed to clean up temp dirs test_getAllLogEntries(org.jenkinsci.plugins.gitclient.JGitAPIImplTest): Failed to clean up temp dirs test_revListAll(org.jenkinsci.plugins.gitclient.JGitAPIImplTest): Failed to clean up temp dirs test_fetch_with_updated_tag(org.jenkinsci.plugins.gitclient.JGitAPIImplTest): Failed to clean up temp dirs Within the failure details, the message "IOException: Unable to delete ... pack" is listed. Note that the same unit tests pass for the command line implementation. I think that makes the current behavior different from the originally reported behavior. The original report required that there was a certain format for the branch to build, while the unit test failure shows that it will fail for any case which uses JGit as the git executable instead of using the command line as the git executable. Unfortunately, those test failures only happen in the JGit implementation, not in the CLI implementation. This bug report was submitted confirmed as a problem with both the JGit and the CLI implementations, so fixing the JGit specific test failures will not be enough.

          Mark Waite added a comment - - edited

          Report conditions seem very similar in JENKINS-19994

          Mark Waite added a comment - - edited Report conditions seem very similar in JENKINS-19994

          Mark Waite added a comment -

          Using git-client-plugin 1.6.2 and git-plugin 2.0.1, I can consistently wipe the workspace on Windows 7, Windows 8.1, Windows Home Server 2011 so long as I use the default Git implementation (which is the command line).

          If I switch to using JGit as the implementation, the job fails because the pack file is still busy. That's the condition described by JENKINS-19994 and JENKINS-18843.

          Can anyone else describe conditions where the command line git implementation shows this problem on any platform?

          Mark Waite added a comment - Using git-client-plugin 1.6.2 and git-plugin 2.0.1, I can consistently wipe the workspace on Windows 7, Windows 8.1, Windows Home Server 2011 so long as I use the default Git implementation (which is the command line). If I switch to using JGit as the implementation, the job fails because the pack file is still busy. That's the condition described by JENKINS-19994 and JENKINS-18843 . Can anyone else describe conditions where the command line git implementation shows this problem on any platform?

          Mark Waite added a comment -

          Confirmed fixed in git-plugin 2.0.1 and git-client-plugin 1.6.2 for command line implementation.

          JENKINS-19994 and others describe the same issue for the JGit implementation, which is not yet fixed.

          Since command line implementation is the default implementation and the most feature rich, this bug is resolved.

          Mark Waite added a comment - Confirmed fixed in git-plugin 2.0.1 and git-client-plugin 1.6.2 for command line implementation. JENKINS-19994 and others describe the same issue for the JGit implementation, which is not yet fixed. Since command line implementation is the default implementation and the most feature rich, this bug is resolved.

            ndeloof Nicolas De Loof
            jkoenig Jeff Koenig
            Votes:
            10 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved: