Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17186

Linux: if there is a recursive directory within the git repo you're building, the initial checkout will fail badly

      If a recursive symlink is present in one's git repo the new git/git-client plugins will fail badly as shown below. While having such a symlink may not be ideal, until git 1.3.0/ git-client 1.0.4 Jenkins did not have problems with it (nor have I had any other git client or tool fail because of this symlink).

      This appears to be specifically related to changes in the new git 1.3.0 plugin or its dependency git-client 1.0.4. Downgrading to the git 1.2.0 plugin and manually downgrading to the git-client 1.0.3 plugin was enough to remove this issue entirely. The problems seems specific to jgit.

      Note that I'm using Jenkins 1.505 and all other plugins are up-to-date.

      The details below contrast the working and broken builds (the only difference is the versions of the git plugins and their effects).

      ----------------------------------------------------------------------------
      A normal build (when using git 1.2.0 and git-client 1.0.3):
      ----------------------------------------------------------------------------
      Retriggered by user myself for Gerrit: http://gerritserver/710
      Building on master in workspace /var/lib/jenkins/jobs/puppet/workspace
      Checkout:workspace / /var/lib/jenkins/jobs/puppet/workspace - hudson.remoting.LocalChannel@1307e9af
      Using strategy: Gerrit Trigger
      Last Built Revision: Revision b3f2bd79784969bd18866ff666f0ab46136ff196 (mybranch)
      Using shallow clone
      Wiping out workspace first.
      Cloning the remote Git repository
      Cloning repository git://gitserver/puppet
      git --version
      git version 1.7.9.5
      Fetching upstream changes from origin
      Commencing build of Revision b3f2bd79784969bd18866ff666f0ab46136ff196 (mybranch)
      Checking out Revision b3f2bd79784969bd18866ff666f0ab46136ff196 (mybranch)
      [workspace] $ /bin/bash -ex /tmp/hudson3918244826320546803.sh
      [--== ...and so on to a successful build ==--]

      ----------------------------------------------------------------------------
      Build break (when using git 1.3.0 and git-client 1.0.4):
      ----------------------------------------------------------------------------
      Retriggered by user myself for Gerrit: http://gerritserver/710
      Building on master in workspace /var/lib/jenkins/jobs/puppet/workspace
      Checkout:workspace / /var/lib/jenkins/jobs/puppet/workspace - hudson.remoting.LocalChannel@6f0bc962
      Using strategy: Gerrit Trigger
      Last Built Revision: Revision b3f2bd79784969bd18866ff666f0ab46136ff196 (mybranch)
      Using shallow clone
      Wiping out workspace first.
      Cloning the remote Git repository
      Cloning repository git://gitserver/puppet
      git --version
      git version 1.7.9.5
      Commencing build of Revision b3f2bd79784969bd18866ff666f0ab46136ff196 (mybranch)
      Checking out Revision b3f2bd79784969bd18866ff666f0ab46136ff196 (mybranch)
      FATAL: Could not checkout b3f2bd79784969bd18866ff666f0ab46136ff196
      hudson.plugins.git.GitException: Could not checkout b3f2bd79784969bd18866ff666f0ab46136ff196
      at org.jenkinsci.plugins.gitclient.JGitAPIImpl.checkout(JGitAPIImpl.java:68)
      at hudson.plugins.git.GitAPI.checkout(GitAPI.java:208)
      at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1273)
      at hudson.plugins.git.GitSCM.access$1100(GitSCM.java:57)
      at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1232)
      at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1208)
      at hudson.FilePath.act(FilePath.java:865)
      at hudson.FilePath.act(FilePath.java:838)
      at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1208)
      at hudson.model.AbstractProject.checkout(AbstractProject.java:1353)
      at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:683)
      at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
      at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:588)
      at hudson.model.Run.execute(Run.java:1567)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:237)
      Caused by: org.eclipse.jgit.api.errors.CheckoutConflictException: Checkout conflict with files:
      apt/spec/fixtures/modules/apt <------ [ this is a symlink to ../../.. ]
      apt/spec/fixtures/modules/apt/.gitignore
      apt/spec/fixtures/modules/apt/CHANGELOG
      apt/spec/fixtures/modules/apt/manifests
      apt/spec/fixtures/modules/apt/manifests/init.pp
      apt/spec/fixtures/modules/apt/spec
      apt/spec/fixtures/modules/apt/spec/fixtures
      apt/spec/fixtures/modules/apt/spec/fixtures/manifests
      apt/spec/fixtures/modules/apt/spec/fixtures/manifests/site.pp
      apt/spec/fixtures/modules/apt/spec/fixtures/modules
      apt/spec/fixtures/modules/apt/spec/fixtures/modules/apt <--------- [ and so on... ]
      apt/spec/fixtures/modules/apt/spec/fixtures/modules/apt/.gitignore
      apt/spec/fixtures/modules/apt/spec/fixtures/modules/apt/CHANGELOG

      [--== trimmed ~9 MB of recursive output ==--]

      apt/spec/fixtures/modules/apt/tests/source.pp
      at org.eclipse.jgit.dircache.DirCacheCheckout.doCheckout(DirCacheCheckout.java:411)
      at org.eclipse.jgit.dircache.DirCacheCheckout.checkout(DirCacheCheckout.java:391)
      at org.eclipse.jgit.api.CheckoutCommand.call(CheckoutCommand.java:240)
      ... 17 more

      [--== build failed ==--]

          [JENKINS-17186] Linux: if there is a recursive directory within the git repo you're building, the initial checkout will fail badly

          Martin Falatic added a comment - - edited

          I updated to plugins git 1.3.0 and git-client 1.0.4, then updated the git-client to the 1.0.5 test snapshot, then restarted Jenkins and re-triggered an affected build. The test failed, though differently than before (output below).

          Note: after this, I did have the option to downgrade git-client to 1.0.3, just like 1.2.0 for git. I clicked both, then restarted, but again the git-client never actually downgraded (and I lost the option to downgrade that one after the reboot). Strange. I manually downgraded it to restore normal functionality.

          Commencing build of Revision 2c09adc94ac7245ae8dacdaac82227443ce0d7bc (mybranch)
          Checking out Revision 2c09adc94ac7245ae8dacdaac82227443ce0d7bc (mybranch)
          FATAL: Could not checkout 2c09adc94ac7245ae8dacdaac82227443ce0d7bc
          hudson.plugins.git.GitException: Could not checkout 2c09adc94ac7245ae8dacdaac82227443ce0d7bc
          at org.jenkinsci.plugins.gitclient.JGitAPIImpl.checkout(JGitAPIImpl.java:68)
          at hudson.plugins.git.GitAPI.checkout(GitAPI.java:208)
          at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1273)
          at hudson.plugins.git.GitSCM.access$1100(GitSCM.java:57)
          at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1232)
          at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1208)
          at hudson.FilePath.act(FilePath.java:865)
          at hudson.FilePath.act(FilePath.java:838)
          at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1208)
          at hudson.model.AbstractProject.checkout(AbstractProject.java:1353)
          at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:683)
          at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:588)
          at hudson.model.Run.execute(Run.java:1567)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
          at hudson.model.ResourceController.execute(ResourceController.java:88)
          at hudson.model.Executor.run(Executor.java:237)
          Caused by: org.eclipse.jgit.api.errors.CheckoutConflictException: Checkout conflict with file: Cannot delete file: apt/spec/fixtures/modules/apt/.gitignore
          at org.eclipse.jgit.api.CheckoutCommand.call(CheckoutCommand.java:246)
          at org.jenkinsci.plugins.gitclient.JGitAPIImpl.checkout(JGitAPIImpl.java:64)
          ... 16 more
          Caused by: org.eclipse.jgit.errors.CheckoutConflictException: Checkout conflict with file: Cannot delete file: apt/spec/fixtures/modules/apt/.gitignore
          at org.eclipse.jgit.dircache.DirCacheCheckout.cleanUpConflicts(DirCacheCheckout.java:1011)
          at org.eclipse.jgit.dircache.DirCacheCheckout.doCheckout(DirCacheCheckout.java:413)
          at org.eclipse.jgit.dircache.DirCacheCheckout.checkout(DirCacheCheckout.java:391)
          at org.eclipse.jgit.api.CheckoutCommand.call(CheckoutCommand.java:242)
          ... 17 more

          Martin Falatic added a comment - - edited I updated to plugins git 1.3.0 and git-client 1.0.4, then updated the git-client to the 1.0.5 test snapshot, then restarted Jenkins and re-triggered an affected build. The test failed, though differently than before (output below). Note: after this, I did have the option to downgrade git-client to 1.0.3, just like 1.2.0 for git. I clicked both, then restarted, but again the git-client never actually downgraded (and I lost the option to downgrade that one after the reboot). Strange. I manually downgraded it to restore normal functionality. Commencing build of Revision 2c09adc94ac7245ae8dacdaac82227443ce0d7bc (mybranch) Checking out Revision 2c09adc94ac7245ae8dacdaac82227443ce0d7bc (mybranch) FATAL: Could not checkout 2c09adc94ac7245ae8dacdaac82227443ce0d7bc hudson.plugins.git.GitException: Could not checkout 2c09adc94ac7245ae8dacdaac82227443ce0d7bc at org.jenkinsci.plugins.gitclient.JGitAPIImpl.checkout(JGitAPIImpl.java:68) at hudson.plugins.git.GitAPI.checkout(GitAPI.java:208) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1273) at hudson.plugins.git.GitSCM.access$1100(GitSCM.java:57) at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1232) at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1208) at hudson.FilePath.act(FilePath.java:865) at hudson.FilePath.act(FilePath.java:838) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1208) at hudson.model.AbstractProject.checkout(AbstractProject.java:1353) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:683) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:588) at hudson.model.Run.execute(Run.java:1567) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:237) Caused by: org.eclipse.jgit.api.errors.CheckoutConflictException: Checkout conflict with file: Cannot delete file: apt/spec/fixtures/modules/apt/.gitignore at org.eclipse.jgit.api.CheckoutCommand.call(CheckoutCommand.java:246) at org.jenkinsci.plugins.gitclient.JGitAPIImpl.checkout(JGitAPIImpl.java:64) ... 16 more Caused by: org.eclipse.jgit.errors.CheckoutConflictException: Checkout conflict with file: Cannot delete file: apt/spec/fixtures/modules/apt/.gitignore at org.eclipse.jgit.dircache.DirCacheCheckout.cleanUpConflicts(DirCacheCheckout.java:1011) at org.eclipse.jgit.dircache.DirCacheCheckout.doCheckout(DirCacheCheckout.java:413) at org.eclipse.jgit.dircache.DirCacheCheckout.checkout(DirCacheCheckout.java:391) at org.eclipse.jgit.api.CheckoutCommand.call(CheckoutCommand.java:242) ... 17 more

          David Barri added a comment -

          Wow the new version of this plugin completely broke my jobs. I have a single symlink from one directory to another (non-recursive) and my builds refuse to update, showing the same "Cannot delete file" error.

          I don't think JGit handles symlinks at all does it? I remember reading something a while back about the JGit team not planning on building in symlink support at all until Java gets native support for symlinks. If that's still the case then migrating to JGit is a disastrous idea for those of us in the Linux world.

          And what's this about Git CLI being fragile, yada yada? Git CLI has a porcelain mode so that tools can rely on it without worrying about version or output fragility.

          David Barri added a comment - Wow the new version of this plugin completely broke my jobs. I have a single symlink from one directory to another (non-recursive) and my builds refuse to update, showing the same "Cannot delete file" error. I don't think JGit handles symlinks at all does it? I remember reading something a while back about the JGit team not planning on building in symlink support at all until Java gets native support for symlinks. If that's still the case then migrating to JGit is a disastrous idea for those of us in the Linux world. And what's this about Git CLI being fragile, yada yada? Git CLI has a porcelain mode so that tools can rely on it without worrying about version or output fragility.

          David Barri added a comment -

          David Barri added a comment - My god.... https://bugs.eclipse.org/bugs/show_bug.cgi?id=354367

          Indeed. I realize there's a drive to move to Jgit for git-client, but it's apparently not ready for prime time when it comes to complex filesystems in Unix.

          More info on git's "procelain" versus "plumbing" (and the counter-intuitive use of the term porcelain versus the option "--porcelain") can be found here:

          http://stackoverflow.com/a/6978402/760905

          Since other VCS plugins were switched to pure Java versions, do those plugins handle symlinks and other low-level, OS-dependent features properly? (Of course CVS doesn't know about symlinks to begin with, but SVN certainly does.)

          Martin Falatic added a comment - Indeed. I realize there's a drive to move to Jgit for git-client, but it's apparently not ready for prime time when it comes to complex filesystems in Unix. More info on git's "procelain" versus "plumbing" (and the counter-intuitive use of the term porcelain versus the option "--porcelain") can be found here: http://stackoverflow.com/a/6978402/760905 Since other VCS plugins were switched to pure Java versions, do those plugins handle symlinks and other low-level, OS-dependent features properly? (Of course CVS doesn't know about symlinks to begin with, but SVN certainly does.)

          Mark Waite added a comment -

          Does git-client plugin 1.0.5 resolve this issue with its switch from JGit implementation back to the Git command line client implementation?

          Mark Waite added a comment - Does git-client plugin 1.0.5 resolve this issue with its switch from JGit implementation back to the Git command line client implementation?

          I updated my JIRA with this git-client plugin and so far I didn't see this issue coming back. I should say the new plugin resolve the issue.

          Nicolas Berniolles added a comment - I updated my JIRA with this git-client plugin and so far I didn't see this issue coming back. I should say the new plugin resolve the issue.

          Mark Waite added a comment -

          Fixed by git-client-plugin 1.0.5 which switches the default implementation back to command line git. The JGit implementation is not yet complete enough to satisfy all the Jenkins use cases.

          Mark Waite added a comment - Fixed by git-client-plugin 1.0.5 which switches the default implementation back to command line git. The JGit implementation is not yet complete enough to satisfy all the Jenkins use cases.

          I just updated to 1.510 (and updated the plugins) and this issue appears to be resolved.

          Martin Falatic added a comment - I just updated to 1.510 (and updated the plugins) and this issue appears to be resolved.

          This appears to be resolved.

          Martin Falatic added a comment - This appears to be resolved.

          James Coleman added a comment -

          There seems to be a similar problem with cvs plugin upon cvs update.
          A recursive directory causes job to hang forever.
          Using strace can see process is recursing into directory to multiple levels . . .
          CVS Plugin 2.11
          http://wiki.jenkins-ci.org/display/JENKINS/CVS+Plugin
          Jenkins ver. 1.562

          James Coleman added a comment - There seems to be a similar problem with cvs plugin upon cvs update. A recursive directory causes job to hang forever. Using strace can see process is recursing into directory to multiple levels . . . CVS Plugin 2.11 http://wiki.jenkins-ci.org/display/JENKINS/CVS+Plugin Jenkins ver. 1.562

            ndeloof Nicolas De Loof
            martymacgyver Martin Falatic
            Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: