Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-23694

git clean fails with submodules (failed to fetch)

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • git-plugin
    • None
    • windows node
      git plugin version 2.2.2
      git client plugin version 1.9.1

      Hi,

      i try to build the c++ library boost on a windows node. But as soon as i add "clean after checkout" or "clean before checkout" the job fails with:

      > git clean -fdx
      FATAL: Failed to fetch from https://github.com/boostorg/boost.git
      hudson.plugins.git.GitException: Failed to fetch from https://github.com/boostorg/boost.git
      at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:622)
      at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:854)
      at hudson.plugins.git.GitSCM.checkout(GitSCM.java:879)
      at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
      at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:624)
      at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
      at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:530)
      at hudson.model.Run.execute(Run.java:1732)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:234)
      Caused by: hudson.plugins.git.GitException: Command "git clean -fdx" returned status code 1:

          [JENKINS-23694] git clean fails with submodules (failed to fetch)

          Mark Waite added a comment - - edited

          Comment deleted - incorrect. The issue is related to git clean -xfd not removing untracked directories in a project which uses git submodules. The command "git clean -xffd" needs to be used instead, though there are cases which make that problematic as well.

          Mark Waite added a comment - - edited Comment deleted - incorrect. The issue is related to git clean -xfd not removing untracked directories in a project which uses git submodules. The command "git clean -xffd" needs to be used instead, though there are cases which make that problematic as well.

          Mark Waite added a comment -

          Since https://github.com/boostorg/boost.git is using submodules, I think you'll need to experiment with various settings, or you may even need to add the Pre-scm build step plugin and insert a pre-SCM build step which performs a

          git clean -xffd
          

          When git clean is passed two "f" options and the "d" option, it cleans untracked directories as well as untracked files. Refer to http://git-scm.com/docs/git-clean for more details.

          Mark Waite added a comment - Since https://github.com/boostorg/boost.git is using submodules, I think you'll need to experiment with various settings, or you may even need to add the Pre-scm build step plugin and insert a pre-SCM build step which performs a git clean -xffd When git clean is passed two "f" options and the "d" option, it cleans untracked directories as well as untracked files. Refer to http://git-scm.com/docs/git-clean for more details.

          Daniel Neus added a comment - - edited

          I was able to get it working by using the pre-SCM build step plugin and the command:

          git submodule foreach --recursive "git clean -dfx"

          thanks!!

          Daniel Neus added a comment - - edited I was able to get it working by using the pre-SCM build step plugin and the command: git submodule foreach --recursive "git clean -dfx" thanks!!

          Mark Waite added a comment -

          As far as I can tell from watching how the plugin behaved for me, that "submodule foreach" sequence is exactly what is used if you enable submodule recursive update. I'm surprised you needed the pre-SCM build step, but glad it worked for you.

          Mark Waite added a comment - As far as I can tell from watching how the plugin behaved for me, that "submodule foreach" sequence is exactly what is used if you enable submodule recursive update. I'm surprised you needed the pre-SCM build step, but glad it worked for you.

          Daniel Neus added a comment - - edited

          I think "submodule recursive update" is doing a

           
          git submodule foreach --recursive "git pull"
          

          instead of what i am trying to archive:

           
          git submodule foreach --recursive "git clean -dfx"
          

          would be nice if there is an option for this

          Daniel Neus added a comment - - edited I think "submodule recursive update" is doing a git submodule foreach --recursive "git pull" instead of what i am trying to archive: git submodule foreach --recursive "git clean -dfx" would be nice if there is an option for this

          Mark Waite added a comment - - edited

          The job I had defined (initially running on Linux), included the "Clean before checkout" behaviour added to the git SCM "Additional behaviour" section of the job definition. With that added to the job definition, I see:

          git submodule foreach --recursive git clean -fdx
          

          Full console output:

          Started by user anonymous
          Building remotely on jessie64a (Debian amd64-Debian linux testing Debian-testing amd64-Debian-testing amd64) in workspace /var/lib/jenkins/wheezy64b-slave/workspace/JENKINS-23694-clean-fails-with-submodules-present
           > git rev-parse --is-inside-work-tree
          Fetching changes from the remote Git repository
           > git config remote.origin.url git://github.com/boostorg/boost.git
          Cleaning workspace
           > git rev-parse --verify HEAD
          Resetting working tree
           > git reset --hard
           > git clean -fdx
           > git submodule foreach --recursive git reset --hard
           > git submodule foreach --recursive git clean -fdx
          Fetching upstream changes from git://github.com/boostorg/boost.git
           > git --version
           > git fetch --tags --progress git://github.com/boostorg/boost.git +refs/heads/*:refs/remotes/origin/*
           > git rev-parse origin/master^{commit}
          Checking out Revision 85b8813a1cf348ea84a0f99ed5319f39c140900f (origin/master)
           > git config core.sparsecheckout
           > git checkout -f 85b8813a1cf348ea84a0f99ed5319f39c140900f
           > git rev-list 85b8813a1cf348ea84a0f99ed5319f39c140900f
           > git remote
           > git submodule init
           > git submodule sync
           > git config --get remote.origin.url
           > git submodule update --init --recursive
          [JENKINS-23694-clean-fails-with-submodules-present] $ /bin/sh -xe /tmp/hudson1079411972338520157.sh
          + date
          + date
          Finished: SUCCESS
          

          Mark Waite added a comment - - edited The job I had defined (initially running on Linux), included the "Clean before checkout" behaviour added to the git SCM "Additional behaviour" section of the job definition. With that added to the job definition, I see: git submodule foreach --recursive git clean -fdx Full console output: Started by user anonymous Building remotely on jessie64a (Debian amd64-Debian linux testing Debian-testing amd64-Debian-testing amd64) in workspace /var/lib/jenkins/wheezy64b-slave/workspace/JENKINS-23694-clean-fails-with-submodules-present > git rev-parse --is-inside-work-tree Fetching changes from the remote Git repository > git config remote.origin.url git://github.com/boostorg/boost.git Cleaning workspace > git rev-parse --verify HEAD Resetting working tree > git reset --hard > git clean -fdx > git submodule foreach --recursive git reset --hard > git submodule foreach --recursive git clean -fdx Fetching upstream changes from git://github.com/boostorg/boost.git > git --version > git fetch --tags --progress git://github.com/boostorg/boost.git +refs/heads/*:refs/remotes/origin/* > git rev-parse origin/master^{commit} Checking out Revision 85b8813a1cf348ea84a0f99ed5319f39c140900f (origin/master) > git config core.sparsecheckout > git checkout -f 85b8813a1cf348ea84a0f99ed5319f39c140900f > git rev-list 85b8813a1cf348ea84a0f99ed5319f39c140900f > git remote > git submodule init > git submodule sync > git config --get remote.origin.url > git submodule update --init --recursive [JENKINS-23694-clean-fails-with-submodules-present] $ /bin/sh -xe /tmp/hudson1079411972338520157.sh + date + date Finished: SUCCESS

          Daniel Neus added a comment -

          ok, i tried this again with following output:

           
          ...
           > git reset --hard
           > git clean -fdx
          FATAL: Failed to fetch from https://github.com/boostorg/boost.git
          hudson.plugins.git.GitException: Failed to fetch from https://github.com/boostorg/boost.git
          	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:622)
          	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:854)
          	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:879)
          	at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:624)
          	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:530)
          	at hudson.model.Run.execute(Run.java:1732)
          	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          	at hudson.model.ResourceController.execute(ResourceController.java:88)
          	at hudson.model.Executor.run(Executor.java:234)
          Caused by: hudson.plugins.git.GitException: Command "git clean -fdx" returned status code 1:
          stdout: 
          stderr: warning: failed to remove boost/accumulators
          ...
          

          the first "git clean -fdx" is failing and therefore i'm not seeing the next step "git submodule foreach --recursive git reset --hard".

          maybe there is a problem with this on windows nodes.

          Daniel Neus added a comment - ok, i tried this again with following output: ... > git reset --hard > git clean -fdx FATAL: Failed to fetch from https://github.com/boostorg/boost.git hudson.plugins.git.GitException: Failed to fetch from https://github.com/boostorg/boost.git at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:622) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:854) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:879) at hudson.model.AbstractProject.checkout(AbstractProject.java:1252) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:624) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:530) at hudson.model.Run.execute(Run.java:1732) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:234) Caused by: hudson.plugins.git.GitException: Command "git clean -fdx" returned status code 1: stdout: stderr: warning: failed to remove boost/accumulators ... the first "git clean -fdx" is failing and therefore i'm not seeing the next step "git submodule foreach --recursive git reset --hard". maybe there is a problem with this on windows nodes.

          Mark Doliner added a comment -

          Related pull request: https://github.com/jenkinsci/git-client-plugin/pull/163
          It hasn't been tested and is missing unit tests so it's unlikely it'll get merged in. If anyone in the future encounters this issue and has time to test and write unit tests, please feel free!

          Mark Doliner added a comment - Related pull request: https://github.com/jenkinsci/git-client-plugin/pull/163 It hasn't been tested and is missing unit tests so it's unlikely it'll get merged in. If anyone in the future encounters this issue and has time to test and write unit tests, please feel free!

          Just chiming in that this repros for me as well, with two errors:

          ERROR: Error fetching remote repo 'origin'
          ...
          Caused by: hudson.plugins.git.GitException: Command "git clean -fdx" returned status code 128:
          stdout: 
          stderr: fatal: Not a git repository: ...
          

          Also fixed with a pre-SCM build step:

          git reset --hard
          git clean -fx
          git submodule foreach --recursive git reset --hard
          git submodule foreach --recursive git clean -fx
          

          Interestingly, adding -d to git clean causes the error to repro, even in the pre-SCM build step.

          Andrew Schwartzmeyer added a comment - Just chiming in that this repros for me as well, with two errors: ERROR: Error fetching remote repo 'origin' ... Caused by: hudson.plugins.git.GitException: Command "git clean -fdx" returned status code 128: stdout: stderr: fatal: Not a git repository: ... Also fixed with a pre-SCM build step: git reset --hard git clean -fx git submodule foreach --recursive git reset --hard git submodule foreach --recursive git clean -fx Interestingly, adding -d to git clean causes the error to repro, even in the pre-SCM build step.

            Unassigned Unassigned
            neusdan Daniel Neus
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: