Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27082

Sparse checkouts and submodules: Error on checkout, submodule missing

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • git-plugin
    • None
    • jenkins 1.580.3, git-plugin-2.3.5

      When using a repository with (relative) submodules and sparse checkouts, the GIT plugin fails on the initial (and subsequent) checkouts and the resulting working copy does not include the files of the submodule:

      Started by user Ronny Schütz
      Building on master in workspace /var/lib/jenkins/jobs/testrs_git/workspace
      Using strategy: Default
      Last Built Revision: Revision dfe102e4a93d71f242bd23d67363c9398f01ea54 (refs/remotes/origin/master)
      Cloning the remote Git repository
      Using no checkout clone with sparse checkout.
      Cloning repository ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git
       > /usr/bin/git init /var/lib/jenkins/jobs/testrs_git/workspace # timeout=360
      Fetching upstream changes from ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git
       > /usr/bin/git --version # timeout=360
      using GIT_SSH to set credentials ttbsbldmac101.ttg.global
       > /usr/bin/git -c core.askpass=true fetch --tags --progress ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git +refs/heads/*:refs/remotes/origin/*
      
       > /usr/bin/git config remote.origin.url ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git # timeout=360
       > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=360
       > /usr/bin/git config remote.origin.url ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git # timeout=360
      Cleaning workspace
       > /usr/bin/git rev-parse --verify HEAD # timeout=360
      No valid HEAD. Skipping the resetting
       > /usr/bin/git clean -fdx # timeout=360
      Fetching upstream changes from ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git
      using GIT_SSH to set credentials ttbsbldmac101.ttg.global
       > /usr/bin/git -c core.askpass=true fetch --tags --progress ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git +refs/heads/*:refs/remotes/origin/*
      getCandidateRevisions(false,origin/master,,,,remoteUrls=[ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git],buildsByBranchName={refs/remotes/origin/master=Build #15 of Revision dfe102e4a93d71f242bd23d67363c9398f01ea54 (refs/remotes/origin/master)},lastBuild=Build #15 of Revision dfe102e4a93d71f242bd23d67363c9398f01ea54 (refs/remotes/origin/master)]) considering branches to build
      Qualifying origin/master as a branch in repository origin -> refs/remotes/origin/master
      Qualifying origin/master as a branch in repository origin -> refs/remotes/origin/origin/master
       > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=360
      rev-parse refs/remotes/origin/master -> AnyObjectId[dfe102e4a93d71f242bd23d67363c9398f01ea54]
      Found a new commit AnyObjectId[dfe102e4a93d71f242bd23d67363c9398f01ea54] to be built on refs/remotes/origin/master
       > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=360
      Failed to rev-parse: refs/remotes/origin/origin/master
      Checking out Revision dfe102e4a93d71f242bd23d67363c9398f01ea54 (refs/remotes/origin/master)
       > /usr/bin/git config core.sparsecheckout # timeout=360
       > /usr/bin/git config core.sparsecheckout true # timeout=360
       > /usr/bin/git read-tree -mu HEAD # timeout=360
      Command "/usr/bin/git read-tree -mu HEAD" returned status code 128:
      stdout: 
      stderr: fatal: Not a valid object name HEAD
      
       > /usr/bin/git checkout -f dfe102e4a93d71f242bd23d67363c9398f01ea54
       > /usr/bin/git rev-list dfe102e4a93d71f242bd23d67363c9398f01ea54 # timeout=360
      [workspace] $ /bin/sh -xe /var/lib/jenkins/tmp/hudson5727280207521334905.sh
      + find . '!' -path '*/.git/*'
      + sort
      .
      ./dir1
      ./dir1/test1
      ./dir2
      ./dir2/test2
      ./.git
      Started calculate disk usage of build
      Finished Calculation of disk usage of build in 0 seconds
      Started calculate disk usage of workspace
      Finished Calculation of disk usage of workspace in 0 seconds
      Finished: SUCCESS
      

      Repositories (attached):

      • testcommon
        • commonfile1
        • commonfile2
      • testmain
        • dir1/test1
        • dir2/test2
        • dir3/test3
        • testcommon (submodule; referencing testcommon via ../testcommon.git)

      Jenkins job settings:

      • Source code management: GIT
        • Repository: ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git
        • Branches: */master
        • Clean after checkout
        • Advanced submodules behavior
          • Recursively update submodules: checked
        • Sparse checkout paths
          • dir1
          • dir2
      • Execute shell (to log working copy)
        • find . ! -path "*/.git/*" | sort

      If I remove the sparse checkout option, the submodule is checked out correctly.

          [JENKINS-27082] Sparse checkouts and submodules: Error on checkout, submodule missing

          Ronny Schuetz created issue -
          Ronny Schuetz made changes -
          Description Original: When using a repository with (relative) submodules and sparse checkouts, the GIT plugin fails on the initial (and subsequent) checkouts and the resulting working copy does not include the files of the submodule:

          {code}
          Started by user Ronny Schütz
          Building on master in workspace /var/lib/jenkins/jobs/testrs_git/workspace
          Using strategy: Default
          Last Built Revision: Revision dfe102e4a93d71f242bd23d67363c9398f01ea54 (refs/remotes/origin/master)
          Cloning the remote Git repository
          Using no checkout clone with sparse checkout.
          Cloning repository ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git
           > /usr/bin/git init /var/lib/jenkins/jobs/testrs_git/workspace # timeout=360
          Fetching upstream changes from ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git
           > /usr/bin/git --version # timeout=360
          using GIT_SSH to set credentials ttbsbldmac101.ttg.global
           > /usr/bin/git -c core.askpass=true fetch --tags --progress ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git +refs/heads/*:refs/remotes/origin/*

           > /usr/bin/git config remote.origin.url ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git # timeout=360
           > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=360
           > /usr/bin/git config remote.origin.url ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git # timeout=360
          Cleaning workspace
           > /usr/bin/git rev-parse --verify HEAD # timeout=360
          No valid HEAD. Skipping the resetting
           > /usr/bin/git clean -fdx # timeout=360
          Fetching upstream changes from ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git
          using GIT_SSH to set credentials ttbsbldmac101.ttg.global
           > /usr/bin/git -c core.askpass=true fetch --tags --progress ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git +refs/heads/*:refs/remotes/origin/*
          getCandidateRevisions(false,origin/master,,,,remoteUrls=[ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git],buildsByBranchName={refs/remotes/origin/master=Build #15 of Revision dfe102e4a93d71f242bd23d67363c9398f01ea54 (refs/remotes/origin/master)},lastBuild=Build #15 of Revision dfe102e4a93d71f242bd23d67363c9398f01ea54 (refs/remotes/origin/master)]) considering branches to build
          Qualifying origin/master as a branch in repository origin -> refs/remotes/origin/master
          Qualifying origin/master as a branch in repository origin -> refs/remotes/origin/origin/master
           > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=360
          rev-parse refs/remotes/origin/master -> AnyObjectId[dfe102e4a93d71f242bd23d67363c9398f01ea54]
          Found a new commit AnyObjectId[dfe102e4a93d71f242bd23d67363c9398f01ea54] to be built on refs/remotes/origin/master
           > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=360
          Failed to rev-parse: refs/remotes/origin/origin/master
          Checking out Revision dfe102e4a93d71f242bd23d67363c9398f01ea54 (refs/remotes/origin/master)
           > /usr/bin/git config core.sparsecheckout # timeout=360
           > /usr/bin/git config core.sparsecheckout true # timeout=360
           > /usr/bin/git read-tree -mu HEAD # timeout=360
          Command "/usr/bin/git read-tree -mu HEAD" returned status code 128:
          stdout:
          stderr: fatal: Not a valid object name HEAD

           > /usr/bin/git checkout -f dfe102e4a93d71f242bd23d67363c9398f01ea54
           > /usr/bin/git rev-list dfe102e4a93d71f242bd23d67363c9398f01ea54 # timeout=360
          [workspace] $ /bin/sh -xe /var/lib/jenkins/tmp/hudson5727280207521334905.sh
          + find . '!' -path '*/.git/*'
          + sort
          .
          ./dir1
          ./dir1/test1
          ./dir2
          ./dir2/test2
          ./.git
          Started calculate disk usage of build
          Finished Calculation of disk usage of build in 0 seconds
          Started calculate disk usage of workspace
          Finished Calculation of disk usage of workspace in 0 seconds
          Finished: SUCCESS
          {code}

          Repositories (attached):
          * testcommon
          ** commonfile1
          ** commonfile2

          * testmain
          ** dir1/test1
          ** dir2/test2
          ** dir3/test3
          ** testcommon (submodule; referencing testcommon via ../testcommon.git)

          Jenkins job settings:
          * Source code management: GIT
          ** Repository: ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git
          ** Branches: */master
          ** Clean after checkout
          ** Advanced submodules behavior
          *** Recursively update submodules: checked
          ** Sparse checkout paths
          *** dir1
          *** dir2
          * Execute shell (to log working copy)
          ** find . ! -path "*/.git/*" | sort

          If I remove the sparse checkout option, the submodule is checked out correctly.







          New: When using a repository with (relative) submodules and sparse checkouts, the GIT plugin fails on the initial (and subsequent) checkouts and the resulting working copy does not include the files of the submodule:

          {code}
          Started by user Ronny Schütz
          Building on master in workspace /var/lib/jenkins/jobs/testrs_git/workspace
          Using strategy: Default
          Last Built Revision: Revision dfe102e4a93d71f242bd23d67363c9398f01ea54 (refs/remotes/origin/master)
          Cloning the remote Git repository
          Using no checkout clone with sparse checkout.
          Cloning repository ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git
           > /usr/bin/git init /var/lib/jenkins/jobs/testrs_git/workspace # timeout=360
          Fetching upstream changes from ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git
           > /usr/bin/git --version # timeout=360
          using GIT_SSH to set credentials ttbsbldmac101.ttg.global
           > /usr/bin/git -c core.askpass=true fetch --tags --progress ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git +refs/heads/*:refs/remotes/origin/*

           > /usr/bin/git config remote.origin.url ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git # timeout=360
           > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=360
           > /usr/bin/git config remote.origin.url ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git # timeout=360
          Cleaning workspace
           > /usr/bin/git rev-parse --verify HEAD # timeout=360
          No valid HEAD. Skipping the resetting
           > /usr/bin/git clean -fdx # timeout=360
          Fetching upstream changes from ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git
          using GIT_SSH to set credentials ttbsbldmac101.ttg.global
           > /usr/bin/git -c core.askpass=true fetch --tags --progress ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git +refs/heads/*:refs/remotes/origin/*
          getCandidateRevisions(false,origin/master,,,,remoteUrls=[ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git],buildsByBranchName={refs/remotes/origin/master=Build #15 of Revision dfe102e4a93d71f242bd23d67363c9398f01ea54 (refs/remotes/origin/master)},lastBuild=Build #15 of Revision dfe102e4a93d71f242bd23d67363c9398f01ea54 (refs/remotes/origin/master)]) considering branches to build
          Qualifying origin/master as a branch in repository origin -> refs/remotes/origin/master
          Qualifying origin/master as a branch in repository origin -> refs/remotes/origin/origin/master
           > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=360
          rev-parse refs/remotes/origin/master -> AnyObjectId[dfe102e4a93d71f242bd23d67363c9398f01ea54]
          Found a new commit AnyObjectId[dfe102e4a93d71f242bd23d67363c9398f01ea54] to be built on refs/remotes/origin/master
           > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=360
          Failed to rev-parse: refs/remotes/origin/origin/master
          Checking out Revision dfe102e4a93d71f242bd23d67363c9398f01ea54 (refs/remotes/origin/master)
           > /usr/bin/git config core.sparsecheckout # timeout=360
           > /usr/bin/git config core.sparsecheckout true # timeout=360
           > /usr/bin/git read-tree -mu HEAD # timeout=360
          Command "/usr/bin/git read-tree -mu HEAD" returned status code 128:
          stdout:
          stderr: fatal: Not a valid object name HEAD

           > /usr/bin/git checkout -f dfe102e4a93d71f242bd23d67363c9398f01ea54
           > /usr/bin/git rev-list dfe102e4a93d71f242bd23d67363c9398f01ea54 # timeout=360
          [workspace] $ /bin/sh -xe /var/lib/jenkins/tmp/hudson5727280207521334905.sh
          + find . '!' -path '*/.git/*'
          + sort
          .
          ./dir1
          ./dir1/test1
          ./dir2
          ./dir2/test2
          ./.git
          Started calculate disk usage of build
          Finished Calculation of disk usage of build in 0 seconds
          Started calculate disk usage of workspace
          Finished Calculation of disk usage of workspace in 0 seconds
          Finished: SUCCESS
          {code}

          Repositories (attached):
          * testcommon
          ** commonfile1
          ** commonfile2

          * testmain
          ** dir1/test1
          ** dir2/test2
          ** dir3/test3
          ** testcommon (submodule; referencing testcommon via ../testcommon.git)

          Jenkins job settings:
          * Source code management: GIT
          ** Repository: ssh://git@localhost:19999/mnt/fatdisk/git/testmain.git
          ** Branches: */master
          ** Clean after checkout
          ** Advanced submodules behavior
          *** Recursively update submodules: checked
          ** Sparse checkout paths
          *** dir1
          *** dir2
          * Execute shell (to log working copy)
          ** find . ! -path "\*/.git/\*" | sort

          If I remove the sparse checkout option, the submodule is checked out correctly.







          Ronny Schuetz made changes -
          Environment New: jenkins 1.580.3, git-plugin-2.3.5

          Mark Waite added a comment -

          Have you confirmed that git itself is able to support submodules with sparse checkout in the configuration you're using?

          Mark Waite added a comment - Have you confirmed that git itself is able to support submodules with sparse checkout in the configuration you're using?

          Ronny Schuetz added a comment -

          @Mark: Good point. Just tried it manually and had to add .gitmodules to .git/info/sparse-checkout to make it work; adding .gitmodules to the sparse checkout paths in the Jenkins job seems to help here as well.

          Maybe it would help, if git-internal files would be checked out always, independently of the sparse checkout option?

          Ronny Schuetz added a comment - @Mark: Good point. Just tried it manually and had to add .gitmodules to .git/info/sparse-checkout to make it work; adding .gitmodules to the sparse checkout paths in the Jenkins job seems to help here as well. Maybe it would help, if git-internal files would be checked out always, independently of the sparse checkout option?
          R. Tyler Croy made changes -
          Workflow Original: JNJira [ 161275 ] New: JNJira + In-Review [ 180638 ]

          Ori Wiesel added a comment -

          Hey, i'm having the same problem, but, i tried adding .gitmoudules to sparse checkoust paths.
          and it didn't solve my problem...

          any ideas?

          Ori Wiesel added a comment - Hey, i'm having the same problem, but, i tried adding .gitmoudules to sparse checkoust paths. and it didn't solve my problem... any ideas?

          Weihong Guan added a comment -

          Any workaround for it?

          Weihong Guan added a comment - Any workaround for it?

          Mark Waite added a comment -

          aguegu the workaround is to not use sparse checkout.

          Mark Waite added a comment - aguegu the workaround is to not use sparse checkout.

          Weihong Guan added a comment - - edited

          markewaite My workaround is not use submodule. All I need is a shell script, but I think it is more polite to import it as a submodule not the raw source file. Checking out the whole repo is of too much waste in my case.

          Weihong Guan added a comment - - edited markewaite My workaround is not use submodule. All I need is a shell script, but I think it is more polite to import it as a submodule not the raw source file. Checking out the whole repo is of too much waste in my case.

            Unassigned Unassigned
            rschuetz Ronny Schuetz
            Votes:
            6 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: