Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37495

[REGRESSION] Git Submodules having a name different to their path in working tree, failing to checkout

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • git-plugin
    • Jenkins 2.17, Git Plugin 3.0.0, Git Client Plugin 2.0.0

      I tried to use somehow simplified names for some submodules having more complex path in worktree, ending up in a broken build, as git plugin tries to update-init them using their name instead of their path. leading to a unknown submodule error.

          [JENKINS-37495] [REGRESSION] Git Submodules having a name different to their path in working tree, failing to checkout

          Oh, yeah, this is nasty, I had to downgrade.

          Leandro Lucarella added a comment - Oh, yeah, this is nasty, I had to downgrade.

          I'm having the same issue and can't upgrade to 3.0.0 (from 2.4.0). The issue is reproducible with a submodule move as shown below. The root cause is the name of the submodule in .git/config doesn't need to match the path but the git-plugin assumes that it does. This is why "git submodule update" works, but "git submodule update <name>" does not, and the git-plugin is parsing the .git/config file for submodule names.

          Furthermore, my .git/config file had submodules that didn't exist in the .gitmodules file at that commit. I'm not sure if they were from other branches, or previously deleted but never cleaned up. Long story short, it doesn't seem like the .git/config file is the authoritative source of information about submodules, and neither is the .gitmodules file (people can manually edit it). I think "git ls-files --stage | grep 160000" might be?

          Example of issue (git version 2.10.0):

          git init test
          
          git submodule add <url> UI/translations/submodule
          git commit -m "add submodule"
          
          Content of .git/config at this point:
          [submodule "UI/translations/submodule"]
                  url = <url>
          
          Content of .gitmodules at this point:
          [submodule "UI/translations/submodule"]
                  path = UI/translations/submodule
                  url = <url>
          

          Now we move the submodule.

          mkdir -p  UI/translate
          git mv UI/translations/submodule UI/translate/submodule
          git commit -m "move submodule"
          
          Content of .git/config (unchanged):
          [submodule "UI/translations/submodule"]
                  url = <url>
          
          Content of .gitmodules (only path changed):
          [submodule "UI/translations/submodule"]
                  path = UI/translate/submodule
                  url = <url>
          

          And the git-plugin then does this:

          git config --get-regex ^submodule
          >submodule.components/UI/translations/submodule.url <url>
          
          git submodule update UI/translations/submodule
          error: pathspec 'UI/translations/submodule' did not match any file(s) known to git.
          

          Brandon Jacklyn added a comment - I'm having the same issue and can't upgrade to 3.0.0 (from 2.4.0). The issue is reproducible with a submodule move as shown below. The root cause is the name of the submodule in .git/config doesn't need to match the path but the git-plugin assumes that it does. This is why "git submodule update" works, but "git submodule update <name>" does not, and the git-plugin is parsing the .git/config file for submodule names. Furthermore, my .git/config file had submodules that didn't exist in the .gitmodules file at that commit. I'm not sure if they were from other branches, or previously deleted but never cleaned up. Long story short, it doesn't seem like the .git/config file is the authoritative source of information about submodules, and neither is the .gitmodules file (people can manually edit it). I think "git ls-files --stage | grep 160000" might be? Example of issue (git version 2.10.0): git init test git submodule add <url> UI/translations/submodule git commit -m "add submodule" Content of .git/config at this point: [submodule "UI/translations/submodule" ] url = <url> Content of .gitmodules at this point: [submodule "UI/translations/submodule" ] path = UI/translations/submodule url = <url> Now we move the submodule. mkdir -p UI/translate git mv UI/translations/submodule UI/translate/submodule git commit -m "move submodule" Content of .git/config (unchanged): [submodule "UI/translations/submodule" ] url = <url> Content of .gitmodules (only path changed): [submodule "UI/translations/submodule" ] path = UI/translate/submodule url = <url> And the git-plugin then does this: git config --get-regex ^submodule >submodule.components/UI/translations/submodule.url <url> git submodule update UI/translations/submodule error: pathspec 'UI/translations/submodule' did not match any file(s) known to git.

          Mark Waite added a comment -

          bjacklyn based on an earlier comment I assume you must be seeing a different behavior. The earlier comment said that the problem first appeared in 3.0.0, yet you're seeing a case in an earlier plugin version.

          Mark Waite added a comment - bjacklyn based on an earlier comment I assume you must be seeing a different behavior. The earlier comment said that the problem first appeared in 3.0.0, yet you're seeing a case in an earlier plugin version.

          Brandon Jacklyn added a comment - - edited

          Yeah I think my issue is actually in the git-client-plugin, but I'm not sure what version of the git-plugin this corresponds to: https://github.com/jenkinsci/git-client-plugin/blame/master/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L987

          Edit: We are using 1.19.0 of the git-client-plugin

          Brandon Jacklyn added a comment - - edited Yeah I think my issue is actually in the git-client-plugin, but I'm not sure what version of the git-plugin this corresponds to: https://github.com/jenkinsci/git-client-plugin/blame/master/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L987 Edit: We are using 1.19.0 of the git-client-plugin

          Alexander Berg added a comment - - edited

          Same issue here after our admin has upgraded Jenkins from 1.6 to 2.26.
          Now we have:

          • Git Plugin 3.0.0 (was 2.4.0)
          • Git Client Plugin 2.0.0 (was 1.19.0)

          with git >= 1.8.13 on build hosts

          Error

          Git Client Plugin introduced the new behaviour for submodules.
          While the idea of tracking submodules by name is fine, the git command expects the path instead.
          So the name is the "wrong" thing for the git submodule update command and the path of a module is the "right" one.

          Temporary fix

          Downgrade:

          • Git Client Plugin 1.19.7 (min. for Groovy Pipeline stuff)
          • Git Plugin 2.4.4

          Question

          Are there any plans for fixing this, so our admin is able to escape the dependency hell?

          MWE

          • Usage of "Recursively update submodules" in Jenkins.
          • git repository:
            $ cat .gitmodules 
            [submodule "jsoncpp"]
                    path = lib/jsoncpp
                    url = https://github.com/open-source-parsers/jsoncpp.git
                    branch = 0.y.z
            
            
          Previous there was logged:
          [...]
           > git submodule init # timeout=10
           > git submodule sync # timeout=10
           > git config --get remote.origin.url # timeout=10
           > git submodule update --init --recursive
          [...]
          Finished: SUCCESS
          
          Now we have:
          [...]
           > git submodule init # timeout=10
           > git submodule sync # timeout=10
           > git config --get remote.origin.url # timeout=10
           > git config --get-regexp ^submodule # timeout=10
           > git config --get submodule.jsoncpp.url # timeout=10
           > git submodule update --init --recursive jsoncpp
          FATAL: Command "git submodule update --init --recursive jsoncpp" returned status code 1:
          stdout: 
          stderr: error: pathspec 'jsoncpp' did not match any file(s) known to git.
          Did you forget to 'git add'?
          
          hudson.plugins.git.GitException: Command "git submodule update --init --recursive jsoncpp" returned status code 1:
          stdout: 
          stderr: error: pathspec 'jsoncpp' did not match any file(s) known to git.
          Did you forget to 'git add'?
          
          	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1752)
          	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1495)
          	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:64)
          	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.execute(CliGitAPIImpl.java:1038)
          	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
          	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:153)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
          	at hudson.remoting.Request$2.run(Request.java:332)
          	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          	at java.lang.Thread.run(Thread.java:745)
          	at ......remote call to buildhost(Native Method)
          	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1433)
          	at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
          	at hudson.remoting.Channel.call(Channel.java:797)
          	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:606)
          	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131)
          	at com.sun.proxy.$Proxy85.execute(Unknown Source)
          	at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:96)
          	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1137)
          	at hudson.scm.SCM.checkout(SCM.java:495)
          	at hudson.model.AbstractProject.checkout(AbstractProject.java:1278)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
          	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
          	at hudson.model.Run.execute(Run.java:1720)
          	at hudson.matrix.MatrixBuild.run(MatrixBuild.java:313)
          	at hudson.model.ResourceController.execute(ResourceController.java:98)
          	at hudson.model.Executor.run(Executor.java:401)
          Finished: FAILURE
          

          The correct call in this example should be

          git submodule update --init --recursive lib/jsoncpp

          Regards,
          Alex

          Alexander Berg added a comment - - edited Same issue here after our admin has upgraded Jenkins from 1.6 to 2.26. Now we have: Git Plugin 3.0.0 (was 2.4.0) Git Client Plugin 2.0.0 (was 1.19.0) with git >= 1.8.13 on build hosts Error Git Client Plugin introduced the new behaviour for submodules. While the idea of tracking submodules by name is fine, the git command expects the path instead. So the name is the "wrong" thing for the git submodule update command and the path of a module is the "right" one. Temporary fix Downgrade: Git Client Plugin 1.19.7 (min. for Groovy Pipeline stuff) Git Plugin 2.4.4 Question Are there any plans for fixing this, so our admin is able to escape the dependency hell? MWE Usage of "Recursively update submodules" in Jenkins. git repository: $ cat .gitmodules [submodule "jsoncpp" ] path = lib/jsoncpp url = https: //github.com/open-source-parsers/jsoncpp.git branch = 0.y.z Previous there was logged: [...] > git submodule init # timeout=10 > git submodule sync # timeout=10 > git config --get remote.origin.url # timeout=10 > git submodule update --init --recursive [...] Finished: SUCCESS Now we have: [...] > git submodule init # timeout=10 > git submodule sync # timeout=10 > git config --get remote.origin.url # timeout=10 > git config --get-regexp ^submodule # timeout=10 > git config --get submodule.jsoncpp.url # timeout=10 > git submodule update --init --recursive jsoncpp FATAL: Command "git submodule update --init --recursive jsoncpp" returned status code 1: stdout: stderr: error: pathspec 'jsoncpp' did not match any file(s) known to git. Did you forget to 'git add' ? hudson.plugins.git.GitException: Command "git submodule update --init --recursive jsoncpp" returned status code 1: stdout: stderr: error: pathspec 'jsoncpp' did not match any file(s) known to git. Did you forget to 'git add' ? at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1752) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1495) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:64) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.execute(CliGitAPIImpl.java:1038) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145) at hudson.remoting.UserRequest.perform(UserRequest.java:153) at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:332) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) at ......remote call to buildhost(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1433) at hudson.remoting.UserResponse.retrieve(UserRequest.java:253) at hudson.remoting.Channel.call(Channel.java:797) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131) at com.sun.proxy.$Proxy85.execute(Unknown Source) at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:96) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1137) at hudson.scm.SCM.checkout(SCM.java:495) at hudson.model.AbstractProject.checkout(AbstractProject.java:1278) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529) at hudson.model.Run.execute(Run.java:1720) at hudson.matrix.MatrixBuild.run(MatrixBuild.java:313) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:401) Finished: FAILURE The correct call in this example should be git submodule update --init --recursive lib/jsoncpp Regards, Alex

          Ethan Trewhitt added a comment - - edited

          I've created a pull request that should address this issue:
          https://github.com/jenkinsci/git-client-plugin/pull/225

          Ethan Trewhitt added a comment - - edited I've created a pull request that should address this issue: https://github.com/jenkinsci/git-client-plugin/pull/225

          Thanks Ethan, this is a nasty regression that prevent a lot of users from upgrading. It would be nice if it can get some attention!

          Leandro Lucarella added a comment - Thanks Ethan, this is a nasty regression that prevent a lot of users from upgrading. It would be nice if it can get some attention!

          Code changed in jenkins
          User: Ethan Trewhitt
          Path:
          .gitmodules
          modules/sshkeys
          http://jenkins-ci.org/commit/git-client-plugin/27a8d74a99d450801cfb0e54d51be5ea87a15218
          Log:
          Added puppetlabs-sshkeys repository as a submodule named 'sshkeys' to test that submodule updates work with a submodule whose name is != their path. JENKINS-37495

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Ethan Trewhitt Path: .gitmodules modules/sshkeys http://jenkins-ci.org/commit/git-client-plugin/27a8d74a99d450801cfb0e54d51be5ea87a15218 Log: Added puppetlabs-sshkeys repository as a submodule named 'sshkeys' to test that submodule updates work with a submodule whose name is != their path. JENKINS-37495

          Code changed in jenkins
          User: Ethan Trewhitt
          Path:
          src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
          http://jenkins-ci.org/commit/git-client-plugin/a035c00bc2553ad8c4b1b621d31f2ce2856eaa23
          Log:
          Explicitly look up the submodule path in .gitmodules rather than assuming that the path is the same as the module name. JENKINS-37495

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Ethan Trewhitt Path: src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java http://jenkins-ci.org/commit/git-client-plugin/a035c00bc2553ad8c4b1b621d31f2ce2856eaa23 Log: Explicitly look up the submodule path in .gitmodules rather than assuming that the path is the same as the module name. JENKINS-37495

          Mark Waite added a comment -

          Fixed in git client plugin 2.2.0, released 2 Jan 2017

          Mark Waite added a comment - Fixed in git client plugin 2.2.0, released 2 Jan 2017

            markewaite Mark Waite
            manschwetus Florian Manschwetus
            Votes:
            11 Vote for this issue
            Watchers:
            17 Start watching this issue

              Created:
              Updated:
              Resolved: