Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-28748

git plugin submodule update --init fails on large submodule

      Git clone command works fine but when I do a submodule init I get a failure message in my build log :

      > git submodule update --init --recursive
      FATAL: Command "git submodule update --init --recursive" returned status code 1:
      stdout: 
      stderr: Cloning into 'xxxx'...
      error: RPC failed; result=18, HTTP code = 200
      fatal: The remote end hung up unexpectedly
      fatal: protocol error: bad pack header
      Clone of 'https://git.xxx' into submodule path 'xxx' failed
      

      But when I do the same from the shell I do not have any issue (it's just taking ages since the repo is big and the PC is slow)

      This is why I suspect a similar issue than that one in : JENKINS-9168

      In the git plugin, in the submodule section I just ticked : "Recursively update submodules"

      So far the workaround is to go manually in the workspace, init the submodule the first time then you can play as usual.

          [JENKINS-28748] git plugin submodule update --init fails on large submodule

          I forgot to mention that you can ask me more details if needed.

          Florent Le Saout added a comment - I forgot to mention that you can ask me more details if needed.

          Mark Waite added a comment -

          Since authentication is not supported with submodules, I assume that means your repository is not secured. If it is not secured, could you configure the submodule URL to use the git protocol instead of the http or https protocol? That would then avoid the inactivity timeout problem reported in JENKINS-9168.

          If the repository is secured, what protocol is being used to fetch the repository?

          Mark Waite added a comment - Since authentication is not supported with submodules, I assume that means your repository is not secured. If it is not secured, could you configure the submodule URL to use the git protocol instead of the http or https protocol? That would then avoid the inactivity timeout problem reported in JENKINS-9168 . If the repository is secured, what protocol is being used to fetch the repository?

          In fact we use kerberos ticket to authenticate with the server.
          To be honest I'm not very familiar with that, but basically I do get a ticket before doing clone or submodule update and then it's transparent.
          But I guess at some point this ticket need to be verified by the server to grant the access.
          So I'm not sure that we can do the same with git protocol, but maybe.
          The problem is anyway to convince IT to do so, and specially since there is other project on this server and they never complained about that...

          I mentioned the other ticket (9168) since maybe we have common behaviour with --progress option transparently done in command line and not in the plugin for submodule init ?

          Basically I think I will have to live with https server...
          So I have to find something to make it work properly.

          Florent Le Saout added a comment - In fact we use kerberos ticket to authenticate with the server. To be honest I'm not very familiar with that, but basically I do get a ticket before doing clone or submodule update and then it's transparent. But I guess at some point this ticket need to be verified by the server to grant the access. So I'm not sure that we can do the same with git protocol, but maybe. The problem is anyway to convince IT to do so, and specially since there is other project on this server and they never complained about that... I mentioned the other ticket (9168) since maybe we have common behaviour with --progress option transparently done in command line and not in the plugin for submodule init ? Basically I think I will have to live with https server... So I have to find something to make it work properly.

          Mark Waite added a comment -

          florentls I'm not sure that the "--progress" trick used with the "git fetch" and/or "git clone" command will work in this case. The git submodule man page does not mention a --progress argument, so I fear it may not be available.

          Mark Waite added a comment - florentls I'm not sure that the "--progress" trick used with the "git fetch" and/or "git clone" command will work in this case. The git submodule man page does not mention a --progress argument, so I fear it may not be available.

          Yes Mark you are correct I haven't seen it either, but for sure there is some tricks done in the terminal which is not done in the plugin...

          Is there any way of getting the env used by the plugin (I guess it's not the same than the same user in the shell)?
          Do we see in the log the exact line which is executed by the plugin, or there some hidden stuff?

          Florent Le Saout added a comment - Yes Mark you are correct I haven't seen it either, but for sure there is some tricks done in the terminal which is not done in the plugin... Is there any way of getting the env used by the plugin (I guess it's not the same than the same user in the shell)? Do we see in the log the exact line which is executed by the plugin, or there some hidden stuff?

          Mark Waite added a comment - - edited

          Authentication certainly uses some hidden steps, otherwise it would risk exposing credential information in the log.

          If you've considered a code level solution, then you'll probably want to fork the repository and work on your own copy for development. If you arrive at a solution for your need, that is a good step. If you later discover that it is a general purpose solution, you can submit it as a pull request.

          Mark Waite added a comment - - edited Authentication certainly uses some hidden steps, otherwise it would risk exposing credential information in the log. If you've considered a code level solution, then you'll probably want to fork the repository and work on your own copy for development. If you arrive at a solution for your need, that is a good step. If you later discover that it is a general purpose solution, you can submit it as a pull request.

          Yes I get your point.

          If I have a bit of time I will try to get that fixed myself. Of course I would be happy if I can provide help on some sort on that project I'm using quite intensively since few years now
          But first I need to get that reproduced with another repo since I'm afraid I will have to fix that on my free time, so without access to my corpo repo ...
          I will let you know.

          But anyway if someone think it's a kind of priority task, and have a bit of spare time and want to fix it, I can help with helping to reproduce the issue with more debug info.

          Florent Le Saout added a comment - Yes I get your point. If I have a bit of time I will try to get that fixed myself. Of course I would be happy if I can provide help on some sort on that project I'm using quite intensively since few years now But first I need to get that reproduced with another repo since I'm afraid I will have to fix that on my free time, so without access to my corpo repo ... I will let you know. But anyway if someone think it's a kind of priority task, and have a bit of spare time and want to fix it, I can help with helping to reproduce the issue with more debug info.

          I've had the same problem and solve it.

          I had checked .gitmodules file and found old links to submodules or duplicates. Then I removed that trash.

          Serhii Fedorenko added a comment - I've had the same problem and solve it. I had checked .gitmodules file and found old links to submodules or duplicates. Then I removed that trash.

          Mark Waite added a comment -

          For those cases where users require more detailed control of the arguments to git submodule, they should use the git credentials binding that was added in June 2021.

          Mark Waite added a comment - For those cases where users require more detailed control of the arguments to git submodule , they should use the git credentials binding that was added in June 2021.

            Unassigned Unassigned
            florentls Florent Le Saout
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: