Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59516

There is no way to pull LFS files for submodules in GIT

      As far as I can tell, there is no way to checkout LFS files for git submodules using the Source Code Management tab of the configure screen. This means that I am currently unable to build projects with jenkins that are dependent on a git project with submodules that have LFS files.

          [JENKINS-59516] There is no way to pull LFS files for submodules in GIT

          Lucas Vander Wal added a comment - See  https://stackoverflow.com/q/58085269/6322589

          René Scheibe added a comment -

          I think currently the only option to handle this is outside of the job configuration.

          Many git LFS installers automatically set up a global or system wide "lfs" filter as follows:

          [filter "lfs"]
              clean = git-lfs clean -- %f
              smudge = git-lfs smudge -- %f
              process = git-lfs filter-process
              required = true 

          A configuration like this on a Jenkins agent would fetch all LFS files (even without the "Git LFS pull after checkout" behaviour configured on a Jenkins job).

          René Scheibe added a comment - I think currently the only option to handle this is outside of the job configuration. Many git LFS installers automatically set up a global or system wide "lfs" filter as follows: [filter "lfs" ] clean = git-lfs clean -- %f smudge = git-lfs smudge -- %f process = git-lfs filter-process required = true A configuration like this on a Jenkins agent would fetch all LFS files (even without the "Git LFS pull after checkout" behaviour configured on a Jenkins job).

          Ian Panzer added a comment -

          I have hit this issue and i's going to take a fair bit of work to get around. Allowing a Git LFS pull to happen during the submodule checkout would be awesome.

          Ian Panzer added a comment - I have hit this issue and i's going to take a fair bit of work to get around. Allowing a Git LFS pull to happen during the submodule checkout would be awesome.

          I've also came accross this today. Unfortunately, its a nasty issue. However, I did write a workaround. After doing the checkout, I run a follow-up git pull on the submodules:

          sshagent(credentials: ['my-ssh-key-id']) {
              sh 'GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" git submodule foreach git lfs pull'
          }

          This helps me move forward for the time being.

          Bogomil Vasilev added a comment - I've also came accross this today. Unfortunately, its a nasty issue. However, I did write a workaround. After doing the checkout, I run a follow-up git pull on the submodules: sshagent(credentials: [ 'my-ssh-key-id' ]) { sh 'GIT_SSH_COMMAND= "ssh -o UserKnownHostsFile=/dev/ null -o StrictHostKeyChecking=no" git submodule foreach git lfs pull' } This helps me move forward for the time being.

          Mark Waite added a comment -

          smirky that work around or the workaround using the

          withCredentials { }

          block as described in the git plugin documentation are the preferred technique. I don't intend to add this capability to the git plugin submodule support.

          Mark Waite added a comment - smirky that work around or the workaround using the withCredentials { } block as described in the git plugin documentation are the preferred technique. I don't intend to add this capability to the git plugin submodule support.

          Bogomil Vasilev added a comment - - edited

          markewaite - I will try switching to withCredentials{} rather than sshagent{}. As for the GitLFS support for submodules, why isn't this suppose to be supported? How else can you checkout the submodules without providing such workarounds? There's also the problem that I have to hardcode credential IDs in the Jenkinsfile, rather than rely on the scm ihnerited settings form the Job configuration.

          EDIT: withCredentials{} appears to be only usable when you combine with "gitUsernamePassword", which is not why we use to authenticate to our git repos. We use ssh, therefore my workaround is still the only viable workaround.

          Bogomil Vasilev added a comment - - edited markewaite  - I will try switching to withCredentials{} rather than sshagent{}. As for the GitLFS support for submodules, why isn't this suppose to be supported? How else can you checkout the submodules without providing such workarounds? There's also the problem that I have to hardcode credential IDs in the Jenkinsfile, rather than rely on the scm ihnerited settings form the Job configuration. EDIT: withCredentials{} appears to be only usable when you combine with "gitUsernamePassword", which is not why we use to authenticate to our git repos. We use ssh, therefore my workaround is still the only viable workaround.

          Mark Waite added a comment -

          I agree that the ssh-agent technique you're using is the preferred workaround until ssh private key support has been added to the git plugin withCredentials step.

          I'm not willing to add LFS pull for submodules to the git plugin because the submodule support in the git plugin is already riddled with flaws and complications due to flawed assumptions in the original implementation. Any change to that submodule code risks compatibility for existing use cases.

          Mark Waite added a comment - I agree that the ssh-agent technique you're using is the preferred workaround until ssh private key support has been added to the git plugin withCredentials step. I'm not willing to add LFS pull for submodules to the git plugin because the submodule support in the git plugin is already riddled with flaws and complications due to flawed assumptions in the original implementation. Any change to that submodule code risks compatibility for existing use cases.

          John Rocha added a comment -

          How would we get submodule git lfs pull to work for a Freestyle project?

          The stackoverflow link in the very first comment is dead and I don't know where I would add the filter information as renescheibe mentioned in the second comment.

          John Rocha added a comment - How would we get submodule git lfs pull to work for a Freestyle project? The stackoverflow link in the very first comment is dead and I don't know where I would add the filter information as renescheibe  mentioned in the second comment.

          Mark Waite added a comment - - edited

          rocha_stratovan use the git credentials binding that is available with the current release of the git plugin.. That allows your freestyle project to perform authenticated git commands of your choice. If you know the git command you want to perform and your repository is accessed by HTTPS, then the git credentials binding should allow you to perform the operation.

          If your repository is authenticated with SSH, then you'll need to use the ssh-agent plugin in the freestyle job.

          Mark Waite added a comment - - edited rocha_stratovan use the git credentials binding that is available with the current release of the git plugin.. That allows your freestyle project to perform authenticated git commands of your choice. If you know the git command you want to perform and your repository is accessed by HTTPS, then the git credentials binding should allow you to perform the operation. If your repository is authenticated with SSH, then you'll need to use the ssh-agent plugin in the freestyle job.

          Mark Waite added a comment -

          Credentials binding and separate calls to git are the preferred solution for this use case. I don't plan to add additional facilities to support LFS in submodules.

          Mark Waite added a comment - Credentials binding and separate calls to git are the preferred solution for this use case. I don't plan to add additional facilities to support LFS in submodules.

            Unassigned Unassigned
            lucasvw Lucas Vander Wal
            Votes:
            4 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: