Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56569

Prevent automatic pulling of lfs files if no GitLFSPull option is set

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • git-client-plugin
    • None
    • git-client-plugin 2.7.6

      Depending on the used method to install git on a system, git lfs is enabled per default on a system via these settings in the system-wide config file:

      [filter "lfs"]
          clean = git-lfs clean -- %f
          smudge = git-lfs smudge -- %f
          process = git-lfs filter-process
          required = true
      

      This means that even if no "git lfs pull" is performed explicitly, on checkout lfs files are pulled automatically due to the smudge filter.

      Problems

      1. If the remote server requires authentication and none is configured on the system on a Jenkins agent (e.g. ssh key in ~/.ssh/), the checkout fails.
      2. If lfs files shall not be pulled at all, there is no way to configure this in the plugin.

      I therefore suggest to always set the environment variable GIT_LFS_SKIP_SMUDGE=1 even if no "GitLFSPull" option was specified.
      This makes sure that no "git lfs" commands that are not controlled by the git-client-plugin are launched and thus resolves the issue that the checkout fails in case the remote server requires authentication.
      This also prevents pulling lfs files regardless of a system's configuration and requires to explicitly pull lfs files via the "GitLFSPull" option.

          [JENKINS-56569] Prevent automatic pulling of lfs files if no GitLFSPull option is set

          Florian Klink added a comment -

          If the remote server requires authentication and none is configured on the system (e.g. ssh key), the checkout fails.

          Shouldn't then the clone itself fail, too?
          Normal ssh clone setups nowadays execute `git-lfs-authenticate` to obtain http urls and authentication, even when purely cloning over ssh. Once you're able to clone, resolving LFS pointers should work too.

          If lfs files shall not be pulled at all, there is no way to configure this in the plugin.

          Why don't you want your lfs pointers to be resolved while checking out? If that smudge filter is installed system-wide, why should jenkins override that behaviour at all, or silently flip its default?

          I therefore suggest to always set the environment variable {{GIT_LFS_SKIP_SMUDGE=1}}even if no "GitLFSPull" option was specified.
          This makes sure that no "git lfs" commands that are not controlled by the git-client-plugin are launched and thus resolves the issue that the checkout fails in case the remote server requires authentication.
          This also prevents pulling lfs files regardless of a system's configuration and requires to explicitly pull lfs files via the "GitLFSPull" option.

          Again, this really sounds more like a broken LFS backend on your side, than something that should be changed on Jenkins' side.

           

          This would also be problematic with declarative pipelines, for which there's currently no way to enable the GitLFSPull option.

           

          Florian Klink added a comment - If the remote server requires authentication and none is configured on the system (e.g. ssh key), the checkout fails. Shouldn't then the clone itself fail, too? Normal ssh clone setups nowadays execute `git-lfs-authenticate` to obtain http urls and authentication, even when purely cloning over ssh. Once you're able to clone, resolving LFS pointers should work too. If lfs files shall not be pulled at all, there is no way to configure this in the plugin. Why don't you want your lfs pointers to be resolved while checking out? If that smudge filter is installed system-wide, why should jenkins override that behaviour at all, or silently flip its default? I therefore suggest to always set the environment variable {{GIT_LFS_SKIP_SMUDGE=1}}even if no "GitLFSPull" option was specified. This makes sure that no "git lfs" commands that are not controlled by the git-client-plugin are launched and thus resolves the issue that the checkout fails in case the remote server requires authentication. This also prevents pulling lfs files regardless of a system's configuration and requires to explicitly pull lfs files via the "GitLFSPull" option. Again, this really sounds more like a broken LFS backend on your side, than something that should be changed on Jenkins' side.   This would also be problematic with declarative pipelines, for which there's currently no way to enable the GitLFSPull option.  

          Florian Klink added a comment -

          It looks like this is already implemented in https://github.com/jenkinsci/git-client-plugin/commit/853603cccd4434b116ef9b8e094c3f5b815aa75a - however, the git lfs pull doesn't seem to properly replace lfs pointers…

          Florian Klink added a comment - It looks like this is already implemented in  https://github.com/jenkinsci/git-client-plugin/commit/853603cccd4434b116ef9b8e094c3f5b815aa75a  - however, the git lfs pull doesn't seem to properly replace lfs pointers…

          Florian Klink added a comment -

          Turns out, lfs pull did replace lfs pointers, but some other cleaning-related "additional behaviours" changed the resolved files back to pointers.

           

          I think this issue can be closed.

           

          I opened JENKINS-59139, as general clone performance has improved a lot recently.

          Florian Klink added a comment - Turns out, lfs pull did replace lfs pointers, but some other cleaning-related "additional behaviours" changed the resolved files back to pointers.   I think this issue can be closed.   I opened  JENKINS-59139 , as general clone performance has improved a lot recently.

          René Scheibe added a comment - - edited

          Regarding this point:

          renescheibe

          If the remote server requires authentication and none is configured on the system (e.g. ssh key), the checkout fails.

          flokli

          Shouldn't then the clone itself fail, too?
          Normal ssh clone setups nowadays execute `git-lfs-authenticate` to obtain http urls and authentication, even when purely cloning over ssh. Once you're able to clone, resolving LFS pointers should work too.

          In the git-client-plugin it's implemented that the following steps are performed

          1. git fetch (see here)
          2. git checkout (see here)
          3. in case of a specified "Git LFS pull after checkout" behaviour also git lfs pull (see here)

          If credentials are specified by the user they are only provided for git fetch and git lfs pull.
          Therefore if git-lfs filter-process is triggered automatically via git checkout the checkout fails because the credentials are not provided here.

          If one really wants to go the other direction to never use GIT_LFS_SKIP_SMUDGE as described in JENKINS-59139 than the checkout should at least always be performed with providing the credentials. But JENKINS-47531 (to provide dedicated LFS credentials) is then still not covered. What's your opinion on this markewaite?

          René Scheibe added a comment - - edited Regarding this point: renescheibe If the remote server requires authentication and none is configured on the system (e.g. ssh key), the checkout fails. flokli Shouldn't then the clone itself fail, too? Normal ssh clone setups nowadays execute `git-lfs-authenticate` to obtain http urls and authentication, even when purely cloning over ssh. Once you're able to clone, resolving LFS pointers should work too. In the git-client-plugin it's implemented that the following steps are performed git fetch (see here ) git checkout (see here ) in case of a specified "Git LFS pull after checkout" behaviour also git lfs pull (see here ) If credentials are specified by the user they are only provided for git fetch and git lfs pull . Therefore if git-lfs filter-process is triggered automatically via git checkout the checkout fails because the credentials are not provided here. If one really wants to go the other direction to never use GIT_LFS_SKIP_SMUDGE as described in JENKINS-59139 than the checkout should at least always be performed with providing the credentials. But JENKINS-47531 (to provide dedicated LFS credentials) is then still not covered. What's your opinion on this markewaite ?

          Mark Waite added a comment -

          I think that JENKINS-47531 (dedicated LFS credentials) is less and less relevant to Jenkins users.

          I'm in favor of making the "Git LFS Pull" option truly mean that LFS is enabled. WIthout it, LFS would be disabled. That makes a very clear distinction between LFS pull enabled and disabled for a job.

          However, renescheibe, wouldn't that mean being incompatible with users that explicitly enabled LFS with the configuration file settings that you mentioned?

          Mark Waite added a comment - I think that JENKINS-47531 (dedicated LFS credentials) is less and less relevant to Jenkins users. I'm in favor of making the "Git LFS Pull" option truly mean that LFS is enabled. WIthout it, LFS would be disabled. That makes a very clear distinction between LFS pull enabled and disabled for a job. However, renescheibe , wouldn't that mean being incompatible with users that explicitly enabled LFS with the configuration file settings that you mentioned?

          René Scheibe added a comment -

          Right, that's a breaking change so to speak.

          If GIT_LFS_SKIP_SMUDGE=1 is always set to disable any enabled smudge filter and git lfs pull is only performed when the "Git LFS pull after checkout" behaviour is specified, users that currently have the smudge filter enabled via a system/global/local git config would have to change their job configuration.

          René Scheibe added a comment - Right, that's a breaking change so to speak. If GIT_LFS_SKIP_SMUDGE=1 is always set to disable any enabled smudge filter and git lfs pull is only performed when the "Git LFS pull after checkout" behaviour is specified, users that currently have the smudge filter enabled via a system/global/local git config would have to change their job configuration.

          René Scheibe added a comment -

          Above I also linked JENKINS-45228 which is about git LFS at merge time. (This here is about git LFS at checkout time).

          René Scheibe added a comment - Above I also linked JENKINS-45228 which is about git LFS at merge time. (This here is about git LFS at checkout time).

          René Scheibe added a comment -

          Above I also linked JENKINS-59516 which is about git LFS pull for submodules. In case it's decided here that the smudge filter is always disabled via GIT_LFS_SKIP_SMUDGE=1, then the referenced issue has to be worked on too. Otherwise it would not be possible at all to pull LFS files in submodules (even not with configuration outside of Jenkins).

          René Scheibe added a comment - Above I also linked JENKINS-59516 which is about git LFS pull for submodules. In case it's decided here that the smudge filter is always disabled via GIT_LFS_SKIP_SMUDGE=1 , then the referenced issue has to be worked on too. Otherwise it would not be possible at all to pull LFS files in submodules (even not with configuration outside of Jenkins).

          Helene W added a comment -

          huhu

          would be cool if this request is implemented (soon).

          In my company we have the use case that we need just some information from a repository but not its large files. Right now there is no possibility to clone a repository without large files. Either it's cloned with large files or you checkout with the option "Git LFS pull after checkout" which actually just postpones the pull to after the checkout.

           

          Please, please, please, implement a possibility to checkout/clone completely without lfs.

          Thank you 

           

          Helene W added a comment - huhu would be cool if this request is implemented (soon). In my company we have the use case that we need just some information from a repository but not its large files. Right now there is no possibility to clone a repository without large files. Either it's cloned with large files or you checkout with the option "Git LFS pull after checkout" which actually just postpones the pull to after the checkout.   Please, please, please, implement a possibility to checkout/clone completely without lfs. Thank you   

          Mark Waite added a comment -

          I'm not likely to implement a switch that will disable LFS completely.

          Next large project for the git plugin is to create withGitHTTPCredentials and withGitSSHCredentials pipeline tasks that will provide the necessary files and environment so that the user can perform exactly the authenticated git commands they want from sh, bat, and powershell steps.

          The idea is that instead of many different switches that are always trying to find the right mix of simplicity and power, the user would perform a checkout with a command like this:

              withGitHTTPCredentials(credentialid: 'my-personal-access-token-for-my-provider') {
                  sh 'git clone --single-branch https://github.com/MarkEWaite/my-private-repo'
              }
          

          The previous syntax with checkout scm would continue to work and be fully supported, but this will allow submodule use cases, LFS use cases, single branch use cases, and more.

          Mark Waite added a comment - I'm not likely to implement a switch that will disable LFS completely. Next large project for the git plugin is to create withGitHTTPCredentials and withGitSSHCredentials pipeline tasks that will provide the necessary files and environment so that the user can perform exactly the authenticated git commands they want from sh , bat , and powershell steps. The idea is that instead of many different switches that are always trying to find the right mix of simplicity and power, the user would perform a checkout with a command like this: withGitHTTPCredentials(credentialid: 'my-personal-access-token-for-my-provider') { sh 'git clone --single-branch https://github.com/MarkEWaite/my-private-repo' } The previous syntax with checkout scm would continue to work and be fully supported, but this will allow submodule use cases, LFS use cases, single branch use cases, and more.

            Unassigned Unassigned
            renescheibe René Scheibe
            Votes:
            4 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated: