Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56569

Prevent automatic pulling of lfs files if no GitLFSPull option is set

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • git-client-plugin
    • None
    • git-client-plugin 2.7.6

      Depending on the used method to install git on a system, git lfs is enabled per default on a system via these settings in the system-wide config file:

      [filter "lfs"]
          clean = git-lfs clean -- %f
          smudge = git-lfs smudge -- %f
          process = git-lfs filter-process
          required = true
      

      This means that even if no "git lfs pull" is performed explicitly, on checkout lfs files are pulled automatically due to the smudge filter.

      Problems

      1. If the remote server requires authentication and none is configured on the system on a Jenkins agent (e.g. ssh key in ~/.ssh/), the checkout fails.
      2. If lfs files shall not be pulled at all, there is no way to configure this in the plugin.

      I therefore suggest to always set the environment variable GIT_LFS_SKIP_SMUDGE=1 even if no "GitLFSPull" option was specified.
      This makes sure that no "git lfs" commands that are not controlled by the git-client-plugin are launched and thus resolves the issue that the checkout fails in case the remote server requires authentication.
      This also prevents pulling lfs files regardless of a system's configuration and requires to explicitly pull lfs files via the "GitLFSPull" option.

          [JENKINS-56569] Prevent automatic pulling of lfs files if no GitLFSPull option is set

          René Scheibe created issue -

          Florian Klink added a comment -

          If the remote server requires authentication and none is configured on the system (e.g. ssh key), the checkout fails.

          Shouldn't then the clone itself fail, too?
          Normal ssh clone setups nowadays execute `git-lfs-authenticate` to obtain http urls and authentication, even when purely cloning over ssh. Once you're able to clone, resolving LFS pointers should work too.

          If lfs files shall not be pulled at all, there is no way to configure this in the plugin.

          Why don't you want your lfs pointers to be resolved while checking out? If that smudge filter is installed system-wide, why should jenkins override that behaviour at all, or silently flip its default?

          I therefore suggest to always set the environment variable {{GIT_LFS_SKIP_SMUDGE=1}}even if no "GitLFSPull" option was specified.
          This makes sure that no "git lfs" commands that are not controlled by the git-client-plugin are launched and thus resolves the issue that the checkout fails in case the remote server requires authentication.
          This also prevents pulling lfs files regardless of a system's configuration and requires to explicitly pull lfs files via the "GitLFSPull" option.

          Again, this really sounds more like a broken LFS backend on your side, than something that should be changed on Jenkins' side.

           

          This would also be problematic with declarative pipelines, for which there's currently no way to enable the GitLFSPull option.

           

          Florian Klink added a comment - If the remote server requires authentication and none is configured on the system (e.g. ssh key), the checkout fails. Shouldn't then the clone itself fail, too? Normal ssh clone setups nowadays execute `git-lfs-authenticate` to obtain http urls and authentication, even when purely cloning over ssh. Once you're able to clone, resolving LFS pointers should work too. If lfs files shall not be pulled at all, there is no way to configure this in the plugin. Why don't you want your lfs pointers to be resolved while checking out? If that smudge filter is installed system-wide, why should jenkins override that behaviour at all, or silently flip its default? I therefore suggest to always set the environment variable {{GIT_LFS_SKIP_SMUDGE=1}}even if no "GitLFSPull" option was specified. This makes sure that no "git lfs" commands that are not controlled by the git-client-plugin are launched and thus resolves the issue that the checkout fails in case the remote server requires authentication. This also prevents pulling lfs files regardless of a system's configuration and requires to explicitly pull lfs files via the "GitLFSPull" option. Again, this really sounds more like a broken LFS backend on your side, than something that should be changed on Jenkins' side.   This would also be problematic with declarative pipelines, for which there's currently no way to enable the GitLFSPull option.  

          Florian Klink added a comment -

          It looks like this is already implemented in https://github.com/jenkinsci/git-client-plugin/commit/853603cccd4434b116ef9b8e094c3f5b815aa75a - however, the git lfs pull doesn't seem to properly replace lfs pointers…

          Florian Klink added a comment - It looks like this is already implemented in  https://github.com/jenkinsci/git-client-plugin/commit/853603cccd4434b116ef9b8e094c3f5b815aa75a  - however, the git lfs pull doesn't seem to properly replace lfs pointers…
          Mark Waite made changes -
          Assignee Original: Mark Waite [ markewaite ]

          Florian Klink added a comment -

          Turns out, lfs pull did replace lfs pointers, but some other cleaning-related "additional behaviours" changed the resolved files back to pointers.

           

          I think this issue can be closed.

           

          I opened JENKINS-59139, as general clone performance has improved a lot recently.

          Florian Klink added a comment - Turns out, lfs pull did replace lfs pointers, but some other cleaning-related "additional behaviours" changed the resolved files back to pointers.   I think this issue can be closed.   I opened  JENKINS-59139 , as general clone performance has improved a lot recently.
          René Scheibe made changes -
          Link New: This issue relates to JENKINS-59139 [ JENKINS-59139 ]

          René Scheibe added a comment - - edited

          Regarding this point:

          renescheibe

          If the remote server requires authentication and none is configured on the system (e.g. ssh key), the checkout fails.

          flokli

          Shouldn't then the clone itself fail, too?
          Normal ssh clone setups nowadays execute `git-lfs-authenticate` to obtain http urls and authentication, even when purely cloning over ssh. Once you're able to clone, resolving LFS pointers should work too.

          In the git-client-plugin it's implemented that the following steps are performed

          1. git fetch (see here)
          2. git checkout (see here)
          3. in case of a specified "Git LFS pull after checkout" behaviour also git lfs pull (see here)

          If credentials are specified by the user they are only provided for git fetch and git lfs pull.
          Therefore if git-lfs filter-process is triggered automatically via git checkout the checkout fails because the credentials are not provided here.

          If one really wants to go the other direction to never use GIT_LFS_SKIP_SMUDGE as described in JENKINS-59139 than the checkout should at least always be performed with providing the credentials. But JENKINS-47531 (to provide dedicated LFS credentials) is then still not covered. What's your opinion on this markewaite?

          René Scheibe added a comment - - edited Regarding this point: renescheibe If the remote server requires authentication and none is configured on the system (e.g. ssh key), the checkout fails. flokli Shouldn't then the clone itself fail, too? Normal ssh clone setups nowadays execute `git-lfs-authenticate` to obtain http urls and authentication, even when purely cloning over ssh. Once you're able to clone, resolving LFS pointers should work too. In the git-client-plugin it's implemented that the following steps are performed git fetch (see here ) git checkout (see here ) in case of a specified "Git LFS pull after checkout" behaviour also git lfs pull (see here ) If credentials are specified by the user they are only provided for git fetch and git lfs pull . Therefore if git-lfs filter-process is triggered automatically via git checkout the checkout fails because the credentials are not provided here. If one really wants to go the other direction to never use GIT_LFS_SKIP_SMUDGE as described in JENKINS-59139 than the checkout should at least always be performed with providing the credentials. But JENKINS-47531 (to provide dedicated LFS credentials) is then still not covered. What's your opinion on this markewaite ?
          René Scheibe made changes -
          Description Original: Depending on the used method to install git on a system, git lfs is enabled per default on a system via these settings in the system-wide config file:
          {code:java}
          [filter "lfs"]
              clean = git-lfs clean -- %f
              smudge = git-lfs smudge -- %f
              process = git-lfs filter-process
              required = true
          {code}
          This means that even if no "git lfs pull" is performed explicitly, on checkout lfs files are pulled automatically due to the smudge filter.

          Problems
           # If the remote server requires authentication and none is configured on the system (e.g. ssh key), the checkout fails.
           # If lfs files shall not be pulled at all, there is no way to configure this in the plugin.

          I therefore suggest to always set the environment variable {{GIT_LFS_SKIP_SMUDGE=1}} even if no "GitLFSPull" option was specified.
           This makes sure that no "git lfs" commands that are not controlled by the git-client-plugin are launched and thus resolves the issue that the checkout fails in case the remote server requires authentication.
           This also prevents pulling lfs files regardless of a system's configuration and requires to explicitly pull lfs files via the "GitLFSPull" option.
          New: Depending on the used method to install git on a system, git lfs is enabled per default on a system via these settings in the system-wide config file:
          {code:java}
          [filter "lfs"]
              clean = git-lfs clean -- %f
              smudge = git-lfs smudge -- %f
              process = git-lfs filter-process
              required = true
          {code}
          This means that even if no "git lfs pull" is performed explicitly, on checkout lfs files are pulled automatically due to the smudge filter.

          Problems
           # If the remote server requires authentication and none is configured -on the system- on a Jenkins agent (e.g. ssh key in \{{~/.ssh/}}), the checkout fails.
           # If lfs files shall not be pulled at all, there is no way to configure this in the plugin.

          I therefore suggest to always set the environment variable {{GIT_LFS_SKIP_SMUDGE=1}} even if no "GitLFSPull" option was specified.
           This makes sure that no "git lfs" commands that are not controlled by the git-client-plugin are launched and thus resolves the issue that the checkout fails in case the remote server requires authentication.
           This also prevents pulling lfs files regardless of a system's configuration and requires to explicitly pull lfs files via the "GitLFSPull" option.

          Mark Waite added a comment -

          I think that JENKINS-47531 (dedicated LFS credentials) is less and less relevant to Jenkins users.

          I'm in favor of making the "Git LFS Pull" option truly mean that LFS is enabled. WIthout it, LFS would be disabled. That makes a very clear distinction between LFS pull enabled and disabled for a job.

          However, renescheibe, wouldn't that mean being incompatible with users that explicitly enabled LFS with the configuration file settings that you mentioned?

          Mark Waite added a comment - I think that JENKINS-47531 (dedicated LFS credentials) is less and less relevant to Jenkins users. I'm in favor of making the "Git LFS Pull" option truly mean that LFS is enabled. WIthout it, LFS would be disabled. That makes a very clear distinction between LFS pull enabled and disabled for a job. However, renescheibe , wouldn't that mean being incompatible with users that explicitly enabled LFS with the configuration file settings that you mentioned?
          René Scheibe made changes -
          Link New: This issue relates to JENKINS-45228 [ JENKINS-45228 ]

            Unassigned Unassigned
            renescheibe René Scheibe
            Votes:
            4 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated: