Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-35687

Add explicit support for git lfs

    XMLWordPrintable

Details

    Description

      I would like to use git lfs and I would like to jenkins, and I would like for jenkins to use its same credentials already configured for git to fetch the git lfs resources. Rather than trying to change the init+fetch pattern, I would like to add a custom behavior to the git plugin that will execute `git lfs pull` after checkout using the same credentials as the fetch command.

      It seems that git lfs is the way forward for git repos with large files and I don't see a workaround right now for using git lfs with jenkins, short of sprinkling credentials everywhere so that `git lfs` can be run from the build context which doesn't seem like a good idea.

      I am looking into what it would take to add this. Would the maintainers be interested in taking in such a feature?

      Attachments

        Issue Links

          Activity

            matthauck Matt Hauck added a comment - Added some simple pull requests to get this going: https://github.com/jenkinsci/git-client-plugin/pull/206 https://github.com/jenkinsci/git-plugin/pull/414
            mcsf M Chon added a comment -

            I set up a Mac Mini slave, and installed git-lfs into /usr/local/bin/git
            Then I configured the slave to use /usr/local/bin/git instead of /usr/bin/git
            It seems to work just fine - LFS files are fetched seamlessly.

            mcsf M Chon added a comment - I set up a Mac Mini slave, and installed git-lfs into /usr/local/bin/git Then I configured the slave to use /usr/local/bin/git instead of /usr/bin/git It seems to work just fine - LFS files are fetched seamlessly.

            See the related discussion in the Git LFS issue tracker https://github.com/github/git-lfs/issues/640

            swashbuck1r Spike Washburn added a comment - See the related discussion in the Git LFS issue tracker https://github.com/github/git-lfs/issues/640
            creste3 John Smith added a comment - I created some pull requests that fixed a few issues with matthauck 's PRs: https://github.com/jenkinsci/git-client-plugin/pull/232 https://github.com/jenkinsci/git-plugin/pull/463
            markewaite Mark Waite added a comment - - edited

            I've added an unauthenticated verification job to my lts-with-plugins docker repository. That job passes with git client plugin PR232 and git plugin PR463.

            I've added an authenticated verification job to my private docker repo. That job also passes with git client plugin PR232 and git plugin PR463.

            I've added an authenticated verification job with Jenkins pipeline and it fails to checkout with an error. Refer to the comments on git plugin PR463 for the latest status.

            markewaite Mark Waite added a comment - - edited I've added an unauthenticated verification job to my lts-with-plugins docker repository. That job passes with git client plugin PR232 and git plugin PR463 . I've added an authenticated verification job to my private docker repo. That job also passes with git client plugin PR232 and git plugin PR463 . I've added an authenticated verification job with Jenkins pipeline and it fails to checkout with an error. Refer to the comments on git plugin PR463 for the latest status.

            Code changed in jenkins
            User: Matt Hauck
            Path:
            src/main/java/org/jenkinsci/plugins/gitclient/CheckoutCommand.java
            src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
            src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java
            http://jenkins-ci.org/commit/git-client-plugin/e7ddf98e290de8c38b05807c579359dd1782f7de
            Log:
            JENKINS-35687 Add simple git lfs support

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Matt Hauck Path: src/main/java/org/jenkinsci/plugins/gitclient/CheckoutCommand.java src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java http://jenkins-ci.org/commit/git-client-plugin/e7ddf98e290de8c38b05807c579359dd1782f7de Log: JENKINS-35687 Add simple git lfs support
            markewaite Mark Waite added a comment - - edited

            A pre-release of large file support is being tested now. You can download the git client plugin and the git plugin from the lts-with-plugins branch of my docker instance.

            I deeply appreciate comments on the pull request which say "I've installed the pre-release of this plugin and am using it successfully". I also deeply appreciate comments which indicate when a problem has been encountered.

            markewaite Mark Waite added a comment - - edited A pre-release of large file support is being tested now. You can download the git client plugin and the git plugin from the lts-with-plugins branch of my docker instance. I deeply appreciate comments on the pull request which say "I've installed the pre-release of this plugin and am using it successfully". I also deeply appreciate comments which indicate when a problem has been encountered.
            monitorjbl Taylor Jones added a comment -

            Would it be possible to add support for remotes that are not the default LFS remote URL? It would make it a lot easier for teams that are using a separate LFS repo like Artifactory to create Jenkins jobs.

            monitorjbl Taylor Jones added a comment - Would it be possible to add support for remotes that are not the default LFS remote URL ? It would make it a lot easier for teams that are using a separate LFS repo like Artifactory to create Jenkins jobs.
            markewaite Mark Waite added a comment - - edited

            Adding support for remotes that are not the default LFS remote URL will not be in the initial release.

            I don't plan to implement that support, even after the initial release of git LFS because I need to focus on submodule bugs and on authentication bugs once the basic LFS capability has been added.

            I'm willing to evaluate pull requests from others that propose to add such support, but I don't see it high enough on my list to do that myself. There are too many open pull requests with good ideas for me to spend much time writing improvements myself.

            markewaite Mark Waite added a comment - - edited Adding support for remotes that are not the default LFS remote URL will not be in the initial release. I don't plan to implement that support, even after the initial release of git LFS because I need to focus on submodule bugs and on authentication bugs once the basic LFS capability has been added. I'm willing to evaluate pull requests from others that propose to add such support, but I don't see it high enough on my list to do that myself. There are too many open pull requests with good ideas for me to spend much time writing improvements myself.
            thetaiter Ian Tait added a comment -

            markewaite Is there a way to change the timeout time for the LFS pull? Currently it is set at 10 minutes, but we need it to be longer for the first pull to finish successfully.

            thetaiter Ian Tait added a comment - markewaite Is there a way to change the timeout time for the LFS pull? Currently it is set at 10 minutes, but we need it to be longer for the first pull to finish successfully.
            markewaite Mark Waite added a comment -

            Since the LFS pull is performed by "git checkout", the timeout should be modified by adding the additional behavior "Advanced checkout behaviours" and set the timeout to the value you'd like.

            markewaite Mark Waite added a comment - Since the LFS pull is performed by "git checkout", the timeout should be modified by adding the additional behavior "Advanced checkout behaviours" and set the timeout to the value you'd like.
            thetaiter Ian Tait added a comment -

            Ah there it is. Thank you for the help!

            thetaiter Ian Tait added a comment - Ah there it is. Thank you for the help!

            Code changed in jenkins
            User: Mark Waite
            Path:
            src/main/java/org/jenkinsci/plugins/gitclient/CheckoutCommand.java
            src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
            src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java
            src/test/java/org/jenkinsci/plugins/gitclient/GitClientTest.java
            http://jenkins-ci.org/commit/git-client-plugin/470f1257482e0b04c67fcadcbe66d9b4462d2f84
            Log:
            Merge pull request #232 from creste/JENKINS-35687

            JENKINS-35687 Add simple git lfs support

            Compare: https://github.com/jenkinsci/git-client-plugin/compare/9c12a007c5a0...470f1257482e

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/main/java/org/jenkinsci/plugins/gitclient/CheckoutCommand.java src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java src/test/java/org/jenkinsci/plugins/gitclient/GitClientTest.java http://jenkins-ci.org/commit/git-client-plugin/470f1257482e0b04c67fcadcbe66d9b4462d2f84 Log: Merge pull request #232 from creste/ JENKINS-35687 JENKINS-35687 Add simple git lfs support Compare: https://github.com/jenkinsci/git-client-plugin/compare/9c12a007c5a0...470f1257482e

            Code changed in jenkins
            User: Matt Hauck
            Path:
            src/main/java/hudson/plugins/git/extensions/impl/GitLFSPull.java
            src/test/java/hudson/plugins/git/extensions/impl/CheckoutOptionTest.java
            http://jenkins-ci.org/commit/git-plugin/a809052a9b8d59975477e52f106383d63b869830
            Log:
            JENKINS-35687 Add simple git lfs support

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Matt Hauck Path: src/main/java/hudson/plugins/git/extensions/impl/GitLFSPull.java src/test/java/hudson/plugins/git/extensions/impl/CheckoutOptionTest.java http://jenkins-ci.org/commit/git-plugin/a809052a9b8d59975477e52f106383d63b869830 Log: JENKINS-35687 Add simple git lfs support

            Code changed in jenkins
            User: Mark Waite
            Path:
            src/main/java/hudson/plugins/git/extensions/impl/GitLFSPull.java
            src/test/java/hudson/plugins/git/extensions/impl/CheckoutOptionTest.java
            http://jenkins-ci.org/commit/git-plugin/ae6883b49bb3973e6139b666b7cea81464317274
            Log:
            Merge pull request #463 from creste/JENKINS-35687

            JENKINS-35687 Add simple git lfs support

            Compare: https://github.com/jenkinsci/git-plugin/compare/7fb16cbe8ff5...ae6883b49bb3

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/main/java/hudson/plugins/git/extensions/impl/GitLFSPull.java src/test/java/hudson/plugins/git/extensions/impl/CheckoutOptionTest.java http://jenkins-ci.org/commit/git-plugin/ae6883b49bb3973e6139b666b7cea81464317274 Log: Merge pull request #463 from creste/ JENKINS-35687 JENKINS-35687 Add simple git lfs support Compare: https://github.com/jenkinsci/git-plugin/compare/7fb16cbe8ff5...ae6883b49bb3
            matthauck Matt Hauck added a comment -

            This can probably be marked as Resolved fixed now?

            matthauck Matt Hauck added a comment - This can probably be marked as Resolved fixed now?
            markewaite Mark Waite added a comment -

            Not yet. I still need to release the git-plugin that includes the fix. I hope to do that within the next 24 hours.

            markewaite Mark Waite added a comment - Not yet. I still need to release the git-plugin that includes the fix. I hope to do that within the next 24 hours.
            matthauck Matt Hauck added a comment -

            Ah, roger. Just upgraded and realized it wasn't there yet, haha.

            matthauck Matt Hauck added a comment - Ah, roger. Just upgraded and realized it wasn't there yet, haha.
            markewaite Mark Waite added a comment -

            Git plugin 3.1.0 released on 4 Mar 2017 now includes support for command line git large file support.

            markewaite Mark Waite added a comment - Git plugin 3.1.0 released on 4 Mar 2017 now includes support for command line git large file support.

            markewaite that's a wonderful job for supporting git LFS, thanks a lot!

            Is there a way I can use it with pipelines? I've been trying to find examples and/or documentation with no luck. 

            lantero Carlos Ruiz Lantero added a comment - markewaite that's a wonderful job for supporting git LFS, thanks a lot! Is there a way I can use it with pipelines? I've been trying to find examples and/or documentation with no luck. 
            markewaite Mark Waite added a comment - - edited

            The "pipeline syntax" link on most pipeline job pages will present a dropdown of pipeline steps.  One of those is "checkout".  Inside that "checkout" step, you can select git as the source control system, then add the "Additional Behaviour" for Git LFS pull.

            If you prefer to read a script example, refer to the Jenkinsfile in my jenkins-bugs repo. That shows an example using GitLFSPull in a pipeline definition.

            markewaite Mark Waite added a comment - - edited The "pipeline syntax" link on most pipeline job pages will present a dropdown of pipeline steps.  One of those is "checkout".  Inside that "checkout" step, you can select git as the source control system, then add the "Additional Behaviour" for Git LFS pull. If you prefer to read a script example, refer to the Jenkinsfile in my jenkins-bugs repo. That shows an example using GitLFSPull in a pipeline definition.

            People

              markewaite Mark Waite
              matthauck Matt Hauck
              Votes:
              14 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: