-
Improvement
-
Resolution: Won't Do
-
Minor
-
None
-
Jenkins ver. 2.73.2
Git plugin ver. 3.6.0
-
Powered by SuggestiMate
I have a git repo, which itself is located on ssh:// remote, but whose [lfs] url points to https:// address.
When I try to check it out, Jenkins job gets stuck at
using GIT_SSH to set credentials
> git lfs pull origin
command. When I run it manually, I see that it is interactively asking for username and password. I don't see any additional settings for "Git LFS pull after checkout" behavior, so I wonder if it would be possible to allow specifying credentials there?
- relates to
-
JENKINS-59139 don't set GIT_LFS_SKIP_SMUDGE=1 anymore while cloning with git 2.15+
-
- Open
-
- links to
[JENKINS-47531] git-lfs: allow specifying separate credentials
In my case git lfs is on https server.
I think my problem is that I need two sets of credentials in order to do the clone:
- ssh public key to get git objects (which I can specify in Repositories->Credentials)
- https password to get lfs blobs
markewaite I am having the same issue, and I would just like to add my 2 cents. I do not think this is as uncommon as you might think. IMO, this is not related to SSH vs HTTPS. The issue is that (as far as I can tell) there is no way to have separate credentials for your git repo and LFS repo. I think this is a common use case for anyone who hosts their LFS files outside of github. In our case, we have our source code in github and our LFS files on a self-hosted nexus server. GitHub uses github credentials (obviously) while LFS authenticates against our LDAP server.
I have implemented this functionality here:
https://github.com/mephi42/git-client-plugin/commits/lfs-credentials
https://github.com/mephi42/git-plugin/commits/lfs-credentials
The patches are not that big or intrusive. The code works in my environment.
markewaite, do you think it is acceptable to put this functionality into Jenkins git plugins? If yes, I can submit pull requests.
mephi42 that change will need a review by stephenconnolly. The plan is to eventually update the credentials implementation in the git plugin to use the latest credentials API. I wouldn't want to bring this change into the plugins if we'll then need to remove it or revert it in the transition to the most recent credentials API.
stephenconnolly, your comments?
The git-client changes seem ok to me... I need to look at the git plugin changes
The git-plugin changes are “close but no cigar”
There’s some tweaks I’d like to see before i’d be Ok with merging the git-plugin changes. Probably better expressed by comments on a PR rather than on this ticket
mephi42 Stephen's review is enough to encourage a pull request to the git client plugin and the git plugin.
Git client plugin component is Included in git client plugin 2.6.0, released 27 Oct 2017
Great!
In the meantime, I have modified my git-plugin pull request (https://github.com/jenkinsci/git-plugin/pull/546) to reference the newly released git client plugin.
There are failing jobs (only windows-8 passed), but it looks as if this is due to timeouts on builders when downloading maven artifacts.
I created two sample jobs that I can use to verify the functionality. We'll need Stephen's comments before the change can be merged.
Any progress? We host our own simple git server and LFS server due to size costs.
A workaround is to add .git-credentials manually on the build slave, then for piplines replace "LFS after checkout" option with a step that runs "git -c credential.helper=store lfs pull origin".
Related: Jenkins will keep trying the same credentials until it times out in 10 minutes. GIT_TRACE=1 shows it's stuck in a loop of seeing Basic auth, setting GIT_ASKPASS, post, 401 error, repeat.
thelq you could download the pull request build and test it in your environment. If it meets your needs and you note that in the pull request, it helps persuade me that the pull request benefits more than just the original submitter.
stephenconnolly, could you please take another look at the pull request? I have some time now to work on improving the pull request in case it's still required.
One more "me too" here.
We have git repos accessed over ssh and lfs is served by https..
afunix if you install the latest release of the git client plugin from the update center and install the pull request build, you can check to see if the proposed change solves the case that is interesting to you.
That helps me and others know if the proposed git plugin pull request meets the needs of more users than the original submitter.
Hi markewaite and stephenconnolly,
We also have this issue with a lot of the projects we support - we use a corporate GitHub Enterprise instance but use a local Artifactory instance to store GitLFS objects, so the credentials are different than the GHE credentials.
We recently had to scrap the SSH authentication to Artifactory, because of a weak cipher vulnerability discovered in its SSH server, so we had to turn to HTTPS authentication and for that we need to input username & API token.
Currently, we are using an ugly workaround of setting the GitLFS smudge filter to "skip" and manually injecting the token to the Jenkins workspace via EnvInject then running "git lfs pull".
bienstock instead of the workaround that you're using, you could download and install a plugin build with the proposed change to add support for separate LFS credentials. Report your results on that pull request so that others know what you observed.
If that solves the problem then you've avoided the workaround and have helped the Jenkins community evaluate the pull request.
If that doesn't solve the problem, you return to your workaround and have helped the Jenkins community evaluate the pull request.
You'll need to install the git client plugin beta release from the experimental update center as well, which will also help the Jenkins community.
For me the plugin fails to load, since I have an outdated Jenkins version.
It's a bit hard to update a live server or a plugin on a live server which serves hundreds builds hourly.
afunix the pre-release plugin depends on git client plugin 3.0.0 beta releases. Be sure that you've installed the updated git client plugin before you install the pre-release git plugin.
Git client plugin 3.0.0 provides a much newer version of JGit and drops the support for Jenkins versions before 2.60.3.
Git plugin 4.0.0 pre-release (like this case) should run on any Jenkins version 2.60.3 or newer.
If your Jenkins version is older than 2.60.3, then you won't be able to use this enhancement anyway. It won't be included in a git plugin version before git plugin 4.0.0, and git plugin 4.0.0 will require at least Jenkins 2.60.3.
Is possible to have some updates on this feature?
Will be a very nice-to-have also for our company because we are using Bitbucket for git repos and JFrog Artifactory for git-lfs artifacs.
amatera that pull request has been labeled as "Later". That means that it is one of over 60 pull requests which were not progressing because they were not high enough priority for me and were not receiving significant attention from others. I labeled those pull requests and closed them so that they would not spend time in the continuous integration servers. They will be reopened when time allows me to visit them again.
My current priorities are:
- Resolve git plugin 4.0 pre-release critical regressions and git client plugin 3.0 pre-release critical regressions
- Test and release git plugin 4.0
- Resolve BuildData bloat bug (JENKINS-19022)
- Test and release BuildData bloat fix
- Improve notifyCommit handling
- Test and release notifyCommit handling improvements
- Improve submodule handling
- Test and release submodule handling improvements
- Improve caching for large git repositories
- Test and release git caching improvements
- Improve changelog handling
- Test and release changelog handling improvements
- Other improvements (like this credentials improvement)
If you are available to help and would like to experiment with this proposed change, you could build it yourself, install it in your environment (development or production depending on how much risk you are willing to take), and report the results of that experiment to the pull request.
If you'd like guidance on how to build that pull request locally, I'm happy to help. I'd prefer to provide that type of help in either the jenkinsci-dev mailing list or in the jenkins gitter channel, but I'm willing to help you build it yourself for testing and use. If you test it and use it, that meets your immediate need and helps me by adding more people that are testing the code.
dominmuda the change has been delivered in git client plugin 2.6.0 and later. The change needed in git plugin has not been delivered and has not been reopened. If you'd like to take PR-546 as a basis for a new pull request, that would be great to have.
The git LFS command line implementation has evolved further since the pull request was originally proposed. This will need to be evaluated with an even broader range of git LFS releases than previously.
I'm trying to revive this. I started from the past PR and addressed some feedback but it's not working quite yet. I'm using LFS on Artifactory, so I need a different credential supplied
https://github.com/patricklangsonos/git-plugin/tree/git-4.2.2/JENKINS-47531
This is currently branched from the 4.2.2 tag because I wanted to test the change in isolation without moving to the 4.3+ codebase. Once I'm happy with it I'll squash it and get a PR open against master.
If anyone wants to test this on a branch other than 4.2.2 or master - it should be pretty easy to cherry-pick the code from my PR to the needed branch. The files I modified haven't changed much if at all since 4.2.2 until now.
I've had a PR open 29 days - is there anything I can do to help move this along?
If you can provide a publicly accessible LFS test repository that requires separate credentials, that would be a great help. I have no way to test your proposed code change. The only Git LFS implementations available to me are from providers like GitHub and Bitbucket. They don't require a separate credential, so I am unable to test your proposed change. I'm hesitant to accept code changes that I can't test.
I'm reviewing the git plugin performance improvement project for Google Summer of Code and will likely not get to this proposal until after Google Summer of Code has completed in September.
Is your CI setup capable of running Docker containers or if not, do the build agents have golang set up? It looks like there may be some test servers that could be used locally.
I'm able to run Docker containers in my working environment.
Agents do not have golang configured.
You can easily test this with a free artifactory cloud account, or with a 30 day trial for self hosting.
[Pricing | JFrog|https://jfrog.com/pricing/#sass]
My team is also encountering issues with this. Our setup involves GitHub Enterprise offloading LFS artifacts to Nexus (which uses username/password credentials). Since Git has removed the ability to use username/password pairs for cloning via HTTPS, we have had to shift to using personal access tokens (which will not work with Nexus).
Is there any intention to ultimately resolve this issue? I can see that this issue has been open for some 5 years now, with the repeated mention that public accessible LFS test repositories are needed to prove-out the change. Can these not be created/leveraged to do so?
mdonah10 I don't intend to make any code change to resolve this issue. Advanced use cases like this one are best satisfied by using the git credentials binding that has been included in git plugin 4.8.0 and later.
The git credentials binding technique allows users to wrap arbitrary shell commands (and bat commands and powershell commands) with git authentication settings. If that technique does not work for this case, I'd be interested to improve the documentation and/or the plugin to make withCredentials better support this use case. I'm not willing to add more options to the git plugin in order to allow separate credentials for git LFS.
I think this is an uncommon use case. The git lfs implementation and documentation focuses on the https access path. I'm unlikely to spend any time adding this use case unless it becomes much more widely used.