If the root of the problem is in JGit, then I don't think we should attempt to solve it in Jenkins. I don't think we're trying to solve other JGit functionality gaps in Jenkins, so this should just be a known limitation. So long as we don't remove support for command line git, that works well for me.

Jesse's proposal make sense. I also don't think gigabytes git repository can be considered a common neither a good practice.
Large binaries shouldn't be stored in git, or at least should rely on git-annex.

I was able to clone linux git repo setting slave -Xmx to a larger value, so for sure JGit require memory allocation to match repo size, but that's expected. Would be interesting to know the git-cli process memory consumption checking out such a large repository

The git command line client on my 64 bit Debian machine reported (through top) 2.5 GB (or more) of virtual memory, though resident memory is typically quite a bit smaller.

Issue is with the upstream jgit implementation, outside the control of the Jenkins project. If they fix it eventually, we can use their fix, otherwise, this won't be fixed.

No change expected from Jenkins for this issue - requires changes in upstream jgit.

Is there a reference/issue on the jgit site we can refer to?

No, there is not a reference or issue in jgit as far as I know. Since the out of memory error could be reproduced with the Linux kernel repository, it probably could be submitted to jgit as an issue.

@Mark as you'r most familiar with JENKINS-19043 and JENKINS-18960, I think you would be the best candidate to open a ticket at jgit - wdyt?

I have confirmed that the hang is visible with the jgit.sh self contained wrapper from the Eclipse jgit site using my large private repository. If time allows, I'll continue the experiment with a smaller repository (like the Linux kernel) and see if the problem can be repeated with a smaller repository. If not, then I can attempt to mutate the smaller repository into a case which shows the problem.

Since all of those activities will come at the expense of my time on the Jenkins project, I'll limit them and may eventually surrender that it is not worth my time to invest significant effort improving jgit when I care more about improving Jenkins.

JGit 3.2.0 resolves this issue, at least with my 2.5 GB repository. JGit 3.2.0 is proposed to be included in the release of git client plugin after 1.6.1 and the release of git plugin after 2.0.1